Ratings, my new method
Printed From: Progarchives.com
Category: Site News, Newbies, Help and Improvements
Forum Name: Help us improve the site
Forum Description: Help us improve the forums, and the site as a whole
URL: http://www.progarchives.com/forum/forum_posts.asp?TID=44542
Printed Date: June 03 2025 at 20:03
Software Version: Web Wiz Forums 11.01 - http://www.webwizforums.com
Topic: Ratings, my new method
Posted By: Fight Club
Subject: Ratings, my new method
Date Posted: December 17 2007 at 23:13
|
This was originally posted in the collaborator's discussion forum, Micky felt general members should have a say on this as well.
I basically took the formula mentioned above: - N = # of votes for an album
- M = Minimum votes required to enter top 100 (assume this is 50) - A = Arithmetic mean of an album - R = Average rating of all the albums on the site. (assume this is 3.5) - O = Overall Rating O = A * [N / (N + M)] + R * [M / (N+M)] but I incorporated the collab/prog reviewer ratings into it better. I'll use Script for a Jester's Tear as an example:
As of now it has a total of 252 ratings, I counted 44 of those by
Collaborators and Prog Reviewers. That makes 208 non-collab ratings.
Suppose everyone's ratings had a weight of 1 while collabs/prog
reviewers had a weight of 3. I think this is fair, as giving us a
weight of 10 is just ridiculous. I'll leave this up to the admins to
decide though.
So say each collab rating counted as 3. That would make virtually 132 ratings by collabs and reviewers. Now for the album.
61% of Script's ratings are 5 stars. So I took the number of
ratings (N) multiplied by 0.61 and subtracted 23, as 23 of these 5 star
ratings were by collabs. I then multiplied 23 by 3 (since each collab
rating has a weight of 3) and added that to the previous answer.
Multiply this by 5 (since it's 5 stars) and you come up with 998.6. Use
this same method for the 4, 3, 2, and 1 stars. Add everything together,
divide by the virtual number of ratings (which is 340 since the collabs
have extra weight) and this comes out to a general average of 4.36
Now incorporate this into my formula A * [N / (N + M)] + R * [M / (N+M)]
and you come up with an Overall score of 4.25
I worked this out on a number of albums and came out with this list for an example:
Selling England - 4.55
Thick as a brick - 4.53
Wish You Were Here - 4.53
Foxtrot - 4.53
Dark Side - 4.51
Close to the Edge - 4.49
In the Court of the - 4.46
Animals - 4.36
Moving Pictures - 4.36
Red - 4.35
Fragile - 4.34
Pawn Hearts - 4.33
Nursery Cryme - 4.33
Godbluff - 4.31
Hamburger Concerto - 4.30
In a Glass House - 4.30
Si On Avait... - 4.28
Crime of the Century - 4.27
Per Un Amico - 4.26
Moonmadness - 4.26
In the Land of Grey... - 4.26
Depois De Fim - 4.25
Script for a Jester - 4.25
A Farewell to Kings - 4.23
Hybris - 4.22
Hot Rats - 4.21
Storia Di... - 4.20
Birds of Fire - 4.20
Scenes froma Mem.. - 4.19
Elegant Gypsy - 4.14
In Absentia - 4.13
Io Sono Nato - 4.13
Quadrophenia - 4.10
Uomo Di... - 4.06
I only did this with most of the high rankers in the top 100 and a
few other experiments such as Scenes and In Absentia. So far I think
this looks REALLY accurate, especially compared to the way it is now.
Also, the ratings have the ability to fluctuate with this method and
won't just stick in the same spot like they would have with the
previous formula.
By having the 3.5 thrown in there it's a little bit like Easy Livin's proposal, but not too radical. If we determine the average of every album on the site, we get an idea of what rating everything might come close to. This way it edges an album's overall rating toward that mean, but only just a little. This is mainly most effective for new albums and albums without a lot of ratings. It also helps balance the rating against biased votes (which is what we're trying to solve here right?) So, from what I can see we're lessening that problem, but also improving upon what we have already. Another VERY positive thing I noticed is that new albums with only a few reviews will actually start out with lower ratings and climb as they grow more popular. Personally, I think this is a huge improvement over the previous system where albums would always start with a 5 and drop over time. I think this is a great improvement on accuracy because now albums with only 5 reviews won't have a ridiculous rating. Say for example a new album only has 5 ratings. 2 of those are by collabs, 3 are not. 3 people rated it a 5 including one collab. 2 people gave it a 4, one of those also a collab. With this new method the album will only come to a rating of 3.66 instead of a weighted average of 4.55. Does this make more sense? What are your thoughts? ------------- 
|
Replies:
Posted By: Fight Club
Date Posted: December 17 2007 at 23:19
|
I also posted this: Well here's what I'm working with right now, you visibly see all the sh*t I have going through it lol. This is King Crimson's Lizard at the moment.  For anyone who wishes to see how the formula works themselves, I uploaded the excel file http://www.divshare.com/download/3156032-4c8 - http://www.divshare.com/download/3156032-4c8 - here Anyone with Microsoft Excel should be able to open it. I also made it so one can change the overall mean of all the albums (which right now is set to 3.5), and the weight of collaborator/prog reviewer ratings. -------------
|
Posted By: Fight Club
Date Posted: December 18 2007 at 02:01
|
I also posted this in response to someone who had some concerns about the treatment of more obscure albums and less popular genres. I do agree with that there are tons of albums out there that are just too obscure or inaccessible to get the recognition they deserve. However, I do not feel this is as much of a concern when it comes to ratings. Anyone delving into the obscure will realize that these albums won't have as high of a rating, and that's understandable. Because the truth is, the opinions of 200 people is just more trustworthy than the opinions of 10. People who have been exploring prog for a while will understand this. As I've told Micky earlier, the ratings serve their main purpose to those new to prog, and the new proggers will be more likely to get into the more popular stuff before the obscure stuff. The ratings should reflect this. If we get tons of zeuhl and RIO stuff with ratings a good deal higher than everything else, the newbies will be thinking "huh?". You understand what I'm getting at? Also as far as the algorithm goes, the number of ratings is one of the factors incorporated into. A minimum number of ratings for an album to be in the top 100, which also reflects the overall rating of the album. I have that number set at 50 at the moment. Depending on what number we agree on, this can be changed. The way it works is, the further below this number the amount of ratings is, the lower the overall rating will be. So if this number was set to 25 instead of 50, the overall rating would increase. The thing about this new method is, every album can be compared solely by its rating, and a list can be compiled this way. There is no extra algorithm needed to compile a top 100, as the new method makes it so everything can be sorted purely by its rating. One of the improvements of it, in my opinion. EDIT: I thought about this a little more and there are a few things that can be done, concerning your worries. First of all, we can have not just one rating, but two. Their will be the overall rating displayed, which uses the new method, depending on number of reviews, average, etc. But we can also have the arithmetic mean displayed. By having two ratings displayed, people can see both sides. This way everyone is happy. Secondly, I'm not sure about this, but as you said, some genres are more popular than others. Perhaps we can have a slightly different rating method for each genre. For example: I have it set to 50 ratings to be included in the top 100 (which factors into the overall rating). However, for Zeuhl, which is significantly less popular than symphonic prog, 50 ratings is much more impressive than 50 ratings in symph prog. Am I right? So perhaps, the amount needed for top 100 placement could vary between genres. Say it would be 50 for symph prog, while 30 for zeuhl, considering the popularity difference. What do you think of this? 2ND EDIT: I did some editing to the formula. I think you might like this, personally I find it pretty awesome. What I did was I set up separate values to input into the minimum # of ratings field for each genre. I came up with these numbers through their popularity. I figured this out by averaging the amount of votes for the top 20 albums of each genre. Through this I could see that symphonic prog was significantly more popular than every other genre, while indo prog was by far the least. I gave symph prog a value of 50, while everything else's values were decided through diving their averages by that of symph prog, then multiplying by 50. For example: Symph prog is 50, Canterbury - 11, Eclectic - 27, Prog Metal - 24, Zeuhl - 5, and so on... This basically means the symphonic prog genre is about 10 times more popular than the zeuhl genre, at least on this site. This is taken into consideration now when coming up with the overall score, so each album's rating can be compared to that of its genre. Thus no genre is favored over another, and each is considered equally when it comes to determining a rating. I think each album's overall rating could be determined using this method, while for top 100 lists everything should stick to my previous method (the minimum # of ratings value staying at 50.) Here's a visible representation of what I have going on ATM:  Anyone wishing to mess around with it themselves, I uploaded the excel file http://www.divshare.com/download/3156032-4c8 - here -------------
|
Posted By: Fight Club
Date Posted: December 18 2007 at 12:47
|
Okay I extended this even further, making separate rating formulas. I made a couple album pages in the file as examples, but right now there's a formula for an overall rating relating specifically to an album's genre, a rating relating to every (for top 100 purposes), an average giving collab votes extra weight, a straight average of every vote, and the current rating on the site. This way someone can view all sides of spectrum and rely on the method he prefers. Here's Caravan's If I Could... as an example:  Anyone wishing to see all this for themselves can download my Microsoft Excel file http://www.divshare.com/download/3156032-4c8 - here -------------
|
Posted By: micky
Date Posted: December 18 2007 at 13:21
Love what you have here Matt  I'll pm Pat to have a looksie if he hasn't broke away for the holidays .. as I will do in a couple of days myself
------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 13:38
I'm glad you like it  -------------
|
micky wrote:Posted By: micky
Date Posted: December 18 2007 at 13:46
|
really like it... back to work I go... I will look at that formula closer tonight.. got caught up in genre team business last night and didn't get to it. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 16:38
A'ight, awesome. I'm interested to see what you make of it  -------------
|
Posted By: Atavachron
Date Posted: December 18 2007 at 16:43
|
y'know FC, I was ready to dismiss this whole ridiculous idea of yours but frankly results don't lie, and those new adjustments for the top albums are much more inline with reality IMO very interesting, good work |
Posted By: 1800iareyay
Date Posted: December 18 2007 at 16:57
Ugh, you just had to make me look at math on my break didn't you?  Seriously though, this looks great as far as I can tell, but I have one question? Where do the ratings without reviews fit in? Maybe this could also be the solution to the issue Bob brought up about the spam ratings. Maybe if the ratings without reviews didn't count toward the overall rating, it would cease the pointless ratings. We could allow people to rate without reviews, but it would explicitly state that it would ahve no effect on teh overall score unless submitted with a written review. Keep up the good work Matt. |
Posted By: Fight Club
Date Posted: December 18 2007 at 16:59
I'm still tweaking it, but I think so too. The thing is, I have two separate rating methods, one to apply everything into a top 100, and then a genre specific method. Since each genre receives a different treatment I think it makes sense to have a different rating method. It's not too different, just takes in the amount of ratings differently. For example: since RIO is a lot less popular than symphonic prog, the amount of ratings is a less important factor in the overall rating. Also, that's not a COMPLETE list of top albums. A lot are probably still missing, as I only figured out the ratings to those that are there. -------------
|
Posted By: micky
Date Posted: December 18 2007 at 17:00
|
ooohh... micky like that idea... let the people have their ratings
only reviews... but they count only with a review.
Controversial to be sure... but I like it. Probably an
admin/owner call.
------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 17:04
Well the way I have it now, ratings w/out reviews count just as much as non collab ratings with reviews. I did it this way, because at the moment I can't actually SEE all the ratings, nor can I see all the non collab reviews alone. So in my method I gave collab/prog reviewer ratings a weight of 3 with everything else a weight of 1. The way I figured it out was so I didn't have to count all the ratings on a page I highlighted the block of collab reviews and opened the source code in firefox. I then copied that to a word document and told it to fine 5stars.gif, 4stars, and so on so I could tell how many of each there was. I can't do this with the other reviews ratings, because to see all non collab reviews, I also have to show all the collab reviews (therefore I cannot tell them apart in Word). And I just can't see non review ratings in general at all anymore  Of course if we can all decide on the specifics of this new method later. Obviously I'm sure whoever does this rating stuff has access to all the information we need. Then we can decide on weights between ratings, the overall mean, and stuff like that. Some of the factors I have in right now are still my own speculation/decisions. -------------
|
Posted By: Fight Club
Date Posted: December 18 2007 at 17:58
|
OK, I added yet another method into all this. There's a separate sheet now in my Excel file, where one can input the total amount of ratings with reviews, which it will only take into account. The way I did this was I had the page display every review and after counting the collab reviews, I simply subtracted them from the whole to figure out the non-collab reviews. The algorithm now gives non collab ratings a weight of 1 and collab ratings a weight of 3. Same as before, except that ratings without reviews aren't even considered in the equation. The old method is still there, but I just have a separate page for both of them now. Play around with whichever you prefer. Here's a screenshot using Images & Words as an example. Notice how its rating significantly drops with only review-ratings. EDIT: I'm actually not how sure I am of this "only ratings with reviews count" thing. From the looks of it, A LOT of ratings will drop significantly this way.  Get the updated excel file to try on your own http://www.divshare.com/download/3159109-dfd - here -------------
|
Posted By: Sofagrisen
Date Posted: December 18 2007 at 18:07
| This is better than the currant top list algorithm... |
Posted By: micky
Date Posted: December 18 2007 at 18:10
|
I like that a lot.... curious as to what the example album list (from above) might like like now.. using that latest tweek...
------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 18:12
Haha... Well it'll take me a while to go through that sh*t load of albums again, but I'll see. Only thing is, the minimum number of reviews (previously ratings) might have to be changed now since there will be a lot less ratings. Also, what about all the people who have rated TONS of albums, but never reviewed? All that time will be wasted now. -------------
|
Posted By: micky
Date Posted: December 18 2007 at 18:14
I think that is a nice middle course.... for those who want ratings without review here... and those that want them eliminated for integreties sake .. for the sake of the lists. Everyone can RATE the albums . but only those that REVIEW the ablums shall affect the rating. that again.. is probably for closed door dicussion.. and not this thread. But I'd like to see.. and have all see.. just what that does to the lists. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: rushfan4
Date Posted: December 18 2007 at 18:17
|
I think that you have to remember that many of the ratings without reviews may have been done by non-English speaking members of the site who aren't comfortable enough with their English skills to write reviews in English. It wouldn't be fair to exclude their reviews because they don't have written reviews. ------------- 
|
Posted By: 1800iareyay
Date Posted: December 18 2007 at 18:23
What can I say, I'm a Middle Path. I like the effect it had on Matt's list. His algorithm might solve a lot of complaining on the site. |
Posted By: micky
Date Posted: December 18 2007 at 18:24
let's not get off on a tangent here... this is only a tweek... and we have not forgot that M@X strongly support ratings without reviews... however.. they still can rate them... they just won't affect the album's placement. Nothing unfair about that. If it is seriously considered.. it will probably get a thread here to discuss. Just playing with things right now... all of this is unofficial. Call it mental masturbation. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: micky
Date Posted: December 18 2007 at 18:32
that's why it's wroth a looksie... one part of rating without reviews i don't like is the manipulation. And I do suspect that much of the bitchin would not be from those who do not speak English... hell..I suspect they won't say much.. the people that bitch will generally BE the ones who use it to f**k around with albums.. to drop albums they don't like.. and raise those they do. Not for the purpose of the site.. to spotlight great albums.. but simply as a game. My two cents.. and back on topic.. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 18:37
|
I can easily work it out so that only the rating with reviews affect the Top 100 equation and the overall rating uses all the ratings. What do you think of this?
-------------
|
Posted By: 1800iareyay
Date Posted: December 18 2007 at 18:38
I'll only keep you off topic a moment longer. I took the idea to the CZ so the admins can mull it over. You're right in that it's a closed door discussion. |
Posted By: micky
Date Posted: December 18 2007 at 18:44
interesting.... that might be the best option of all.. again.. all sides would be satisfied. The rattngs without reviews 'count' toward the album... for leisure and digging up albums to check out... but the lists are seperate of them in a way.... which if you remember. Was the purpose of the 'radical' proposal I made in a way in Bob's thread. A fair compromise in my eyes.. though I was the only one who liked my original idea hahahha. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: micky
Date Posted: December 18 2007 at 18:46
|
Matt... come up with some lists on those models there.. and get
everyone's opinion. I think everyone would be curious to see how
they affect the big list. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 18:59
Alright so what exactly do you want to see, how the top 100 would look if only ratings with reviews affected it? I'll work it out on 50 or so super-popular albums. I can't do this for much longer tonight though, I've held off studying for my Statistics final exam tomorrow for far too long because of this  -------------
|
Posted By: micky
Date Posted: December 18 2007 at 19:06
hahahhaha... christ almighty... take your time... whenever you have the time... and let's just look at the top 50.... Dean has a point.. .if they really don't affect the list. .there is no sense removing the ratings without reviews from the equation so to speak ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 19:08
That's what I'm checking out. So far it doesn't look like it's changing much  -------------
|
Posted By: micky
Date Posted: December 18 2007 at 19:11
ahhh... if not.. we can kill that part... just for politics sake and let all reviews.. written or not ..count. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 19:18
|
Wow, during all this I actually found 1 person who had the balls to write a 1 star review of Close to the Edge! " Well,
here is is folks- the MOST OVERATED ALBUM ON THIS SITE. Easy..no doubt about
it. The
"epic" close to the edge- starts as a headache- the 1st 3 minutes
sound like a truck filled
with musical instruments colliding with a wall- its horrible- not untill the
femanistic vocals
come in does the song take a slight change for the better. Yes- he sounds like
a woman.
The lyrics are not as 'deep' as everyone says- instead, they are rather basic-
most liekly
sophistcated to an 8th grader. Dont try to analyze "Seasons Change..I get
up..I get Down"
ust meaningless lyrics. The rest of the album is DULL- nothing great here at
all- but a headache, and an oveerated pile of trash. <br><br>BLAH!!!" He could at least spell correctly before posting such a negative review of something most people consider to be a masterpiece -------------
|
Posted By: micky
Date Posted: December 18 2007 at 19:22
hahhahah... I've read that one before... sort of stood out... and just what is femanistic anyway... sort of surprised that review hasn't been axed... having a contrary opinion is one thing... but that was not a review... but a cry out for help from some abused and tortured soul hahahhahah ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 19:40
|
Well, from the looks of it, removing non-review ratings doesn't seem to
be having much of an effect on the Top 100. The most it seems to be
changing any album's rating is by .05. Thick as a Brick had a .02
change, Wish You Were Here - .01, Close to the Edge - 0, Foxtrot - .05,
Crime of the Century - .02, Si On Avait... - .01, Quadrophenia - .02. That's not really a noticeable difference if you ask me. Should I still keep continuing? -------------
|
Posted By: Fight Club
Date Posted: December 18 2007 at 20:41
|
Well, I worked out the ratings of a few more albums, and just made a slightly larger top 100 list than I had before. Remember: this doesn't include everything, only the albums I worked out ratings for. Albums with the same rating I sorted alphabetically 1. 4.55 - Selling England By the Pound 2. 4.53 - Thick as a Brick 3. 4.53 - Wish You Were Here 4. 4.52 - Foxtrot 5. 4.51 - Dark Side of the Moon 6. 4.50 - Close to the Edge 7. 4.46 - In the Court of the Crimson King 8. 4.36 - Animals 9. 4.35 - Fragile 10. 4.35 - Moving Pictures 11. 4.35 - Red 12. 4.33 - Nursery Cryme 13. 4.33 - Pawn Hearts 14. 4.31 - Godbluff 15. 4.31 - The Lamb Lies Down on Broadway 16. 4.30 - In a Glass House 17. 4.30 - Hamburger Concerto 18. 4.29 - Hemispheres 19. 4.28 - Octopus 20. 4.28 - Relayer 21. 4.28 - Si on Avait... 22. 4.27 - Crime of the Century 23. 4.27 - Larks' Tongues in Aspic 24. 4.27 - The Snow Goose 25. 4.26 - In the Land of Grey and Pink 26. 4.26 - Moonmadness 27. 4.26 - Per Un Amico 28. 4.26 - Second Life Syndrome 29. 4.25 - Depois de Fim 30. 4.25 - Script for a Jester's Tear 31. 4.23 - A Farewell to Kings 32. 4.22 - Hybris 33. 4.22 - Mirage 34. 4.22 - Misplaced Childhood 35. 4.21 - Free Hand 36. 4.21 - Hot Rats 37. 4.21 - The Human Equation 38. 4.20 - A Trick of the Tail 39. 4.20 - Birds of Fire 40. 4.20 - Images & Words 41. 4.20 - Storia... 42. 4.19 - Metropolis Pt 2: Scenes from a Memory 43. 4.18 - Lateralus 44. 4.17 - Aqualung 45. 4.17 - Permanent Waves 46. 4.16 - Still Life 47. 4.16 - The Yes Album 48. 4.15 - In Absentia 49. 4.14 - Elegant Gypsy 50. 4.13 - Darwin! 51. 4.13 - Io Sono... 52. 4.13 - Scheherazade 53. 4.11 - Emerson Lake & Palmer 54. 4.10 - Quadrophenia 55. 4.09 - Trespass 56. 4.08 - Brain Salad Surgery 57. 4.08 - Leftoverture 58. 4.06 - Ocean 59. 4.06 - Uomo Di... 60. 4.05 - If I could Do It All Over... 61. 4.04 - 2112 Don't forget there's still probably a lot of albums that would go into this. I just don't have the time to work out the formula on all of them. -------------
|
Posted By: Dim
Date Posted: December 18 2007 at 20:47
|
Math, hurt me! ------------- 
|
Posted By: Fight Club
Date Posted: December 18 2007 at 20:50
 Skip the equations then, just look at the results! -------------
|
Posted By: micky
Date Posted: December 18 2007 at 22:21
|
yeah.. .that didn't change things much.... so if we drop
that little tweek.. and just leave all the reviews in there.. I think
we have a winner.
------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 22:24
Alright, so what's next to do? -------------
|
Posted By: micky
Date Posted: December 18 2007 at 22:28
hahhhah. .sell it brother... sell it! I'd do a summary post in your collab area thread. Telling in general terms..
remember Bob's reaction.. we dont' want to make a drunk out of him
hahhahah State how it is an improvement over the one we
have already. Once you do, we'll get the site
superluminaries to drop in from the penthouse of prog.... and take a
look at it.------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Barla
Date Posted: December 18 2007 at 22:31
I actually haven't listened to Close To The Edge *runs* ------------- http://www.last.fm/user/Barla/?chartstyle=LastfmMyspace">  
|
Posted By: micky
Date Posted: December 18 2007 at 22:32
NOOOO!!!!!!!! hahhhaa. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 22:32
Oh man... this is going to be difficult... I have NO idea how to even start that hahaha.... uhhh.................... geez.... EDIT: Maybe we should get a list of everyone's concerns and problems about ratings in general, and see what it solves, and whether or not there is anything left to be solved. For the most part I think it takes care of the majority of people's problems with ratings, though. -------------
|
Posted By: micky
Date Posted: December 18 2007 at 22:38
a bit of learned advice.... take your time, you have plenty of time to pick at it.....NOTHING happens quickly here. An impatient, go getter of a collab... is a burned out one. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 22:38
|
Eh whatever, I'll see what I can come up with.
-------------
|
Posted By: Barla
Date Posted: December 18 2007 at 22:39
jajajaj I think I will buy it for christmas, and I know I really ought to have it! ------------- http://www.last.fm/user/Barla/?chartstyle=LastfmMyspace">
|
Posted By: Fight Club
Date Posted: December 18 2007 at 22:39
|
I'm gonna do some searching into threads dealing with ratings, and see what problems people seem to have. Then I'll address those in my statement.
-------------
|
Posted By: micky
Date Posted: December 18 2007 at 22:42
I'll be gone for the next 3 weeks.. so just play with it... sleep on it. We can yak aboutt it when I get back. Honestly.. with the hollidays.. nothing will happen with this until next year even if they think it is the greatest thing since sliced bread. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: micky
Date Posted: December 18 2007 at 22:45
of course you do...
Even if you think it doesn't live up to the hype (what does). It
obviously is an album every prog fan should have. A true essential
album of prog.. which reviewers like that twit above.. don't begin to
fathom. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 22:46
Well I'm glad you approve! At least I have one collaborator on my side!  -------------
|
Posted By: micky
Date Posted: December 18 2007 at 22:51
I'll make some offers.. that people can't refuse .
M@X loves to f**k around with things anyway. In the last couple
of weeks.. he's screwed around with everything BUT that. SO the
site is due for more change hahhahahIn blunt speak though... the math and the result are very valid.. I would hate to see you invest all this time... for nothing. If you can show an improvement or refinement over what we have... should be no reason not to at least try it. If M@X doesn't like it after some time... he can change it. He's good at that. ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 23:16
Well I just added even more to it hahaha. Now you can change the weights for collab ratings, ratings with review, and ratings without reviews. I used the percentages to figure out how many ratings don't have reviews. It's pretty fun to play around with hahaha Just by simply changing one number, say the amount of weight given to collabs or the total number of rating, you can see how the whole score will be affected. I like this kind of sh*t haha -------------
|
Posted By: micky
Date Posted: December 18 2007 at 23:24
|
ahhahahha... If you can find a way to get Genesis out of the top 10.. and get ELP in there... I'll get you promoted to SC
------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 23:27
With the way the ratings look for those two, that would impossible without heavily distorting the ratings beyond any form of logic hahaha I think it would be a crime for Genesis not to be in the top 10 anyway haha. Unfortunately ELP, in my opinion, is sadly underrated on the archives  Besides, I'm happy enough with my PR status, no need to become SC -------------
|
Posted By: micky
Date Posted: December 18 2007 at 23:29
worth a try......------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 23:37
Well considering 48 collabs rated Selling England by the Pound a 5, and only 19 collabs rated Brain Salad Surgery a 5 it makes a huge difference haha If you keep the current weights of collabs having a 10, all other reviews a 3, and nonreview ratings a 1, Brain Salad Surgery gets an overall score of 4.05 If I bump collab weight down to 3, and other reviews down to 2 it gets a 4.11 Make all ratings equal and it stays with a 4.11. You want it to get any higher then you better start convincing some collabs that ELP is better than Genesis -------------
|
Posted By: micky
Date Posted: December 18 2007 at 23:46
|
HAH!!!!!! It has taken 2+ years to only gather a small elite
strikeforce of ELP fanatics from the gen pop here... it might take
another 10 to get enough collabs with all the Genesis on the brain
around here Oh well...I guess I will have to accept that ------------- The Pedro and Micky Experience - When one no longer requires psychotropics to trip |
Posted By: Fight Club
Date Posted: December 18 2007 at 23:52
It's funny because there are clearly a good deal of reviews that love Brain Salad Surgery. If you give non-collab reviews a higher weight than the collab ones (ie reverse them - collabs 3, reviews 10) the rating shoots up to a 4.24 -------------
|
Posted By: rushfan4
Date Posted: December 18 2007 at 23:53
|
In ten years all of the acne-faced adolescents will have been promoted to SCs and DT will dominate 6 of the 10 top spots with Opeth dominating the other 4. There won't be any room for Genesis or ELP. -------------
|
Posted By: Fight Club
Date Posted: December 18 2007 at 23:55
For a second I was scared, but then I realized anyone worthy of that position probably listens to a hell of a lot more than just prog metal haha. Hey, I'm only 18, and I love everything equally  And hey, I just started getting into Magma today!!! -------------
|
Posted By: Atavachron
Date Posted: December 19 2007 at 00:12
* shudder * |
Posted By: The Whistler
Date Posted: December 19 2007 at 00:16
|
That first post sounds like math. ... I hate math. I didn't start writing reviews so I could do math. ------------- "There seem to be quite a large percentage of young American boys out there tonight. A long way from home, eh? Well so are we... Gotta stick together." -I. Anderson |
Posted By: Fight Club
Date Posted: December 19 2007 at 12:31
All you need to know is that it's about 10000000X better than the current formula -------------
|
Posted By: The T
Date Posted: December 19 2007 at 14:21
I'm way past the acne phase...But I'll gladly join the cause!
All hail the acne-revolution!
(Could I be the "Comandante"?  )------------- 
|
Posted By: Fight Club
Date Posted: December 20 2007 at 11:21
|
I'm trying to get this new algorithm as perfect as possible, so what I'd like to do is gather any concerns or problems anyone has with the rating system in general. Things you like and things you don't like, what should the ratings take into account, etc. List your opinions here, all of them. I'm listening, and I'll address everything I can. -------------
|
Posted By: Easy Livin
Date Posted: December 20 2007 at 12:03
|
Would it be possible for those of us who are unfamiliar with statistical formulae to explain the logic behind each part of the formula, and the effect it has on the overall calculation in layman's terms?
Have you looked at what effect the proposed formula would have on albums with just over the minimum number of reviews required to enter the the top 50/100 (whatever)?
Does the formula automatically exclude those with less than the minimum number?
What purpose does the minimum number serve in the formula? (I guess I covered that in the first question).
Is this a recognised (i.e. published) formula used in other fields? (Our present one is).
Finally, just to clarify, Collabs reviews do not count as 10 times other members, they count as about 3.3 times. The "10 times" is the relationship to Ratings without Reviews (RwR), and is in response to the concerns frequently expressed about these. Ratings with reviews by non-collabs are weighted by 3 times when compared to RwR. Ratings without reviews will always have a place in the calculations, but their influence is small compared to written reviews. It would be as well to assume that these ratios will remain the same meanwhile.
Cheers!
|
Posted By: rushfan4
Date Posted: December 20 2007 at 12:24
|
Something that came to mind was in regards to when general reviewers become reviewers or special collabs. I think that I read somewhere that the additional weighting for reviewers and special collabs is only on their reviews after they become reviewers or special collabs. The reviews that were done prior to becoming reviewers and special collabs still had the weighting as general reviewers. I don't know if this is correct or not, but I thought that I read this somewhere before. If so, I am not sure how this might effect your algorithms. -------------
|
Posted By: Sofagrisen
Date Posted: December 20 2007 at 12:34
In which fields? The thing is I think maybe the assumptions of the formula are not correct on this site, so it would just be interesting to know what it was originally used for, to see if this site really is an analogy to the original purpose. |
Posted By: Fight Club
Date Posted: December 20 2007 at 13:03
The formula I came up with somewhat of a modified version of the ranking formula IMDB.com (Internet Movie Database) uses, which I tweaked a good bit to fit into Prog Archives. IMDB's formula looks like this which can be found on their http://en.wikipedia.org/wiki/Imdb - wikipedia page It expands upon credibility theory, explained in detail http://stats.lse.ac.uk/norberg/links/papers/CRED-eas.pdf - here where:
 ). It could be set as an extra option to exclude any albums with less than 50 votes from the top 100, but the formula does not work that way. Instead the rating of anything having less than 50 votes will be affected more negatively than something in which the votes exceed 50. As of right now, my method has the 50 vote aspect set only for a total top 100 list. I calculated separate "m"s for each genre, based on its popularity compared to the whole. For example: symphonic prog is the most popular therefore, I set it's m at 50. Krautrock and RIO are significantly less popular so their m's are set at 4 and 7. This makes it so each album can be equally compared in their respective genre without suffering due to a genre's obscurity or lack of popularity. However, I also have the top 100 rating set in as well, so each album gets two ratings. One for its genre, and one for the whole site. This way not only can everything be compared only within its genre, but all together for a more accurate top 100. This also makes it so each album can be compared solely by its rating having the highest rated albums at the top, with the lowest rated at the bottom. As far as C goes, I find that this works out a lot of balance issues as well. C is the arithmetic mean of every rated item on the site. While IMDB has a C of 6.8, I made ours 3.5 since: firstly - I do not have access to the information required to figure out the actual total mean, and secondly - our rating scale is out of 5, and 3.5 seemed reasonable. If this formula is put into effect, of course C will probably be a little different. What C does is it draws the overall rating somewhat towards the average rating of everything. This helps balance out the hater and fanboy ratings that either greatly drag an album down or raise it completely out of proportion. C does not affect albums with a lot of ratings as much as say, new albums with only 5 ratings. I noticed this serves a few purposes solving some issues I have noticed people complaining about in the past. For one thing, it works wonders on newly released albums. Say a prog metal album is just released and 5 people decide to rate it, 3 giving it 5 stars, 2 - 4 stars. 2 people accompany their 5 star rating with a review, while 1 person accompanies their 4. With our current rating system, this album would probably have a really high rating somewhere around 4.6. Of course no one would take it seriously, because it has so few ratings. With the upgraded formula, this album would only receive an overall rating of 3.70 and a 3.86 within its genre. Personally I think this represents reality much better than the current system as the "C" variable takes into account what the rating is "probably closer to". Ultimately, the more an album is rated, the more "essential" it will become, while its new it will still be considered merely "good". Also, I know about the Collab reviews being 3.3 times that of other members reviews. But they are 10 times that of ratings without reviews, are they not? I have these current weights set into my present formula, and they can be changed simply with the click of a button. Does this answer all of your questions? -------------
|
= Weighted Rating
= average for the movie as a number from 0 to 10 (mean) = (Rating)
= number of votes for the movie = (votes)
= minimum votes required to be listed in the Top 250 (currently 1300)
= the mean vote across the whole report (currently 6.7)Posted By: Easy Livin
Date Posted: December 20 2007 at 14:04
|
Cheers FC, I'll need to try to digest that! The reason I mentioned the 1/3/10 weightings is because your original post seemed to imply that there was a differential of 10 throughout, and that you thought this excessive.
A question for others reading. Can you follow the proposal OK, or would you like it explained in more basic terms? (I don't mean that to sound as patronising as it might appear, I'm wondering if it's just me who's having trouble getting to grips with it!)
Rushy - No, when a member is upgraded, all their reviews count as Collaborator reviews.
|
Posted By: Fight Club
Date Posted: December 20 2007 at 14:29
Anything else you would like to know, Bob? And what is your overall opinion of all this? -------------
|
Posted By: Easy Livin
Date Posted: December 20 2007 at 17:16
|
To be honest, I'm still having trouble really understanding the implications of the proposed formula. My perception (which may be totally wrong) is that to some extent it starts with the desired answer and works out what the question is. I do want to understand it as I know mailto:M@x - M@x is always open to suggestions in this field. We have discussed the algorithm in the past at great length, with Fitzcaralldo and MikeER contributing a lot.
I'd be interested to hear what Mike thinks of this proposal compared to the current and previous formulae.
|
Posted By: Fight Club
Date Posted: December 21 2007 at 00:18
Sorry, but I'm not quite sure what you mean by this. -------------
|
Posted By: Easy Livin
Date Posted: December 22 2007 at 05:18
| What I'm saying is that the formula is felt to be right because the resultant chart looks OK. What I'm not clear on is how much adjustment or tweaking has been made to the formula or the values so that it produces the list it does. |
Posted By: Sofagrisen
Date Posted: December 22 2007 at 06:15
|
The only thing "tweaked" in the formula must be C and m. Except for that, everything is as far as I can see and think of identical to IMDb. And I do think IMDb knows quite a lot about this stuff, it’s only the largest rating site online... I think this formula is great! It should be implemented as fast as possible!
By the way, what is wrong with tweaking a formula for the best results anyway? I do know you are worried this sort of thinking will lead less popular and well-known albums to lose terrain, anyhow it is not the case here, and like this site is now, albums with few votes, ergo being less popular and well-known than albums with more votes, should lose terrain. Anyhow, as I said, this formula is not of the kind that needs much tweaking at all. |
Posted By: MikeEnRegalia
Date Posted: December 22 2007 at 06:26
I've been very busy this week and didn't have the time to check it out ... but I will have a look soon. Finally there's someone else who takes rating/ranking methods as seriously as I do! ------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to: 
|
Posted By: Fight Club
Date Posted: December 22 2007 at 15:16
Amen to that! I'm also going to be extremely busy in the following weeks, so I won't be on much. But I'm willing to put as much time as necessary into this at the beginning of the new year. PS. Happy Holidays everyone! -------------
|
Posted By: Fight Club
Date Posted: December 22 2007 at 15:20
Yes, the only tweaked values are C and m and the weight difference between ratings. No extra variables or anything were inserted to force the list/ratings out the way they were. What I'm saying is the results aren't manipulated in any way. -------------
|
Posted By: Easy Livin
Date Posted: December 22 2007 at 16:44
 Let's keep the discussion going.
|
Posted By: MikeEnRegalia
Date Posted: December 23 2007 at 05:05
If you like we could test your formula on my website ... I'd be happy to add it. I already have a couple of ranking methods to choose from ( http://ratingfreak.com/home/charts.xhtml - http://ratingfreak.com/home/charts.xhtml ), and I always wanted to add an IMDB-like method. ------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: January 03 2008 at 02:03
|
OK, well I finally have some free time after my long and extremely busy last week or so! So how's everything going here? I guess I didn't miss much! Alright, so what else do we need to discuss here? In response to Mike, I'm not sure about testing the formula on your site, because I'm not very familiar with it. I made it pretty specific to progarchives, so I don't think it would reflect much similarity between how it would affect ratingfreak and progarchives. -------------
|
Posted By: Easy Livin
Date Posted: January 03 2008 at 04:36
| Mike is very familiar with the PA model, so it would probably be a good idea to take up his offer and test it there. All going well, it would add to the evidence to present to mailto:M@x - M@x . |
Posted By: Fight Club
Date Posted: January 03 2008 at 12:47
|
OK, I'll do whatever it takes! Go ahead with whatever you have in mind, Mike!
-------------
|
Posted By: Fight Club
Date Posted: January 07 2008 at 10:57
|
So what's going on? Is Mike using the system or is he away?
-------------
|
Posted By: MikeEnRegalia
Date Posted: January 07 2008 at 11:16
He's currently too busy, unfortunately. 
------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: January 13 2008 at 17:10
|
Well... eventually we might get this going somewhere...
-------------
|
Posted By: Easy Livin
Date Posted: January 14 2008 at 03:31
|
Keep it in our view FC.
At the moment, mailto:M@x - M@x is very busy with the routine aspects of the site anyway.
|
Posted By: MikeEnRegalia
Date Posted: January 15 2008 at 08:11
|
Ok, I'll add the algorithm to my website later today (in the evening). Is the code in the first post still valid, or has it been updated/improved in the meantime?
------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: January 15 2008 at 11:04
|
Yes, actually it has been edited numerous times since the first post. My following posts should describe the changes in detail. Umm... I'm not exactly sure how they would "look" in a formula.... I'll see if I can come up with something to make this simpler for you, but I'm going to be out all day so it could take a little time. Unless you know how to incorporate the other stuff. We'll see
-------------
|
Posted By: Fight Club
Date Posted: January 15 2008 at 11:50
|
Damn I was just typing all this out and FOR SOME REASON MY PAGE CLOSED AND I LOST IT ALL!!!!!! AHHH!!! I'm going to type this out in Word so I can save it!! But honestly Mike, I think it would be much easier if you just took a look at the Excel file I have uploaded http://www.divshare.com/download/3486316-968 - here Any version of Microsoft Excel from 97-08 should be able to open it. ONLY look at the tab labeled NEW METHOD! The other tabs revolve around the my old method, which isn't as good! The example you're seeing on that tab is King Crimson's Red. If you pan over any of the input fields, I left comment boxes that explain what each one is for. Except for the stuff in the DO NOT EDIT box. That's all the complex stuff... I'm sure you can figure out what it's all for, but it's best to be left alone. -------------
|
Posted By: Sofagrisen
Date Posted: January 16 2008 at 18:42
| I really think something should be done her now. There should be a forum vote about the implementation of this formula, or something. Honestly it is far superior to the one we have today, which is just awful, especially when it comes to albums with few votes. Again, because the numbers of votes on albums vary greatly, their score is not directly comparable. You can only try to make a formula that gives rather stable results, which is not biased towards any number of votes. The formula we have now does not produce stable results at all. As an example Fear of a Blank Planet has many times been pushed of the 2007 album of the year throne, by albums with much fewer votes, but it always reach the #1 spot again, as the album that passed it, gets more votes. This is just one example of the effects of how the rating formula is biased towards albums with few votes. And again, if the formula was good, it would be likely for the new #1 album at staying at the top, but it’s clearly not, because they always drop. Anyhow, the formula of Fight Club will be a great improvement in this respect. |
Posted By: MikeEnRegalia
Date Posted: January 16 2008 at 18:51
|
^ I've been preparing this formula on my website ... unfortunately it's not ready yet, hopefully I'll have it online tomorrow. Now all you have to do is to rate albums on my website, so that we have a lot of data for the formula.
------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: January 17 2008 at 01:11
AWESOME! I have no idea how it will look on your site or how you made it applicable, but we'll see. And Sofagrisen, I am glad you approve so much of my new formula. We can only hope for the best -------------
|
Posted By: MikeEnRegalia
Date Posted: January 17 2008 at 02:54
|
^ I'll simply add it to the chart page ( http://ratingfreak.com/home/charts.xhtml - http://ratingfreak.com/home/charts.xhtml ), to the "Ranking Method" dropdown. The only problem is how to determine the fixed values of the formula automatically from the database. ------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: January 17 2008 at 10:34
|
Hey, how do I get the ratings to a 1-10 scale rather than 1-20? 10 makes a little more sense to me and I see you seem to have that scale in your signature.
-------------
|
Posted By: MikeEnRegalia
Date Posted: January 17 2008 at 11:41
|
^ I changed it for you. 0-10 will be the new common scale at Ratingfreak.com. But I think I'll implement a 5 star display in some situations (with partially filled stars of course).
------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: January 17 2008 at 17:15
|
Cool, I like the 1-10 scale. How's the formula coming along?
-------------
|
Posted By: MikeEnRegalia
Date Posted: January 18 2008 at 06:08
|
Ok, here it is: http://ratingfreak.com/home/charts.xhtml?chart.sort_mode=fc - http://ratingfreak.com/home/charts.xhtml?chart.sort_mode=fc A few comments: - In the calculation the averages used are combinations of arithmetic mean and median. - M is set to the average number of ratings per album of the query at hand (minimum: 5). - R is set to the average rating of all the albums of the query at hand. - The result of the calculation is currently only used for sorting the chart, the rating displayed is the standard average. ------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: January 22 2008 at 00:29
|
Awesome! So what do you think of it all?
-------------
|
Posted By: MikeEnRegalia
Date Posted: January 22 2008 at 02:48
|
I like it! But the current chart template isn't really suitable for comparing ranking algorithms, so I'll try to add some "display variations" later today. I'm thinking about a compact table which displays 50 albums at a time, with columns for the data used in the calculations. I'll let you know when it's available!
------------- https://awesomeprog.com/release-polls/pa" rel="nofollow - Release Polls Listened to:
|
Posted By: Fight Club
Date Posted: February 15 2008 at 00:19
|
erm... bump
-------------
|