| Author |
 Topic Search  Topic Options
|
Sofagrisen 
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
 Topic: Data on score development
Posted: October 13 2007 at 08:54 |
|
Does anyone have data on score development for albums here on PA? I am kind of trying to gather some myself, but if someone already has, I don’t need to. What I am trying to prove, is my point that as albums gets more votes, their score will go down. I have to wait a few months before my data gets any good. Anyhow, the only useful data so far is:
2nd Hands
8 votes: 4,59
26 votes: 4,12
Voice in the Light
3 votes: 4,75
30 votes: 4,16
This is typical development of album scores. They will start high, drop pretty fast, but ever slower … My point is, this must be the main consideration in the rating algorithm. At the moment it is not taken enough into consideration, so albums with many votes are effectively punished. I have also noticed this effect is considerably larger here than on for example Rate Your Music, but scores tend to be lower there, because of the larger diversity in people voting.
Edited by Sofagrisen - October 13 2007 at 08:56
|
 |
Easy Livin
Special Collaborator
Honorary Collaborator / Retired Admin
Joined: February 21 2004
Location: Scotland
Status: Offline
Points: 15585
|
Posted: October 13 2007 at 13:12 |
|
Mike is your man for this. I sure he will be able to reassure you that this is not the case.
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: October 13 2007 at 13:36 |
|
Haha, this is the case. I have observed this effect for months, but I haven't started writing down my observations until recently. The effect is very logical. People who vote early on an album will be more likely to be fans and so on, and fans give better grades …
|
|
Easy Livin
Special Collaborator
Honorary Collaborator / Retired Admin
Joined: February 21 2004
Location: Scotland
Status: Offline
Points: 15585
|
Posted: October 13 2007 at 15:55 |
Ah, well that would be a different reason, not simply because there are more ratings. What you say from that perspective does make sense though, an album will not find its true position for a while after it is added.
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: October 13 2007 at 16:15 |
 Easy Livin wrote:
Ah, well that would be a different reason, not simply because there are more ratings. What you say from that perspective does make sense though, an album will not find its true position for a while after it is added.
|
Well, that is correct in some sense and wrong in another. The thing that is correct is that the more votes an album has, the less is the variation in rating, it practically stabilises. However, it is not like as if half of albums start low and climb and the other half start high and decreases in rating. I have mostly studied high scoring albums, so I am just going to say what happens to them, because I am not equally certain in the case of low scoring albums, and anyhow high scoring albums are the most important. They will generally start by getting very good ratings. After 10 votes, they might have an almost perfect score. As time passes, more balanced reviews will come in. In the first 50 votes, the score of an album usually drops quite a lot. In the next 50 votes the score will drop, but not nearly as much. The score continues to drop less and less, and almost stabilises. That’s how it goes over and over and over again, and it needs to be taken into consideration, because it is basically what happens to every high scoring album …
Edited by Sofagrisen - October 13 2007 at 16:17
|
|
rileydog22
Forum Senior Member
Joined: August 24 2005
Location: New Jersey
Status: Offline
Points: 8844
|
Posted: October 18 2007 at 23:47 |
|
The ranking algorithm, as MikeEnRegalia will confirm if he checks this part of the site, incorporates both the rating and the number of ratings (I believe the formula is log(ratings)*rating, but I'm not sure). Things like the P.A. top 100 and a band's "Key Albums" are thus somewhat stable despite the volatile activity of under-reviewed albums.
|
|
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: October 20 2007 at 20:54 |
rileydog22 wrote:
The ranking algorithm, as MikeEnRegalia will confirm if he checks this part of the site, incorporates both the rating and the number of ratings (I believe the formula is log(ratings)*rating, but I'm not sure). Things like the P.A. top 100 and a band's "Key Albums" are thus somewhat stable despite the volatile activity of under-reviewed albums.
|
I know, but the formula still underevaluate albums with many votes. When people read this, they tend to think what I mean is that popularity should count more. But it is not about popularity at all, it is about how as albums get more votes, they tend to drop in rating. But yeah, I get it, you people won't understand what I am talking about until I give you some "hard evidence", some numbers, and not even then, I am sure you are going to get it ...
Edited by Sofagrisen - October 20 2007 at 20:57
|
|
rileydog22
Forum Senior Member
Joined: August 24 2005
Location: New Jersey
Status: Offline
Points: 8844
|
Posted: October 20 2007 at 20:56 |
|
Yes, and as the rating element of the album drops, the log(ratings) element increases. That's the idea of the algorithm.
|
|
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: October 20 2007 at 20:59 |
rileydog22 wrote:
Yes, and as the rating element of the album drops, the log(ratings) element increases. That's the idea of the algorithm.
|
And I totally get it, I know it works that way, but the effect is not large enough, it should be bigger, and it used to be bigger. This is all about finding the right balance, and you haven't found it ...
I can find countless examples in the top list. Look at A.C.T. album Last Epic compared to Tool's Lateralus. The first album has 54 votes and 4,46 in rating. The second has 237 votes and 4,33 in rating. Which album is highest in the top 100 list?
Last Epic, of course, even though there is no chance in hell the album would have had 4,33 or more in rating at 237 votes.
Lateralus is punished because it has so many votes.
Edited by Sofagrisen - October 20 2007 at 21:03
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: December 13 2007 at 08:04 |
|
I have collected some more data now. Does anyone see the pattern?
album title
votes: rating
Fear of a Blank Planet
215: 4,25
221: 4,22
226: 4,23
231: 4,23
238: 4,21
240: 4,20
250: 4,16
259: 4,15
Remember That Night: Live At The Royal Albert Hall
14: 4,74
17: 4,60
18: 4,55
24: 4,33
27: 4,28
Colors
18: 4,50
21: 4,33
26: 4,27
29: 4,34
32: 4,30
35: 4,28
Nil Recurring
68: 4,39
71: 4,23
78: 4,16
82: 4,00
84: 3,91
87: 3,93
91: 3,93
Night
48: 4,35
51: 4,26
53: 4,24
56: 4,21
60: 4,20
64: 4,18
A Time of Day
46: 4,23
48: 4,21
52: 4,19
54: 4,18
Shadows of the Sun
10: 4,68
13: 4,59
17: 4:46
21: 4,34
25: 3,84
27: 3,71
30: 3,97
Frames
9: 4,78
16: 4,52
19: 4,44
22: 4,27
24: 4,33
28: 4,30
Voice in the Light
3: 4,75
30: 4,16
36: 4,22
40: 4,32
43: 4,27
46: 4,22
2nd Hands
8: 4,59
26: 4,12
29: 4,13
32: 4,05
34: 4,06
37: 4,01
In Rainbows
7: 4,09
15: 3,88
26: 3,97
34: 4,01
39: 3,94
42: 3,84
46: 3,92
49: 3,94
51: 3,94
55: 3,93
59: 3,86
Edited by Sofagrisen - December 13 2007 at 08:08
|
|
clarke2001
Special Collaborator
Honorary Collaborator
Joined: June 14 2006
Location: Croatia
Status: Offline
Points: 4160
|
Posted: December 13 2007 at 08:38 |
The descending pattern is obvious, but not quite clear because of different ratings people are giving to an album. Now, if you would count the "usual" (non-number-rating sensitive) average simply by counting the arithmetical "middle" and compare those figures to these you observed, that would be nice. 
(I overlooked the rating weight of review/non-reviewer ratings.)
|
|
|
|
Easy Livin
Special Collaborator
Honorary Collaborator / Retired Admin
Joined: February 21 2004
Location: Scotland
Status: Offline
Points: 15585
|
Posted: December 13 2007 at 10:17 |
Indeed, it would be useful if you could shown in the lsit what the star ratings were each time for the additional review.
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: December 13 2007 at 10:59 |
Easy Livin wrote:
Indeed, it would be useful if you could shown in the lsit what the star ratings were each time for the additional review.
|
Why would that be interesting? The point is ratings of albums that have high ratings decreases, almost exclusively. And my point then is, when top lists are made, this should be very heavily taken into account. I think the whole algorithm should be based around it, to be honest.
|
|
MikeEnRegalia
Special Collaborator
Honorary Collaborator
Joined: April 22 2005
Location: Sweden
Status: Offline
Points: 21840
|
Posted: December 13 2007 at 11:25 |
|
^ how could this phenomenon be taken into account? It's only natural that as an album "ages", reality starts kicking in. Another possibility is that once an album reaches a critical number of review which put it in the top 100 or at the spot of "album of the week", bashers begin submitting 1 star ratings. Regardless of the reasons, how can this be taken into account ... are you suggesting that we should artificially tune album averages depending on how old they are in relation to the release date?
|
|
|
|
Seyo
Special Collaborator
Honorary Collaborator
Joined: May 08 2004
Location: Bosnia
Status: Offline
Points: 1320
|
Posted: December 13 2007 at 12:24 |
|
Yes, the pattern of declining average ratings, as shown on those examples, is clear. But, is it something that we should be worried about?
|
|
Logan
Forum & Site Admin Group
Site Admin
Joined: April 05 2006
Location: In repose.
Status: Offline
Points: 38969
|
Posted: December 13 2007 at 12:51 |
|
I understand what's being said, but I don't really understand the problem with this. Initial reviewers commonly rate too high (especially when an album is new). And others play games with the ratings to bump one album over another. Perhaps early ratings could be weighted down... No, I don't believe in that.
Anyway, I'd prefer a straight average rating while the number of ratings affect the placement on the list (and ratings without reviews could not count for much in terms of placement).
To go completely on a tangent:
One big problem, though, is that there is not a standard rating methodology being used by enough people (I think there should be more detailed criteria for doing a rating, I know that reviews can justify one's rating, but...). Perhaps a table with various criteria, each with a different rating which is then averaged: Originality, creativity, technicality progressiveness, how it rates versus other albums by the artist (if there are any and the person has heard others), how it rates against other artists in the category, how it rates generally compared to the rest of Prog, subjective rating, "objective" rating, or whatever... and then an average of those. Whatever criteria one chooses,; just ideas and some redundancy). One can justify the rating in a review, but the rating process itself should be meaningful.
|
|
Watching while most appreciating a sunset in the moment need not diminish all the glorious sunsets I have observed before. It can be much like that with music for me.
|
|
Easy Livin
Special Collaborator
Honorary Collaborator / Retired Admin
Joined: February 21 2004
Location: Scotland
Status: Offline
Points: 15585
|
Posted: December 13 2007 at 15:34 |
Sofagrisen wrote:
Easy Livin wrote:
Indeed, it would be useful if you could shown in the lsit what the star ratings were each time for the additional review.
|
Why would that be interesting? The point is ratings of albums that have high ratings decreases, almost exclusively. And my point then is, when top lists are made, this should be very heavily taken into account. I think the whole algorithm should be based around it, to be honest. |
It would be of interest because, as Mike suggests, it may actually be lower ratings which are reducing the average. Showing what the ratings were would help you to prove otherwise (if that is the case).
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: December 14 2007 at 05:25 |
|
The thing is, you have to take this into account to realistically compare an album with 50 votes to an album with 500. 5,40 at 500 votes, is better than 4,50 at 50, for example. With 450 extra votes, the album with only 50 votes, is expected to maybe have a rating of 4,00. The differences are pretty huge. Personally I would like to use some kind of regression analysis in determining expected score development based on number of votes, and then the expected score given by a function of score development at maybe 100 votes should be what the top lists are based on. But I am not an actual mathematician. Hopefully there are easier ways to do it.
|
|
MikeEnRegalia
Special Collaborator
Honorary Collaborator
Joined: April 22 2005
Location: Sweden
Status: Offline
Points: 21840
|
Posted: December 14 2007 at 05:51 |
|
^ suppose we would "correct" the score of newly released albums based on our expectancy of how the score is affected by the number of votes ... wouldn't then users adapt to the corrected score? Usually people are motivated to submit their rating because they see the current average, don't agree with it and want to change it. So by "correcting" the average you always influence the voters ...
|
|
|
|
Sofagrisen
Forum Groupie
Joined: January 18 2007
Location: Norway
Status: Offline
Points: 45
|
Posted: December 14 2007 at 06:12 |
MikeEnRegalia wrote:
^ suppose we would "correct" the score of newly released albums based on our expectancy of how the score is affected by the number of votes ... wouldn't then users adapt to the corrected score? Usually people are motivated to submit their rating because they see the current average, don't agree with it and want to change it. So by "correcting" the average you always influence the voters ...
|
I wouldn't like to correct the score you see, but I would like the top lists to be based on for example the expected score at 100 votes, given by a function. It's like that now too, the score that is shown, is the score at this moment...
|
|
/PAlogo_v2.gif "Progarchives.com Homepage")
Data on score development
