Data on score development

Yes, the pattern of declining average ratings, as shown on those examples, is clear. But, is it something that we should be worried about?

Easy Livin wrote: Indeed, it would be useful if you could shown in the lsit what the star ratings were each time for the additional review.

Why would that be interesting? The point is ratings of albums that have high ratings decreases, almost exclusively. And my point then is, when top lists are made, this should be very heavily taken into account. I think the whole algorithm should be based around it, to be honest.

The descending pattern is obvious, but not quite clear because of different ratings people are giving to an album. Now, if you would count the "usual" (non-number-rating sensitive) average simply by counting the arithmetical "middle" and compare those figures to these you observed, that would be nice.

rileydog22 wrote: Yes, and as the rating element of the album drops, the log(ratings) element increases.  That's the idea of the algorithm.

And I totally get it, I know it works that way, but the effect is not large enough, it should be bigger, and it used to be bigger. This is all about finding the right balance, and you haven't found it ...

I can find countless examples in the top list. Look at A.C.T. album Last Epic compared to Tool's Lateralus. The first album has 54 votes and 4,46 in rating. The second has 237 votes and 4,33 in rating. Which album is highest in the top 100 list?

Last Epic, of course, even though there is no chance in hell the album would have had 4,33 or more in rating at 237 votes.

Lateralus is punished because it has so many votes.

Edited by Sofagrisen - October 20 2007 at 21:03

rileydog22 wrote: The ranking algorithm, as MikeEnRegalia will confirm if he checks this part of the site, incorporates both the rating and the number of ratings (I believe the formula is log(ratings)*rating, but I'm not sure).  Things like the P.A. top 100 and a band's "Key Albums" are thus somewhat stable despite the volatile activity of under-reviewed albums.

I know, but the formula still underevaluate albums with many votes. When people read this, they tend to think what I mean is that popularity should count more. But it is not about popularity at all, it is about how as albums get more votes, they tend to drop in rating. But yeah, I get it, you people won't understand what I am talking about until I give you some "hard evidence", some numbers, and not even then, I am sure you are going to get it ...

Edited by Sofagrisen - October 20 2007 at 20:57

Easy Livin wrote: Ah, well that would be a different reason, not simply because there are more ratings. What you say from that perspective does make sense though, an album will not find its true position for a while after it is added.

Well, that is correct in some sense and wrong in another. The thing that is correct is that the more votes an album has, the less is the variation in rating, it practically stabilises. However, it is not like as if half of albums start low and climb and the other half start high and decreases in rating. I have mostly studied high scoring albums, so I am just going to say what happens to them, because I am not equally certain in the case of low scoring albums, and anyhow high scoring albums are the most important. They will generally start by getting very good ratings. After 10 votes, they might have an almost perfect score. As time passes, more balanced reviews will come in. In the first 50 votes, the score of an album usually drops quite a lot. In the next 50 votes the score will drop, but not nearly as much. The score continues to drop less and less, and almost stabilises. That’s how it goes over and over and over again, and it needs to be taken into consideration, because it is basically what happens to every high scoring album …

Edited by Sofagrisen - October 13 2007 at 16:17

Haha, this is the case. I have observed this effect for months, but I haven't started writing down my observations until recently. The effect is very logical. People who vote early on an album will be more likely to be fans and so on, and fans give better grades …

Does anyone have data on score development for albums here on PA? I am kind of trying to gather some myself, but if someone already has, I don’t need to. What I am trying to prove, is my point that as albums gets more votes, their score will go down. I have to wait a few months before my data gets any good. Anyhow, the only useful data so far is:

2nd Hands
8 votes: 4,59
26 votes: 4,12

Voice in the Light
3 votes: 4,75
30 votes: 4,16

This is typical development of album scores. They will start high, drop pretty fast, but ever slower … My point is, this must be the main consideration in the rating algorithm. At the moment it is not taken enough into consideration, so albums with many votes are effectively punished. I have also noticed this effect is considerably larger here than on for example Rate Your Music, but scores tend to be lower there, because of the larger diversity in people voting.

Edited by Sofagrisen - October 13 2007 at 08:56