Author |
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: January 31 2015 at 12:08 |
Dean wrote:
As the article stated, early votes and reviews for an album tend to come from people who like the album. This is perfectly reasonable since if an album doesn't move you then you will be less motivated to write anything about it.
|
Bands with a large fan (or the opposite!) base will receive skewed ratings for a longer period. A high rating average would imply a small standard deviation (had the rating followed a normal distribution due to the upper rating limit of 5 - perhaps an upper rating limit of 10 would reduce the skewness?). Using the mean favors the fans (or haters) while the median reduces the skewness effect i.e. disfavors the extreme views. Of course, you could encompass all by stating the mode, the mean, the average, the standard deviation and the skewness (and probably more measures) of an album's rating . But to what purpose? who really cares about an albums rating as long as it just indicates how 'good' it is? So in the end does it really matter if PA use the average, mean or whatever to rank an album?
|
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: January 31 2015 at 12:24 |
Dean wrote:
As I have said many times, the best indicator of a bands true rating is not the average, but shape of the rating distribution (the article also used this as an indicator), which does not take into account any reviewer/collab weighting
|
Please define 'average': aritmetic mean, median, mode?
And what do you mean by the 'shape of the rating distribution': the actual curve showing the distribution or some derived measures of that distribution?
Of course the shape of the rating distribution is the best indicator of an album's rating but how would you present this to those who visit PA's site unless by some derived measure(s, like the 'average', standard deviation etc) ? Notwithstanding this, it would be neat with a curve showing the distribution on the site - but wouldn't it scare of potential PA members
Edited by earlyprog - January 31 2015 at 12:25
|
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout
Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
|
Posted: January 31 2015 at 12:36 |
earlyprog wrote:
Dean wrote:
As I have said many times, the best indicator of a bands true rating is not the average, but shape of the rating distribution (the article also used this as an indicator), which does not take into account any reviewer/collab weighting
|
Please define 'average': aritmetic mean, median, mode? |
This was covered in my posts, the average used here is the arithmetic mean.
earlyprog wrote:
And what do you mean by the 'shape of the rating distribution': the actual curve showing the distribution or some derived measures of that distribution? |
The actual curve.
earlyprog wrote:
Of course the shape of the rating distribution is the best indicator of an album's rating but how would you present this to those who visit PA's site unless by some derived measure(s, like the 'average', standard deviation etc) ? Notwithstanding this, it would be neat with a curve showing the distribution on the site - but wouldn't it scare of potential PA members |
Yeah.
|
What?
|
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: January 31 2015 at 12:42 |
earlyprog wrote:
Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II, the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.
*Perhaps I just didn't 'get' the music, but repeated listening convinced me that this emperor in fact did not wear any clothes (someone just had to say so before it was realised by all).
|
This is directed at Kati who appears to continue her CS II crusade under a different banner in this offshoot thread of another thread http://www.progarchives.com/forum/forum_posts.asp?TID=101074.
I could be mistaking?
Hugs to you Kati
Edited by earlyprog - January 31 2015 at 13:18
|
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: January 31 2015 at 12:44 |
Dean wrote:
earlyprog wrote:
Dean wrote:
As I have said many times, the best indicator of a bands true rating is not the average, but shape of the rating distribution (the article also used this as an indicator), which does not take into account any reviewer/collab weighting
|
Please define 'average': aritmetic mean, median, mode? |
This was covered in my posts, the average used here is the arithmetic mean.
earlyprog wrote:
And what do you mean by the 'shape of the rating distribution': the actual curve showing the distribution or some derived measures of that distribution? |
The actual curve.
earlyprog wrote:
Of course the shape of the rating distribution is the best indicator of an album's rating but how would you present this to those who visit PA's site unless by some derived measure(s, like the 'average', standard deviation etc) ? Notwithstanding this, it would be neat with a curve showing the distribution on the site - but wouldn't it scare of potential PA members |
Yeah. |
|
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: January 31 2015 at 13:05 |
Dean wrote:
As the article stated.....
|
Honestly, I don't care much about an article from 2008. This is 2015.
That's why I inserted:
(I probably should read it... sorry I haven't )
Edited by earlyprog - January 31 2015 at 13:25
|
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout
Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
|
Posted: January 31 2015 at 13:49 |
earlyprog wrote:
Dean wrote:
As the article stated.....
|
Honestly, I don't care much about an article from 2008. This is 2015.
That's why I inserted:
(I probably should read it... sorry I haven't )
|
You sir, are a waste of my time and energy. Read the damn article or stfu.
|
What?
|
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: January 31 2015 at 13:57 |
^And this is within your socalled PA 'regulation'
Edited by earlyprog - January 31 2015 at 13:58
|
|
Andy Webb
Special Collaborator
Retired Admin
Joined: June 04 2010
Location: Terria
Status: Offline
Points: 13298
|
Posted: February 02 2015 at 22:03 |
Guldbamsen wrote:
^So if one is looking for say progressive electronic they will have to wade through individual countries and study biographies to see if it's there? Doesn't compute to be honest. I do however miss the opportunity of band listing by country. If we somehow could implement that as a feature on PA, I'd be a happy bunny |
|
|
|
BrufordFreak
Collaborator
Honorary Collaborator
Joined: January 25 2008
Location: Wisconsin
Status: Offline
Points: 7951
|
Posted: February 03 2015 at 09:16 |
Very interesting! Thank you for posting that, Katie. It would seem that PA member/contributors are a bit more judicious in their ratings--probably due to their more serious commitment to "quality" versus "popularity." This is probably why I visit PA thirty times more often than RYM.
|
Drew Fisher https://progisaliveandwell.blogspot.com/
|
|
tboyd1802
Forum Senior Member
Joined: June 01 2012
Location: USA
Status: Offline
Points: 898
|
Posted: February 04 2015 at 19:25 |
earlyprog wrote:
[QUOTE=Dean]
Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II, the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.
|
I think this is pretty fascinating stuff - and my geek apologizes for this. My take is that this is pretty common behavior for new releases, and I would posit that this behavior may be predictable. That is, if you measure the rating at some time point close to album release, you may be able to predict the rating that album eventually settles into. To test this though we would need snap shots of the ratings over time. Do we know if these exist?
|
|
Kazza3
Forum Senior Member
Joined: November 29 2009
Location: Australia
Status: Offline
Points: 557
|
Posted: February 05 2015 at 04:13 |
Yeah, you've posted a lot of interesting information, Dean, thanks!
I wonder if it wouldn't be interesting, if the average on the site is around 3.75, to conform the ratings output to a normal distribution with an average of 3, in the way that IQ distribution is made to always average (or maybe median?) at 100, with standard deviations of set sizes on either side. It'd make it clearer where albums sit in relation to the whole- a ratings system out of 5 where most albums are rated above 4 and most have more 5 star ratings than anything else begins to be less than useful. I know it's like that because this is a site for fans of a specific genre/style who are more than likely going to rate a majority of albums highly, but I'm not sure that's a good argument against the idea- if it's mostly for fans then they already know that most albums are pretty well-liked and it'd be more useful to see clearly where they sit against the whole. On the other hand, it'd obviously screw around with the relationship between the ratings people input and the actual rating the system comes up with, so that despite most people giving an album, say, 3 stars, it'd probably end up closer to 2, reflecting "Collectors/fans only" instead of the generally agreed-upon "Good but non-essential". Interesting anyway.
|
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: February 05 2015 at 04:41 |
tboyd1802 wrote:
earlyprog wrote:
[QUOTE=Dean]
Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II, the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.
|
I think this is pretty fascinating stuff - and my geek apologizes for this. My take is that this is pretty common behavior for new releases, and I would posit that this behavior may be predictable. That is, if you measure the rating at some time point close to album release, you may be able to predict the rating that album eventually settles into. To test this though we would need snap shots of the ratings over time. Do we know if these exist? |
Interesting question you pose there, tboyd1802. I would like to see PA have a statistics group that did this kind of research. With all these rating data available - and properly processed - PA could become a great source for presenting the evolution in peoples' prog tastes. Who knows, 20 years from now top 100 will look entirely different from now. I agree that snap shots of the ratings over time would be a step forward in the statistcis department. However, this kind of data processing it not performed in PA as far as I know.
|
|
earlyprog
Collaborator
Neo / PSIKE / Heavy Teams
Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
|
Posted: February 05 2015 at 04:56 |
Kazza3 wrote:
Yeah, you've posted a lot of interesting information, Dean, thanks!
I wonder if it wouldn't be interesting, if the average on the site is around 3.75, to conform the ratings output to a normal distribution with an average of 3, in the way that IQ distribution is made to always average (or maybe median?) at 100, with standard deviations of set sizes on either side. It'd make it clearer where albums sit in relation to the whole- a ratings system out of 5 where most albums are rated above 4 and most have more 5 star ratings than anything else begins to be less than useful. I know it's like that because this is a site for fans of a specific genre/style who are more than likely going to rate a majority of albums highly, but I'm not sure that's a good argument against the idea- if it's mostly for fans then they already know that most albums are pretty well-liked and it'd be more useful to see clearly where they sit against the whole. On the other hand, it'd obviously screw around with the relationship between the ratings people input and the actual rating the system comes up with, so that despite most people giving an album, say, 3 stars, it'd probably end up closer to 2, reflecting "Collectors/fans only" instead of the generally agreed-upon "Good but non-essential". Interesting anyway. |
Ratings that result in an average near the bounds (1 and 5) should have a narrow standard deviation provided a standard rating distribution which in reality is not the case as a long tail will be present at the opposite limit (1 for high averages, 5 for small averages). More elasticity in the ratings permitted, say up to 10 instead of 5 would allow the rating to become 'more' normally distributed. Upper and lower limits on the ranking is part of the problem. Of course, you could shift highly or low rated albums' distribution curve so that the deviation is allowed upwards or downwards; for instance, shifting the curve to enable a 5% (or less) fractile to coincide with the limit.
Edited by earlyprog - February 05 2015 at 04:57
|
|
Dean
Special Collaborator
Retired Admin and Amateur Layabout
Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
|
Posted: February 05 2015 at 05:24 |
The problem there Kazza is the sample sizes for most albums are too small to have any real statistical significance. As has already been noted albums with less than say 50 votes will have a disproportionate number of 5-star ratings. That inevitably shifts the site-average upwards because the number of albums with less than 50 votes is statistically significant. However, as the article in the OP observed, voting here often tends to follow a power distribution rather than a normal distribution, because all high-rated albums must inevitably have a high proportion of 5-star votes. I suspect it is more likely to be truncated normal distribution rather than a power distribution which, as Dan's post has suggested, is due to the limited bounds of a 5-star system.
Ratings are comparative - we rate an album compared to "something" even if we are not sure what that "something" is, or whether it is the same as everyone else's "something". Normalising the site average to "100" would be to assume that everyone rated an album relative to this site average, rather than the median (i.e., presumed average) of 3-stars. Yet the distribution shapes would suggest that most people rate albums relative to 5-stars. That is, "Essential" becomes the norm.
Opeth ~ Pale Communion average=4.25 from 522 ratings:
***** |
49% |
################################################# |
**** |
32% |
################################ |
*** |
12% |
############ |
** |
4% |
#### |
* |
4% |
#### |
This distribution is not uncommon, practically every other album in the all-time Top 100 has this shape because those albums score higher than the site average.
Once you start looking at albums that score around the site average then you start to see the normal distribution appear but even that is truncated, for example:
Pendragon ~ Believe average 3.54 from 339 ratings:
***** |
20% |
#################### |
**** |
39% |
####################################### |
*** |
28% |
############################ |
** |
10% |
########## |
* |
3% |
### |
King Crimson ~ Three of a Perfect Pair average 3.24 from 823 ratings:
***** |
11% |
########### |
**** |
31% |
############################### |
*** |
41% |
######################################### |
** |
14% |
############## |
* |
3% |
### |
So what can see here is a normal distribution (that would span more than 5 bins) sliding down the 1-5 star window. As we get down to the lower averages the truncation occurs at the bottom end:
ELP ~ Love Beach average 2.06 from 485 ratings:
***** |
3% |
### |
**** |
6% |
###### |
*** |
22% |
###################### |
** |
28% |
############################ |
* |
41% |
######################################### |
However, I do not believe that having a wider set of ratings would change this scoring from 0 to 10 would still result in a disproportionate number of top scores.
Personally I prefer raw data to adjusted or manipulated data so I can see any anomalous voting (double peaks for example).
Also, I don't use the numbers to compare one album against another. Is Pale Communion that much better than Three Of A Perfect Pair? Is it better than "average"? Is it better than the "average" Opeth album even? The votes would suggest that it is as good as Blackwater Park...
Edited by Dean - February 05 2015 at 05:28
|
What?
|
|
Kazza3
Forum Senior Member
Joined: November 29 2009
Location: Australia
Status: Offline
Points: 557
|
Posted: February 06 2015 at 03:52 |
Dean wrote:
Also, I don't use the numbers to compare one album against another. Is Pale Communion that much better than Three Of A Perfect Pair? Is it better than "average"? Is it better than the "average" Opeth album even? The votes would suggest that it is as good as Blackwater Park...
|
Yeah, of course the ratings are not the be all and end all, but they are a prominent feature of the site, and even if the current system is fine and we never feel a need to change it, it's interesting to think about. Thanks very much to you and earlyprog for your answers.
|
|
tboyd1802
Forum Senior Member
Joined: June 01 2012
Location: USA
Status: Offline
Points: 898
|
Posted: February 06 2015 at 17:57 |
earlyprog wrote:
I would like to see PA have a statistics group that did this kind of research. With all these rating data available - and properly processed - PA could become a great source for presenting the evolution in peoples' prog tastes. Who knows, 20 years from now top 100 will look entirely different from now. I agree that snap shots of the ratings over time would be a step forward in the statistcis department. However, this kind of data processing it not performed in PA as far as I know. |
If we could make this happen, count me in
|
|
Roland113
Special Collaborator
Honorary Collaborator
Joined: March 30 2008
Location: Pittsburgh, PA
Status: Offline
Points: 3841
|
Posted: February 06 2015 at 18:16 |
earlyprog wrote:
A minor change im the weighting factors of various reviewer groups can have a profound effect on the overall album ranking. Over the years I have seen sudden, abrupt changes in top 100 which can only be attributed to changes in the weights. I assume and accept that the PA team has favorable weightings in the ranking process after all this is one way to thank them for their efforts. Fans and haters are found in tail regions of an album's rating distribution and can be (are?) weighted disfavorably. The rating and reviewing system of PA is not transparent on this, at least not to me, but then I haven't searched this site in an attempt to find the weighting factors.
|
Just to clarify one point, my understanding is that collaborator ratings receive a higher weight not to thank us, but to trust that we are less prone to manipulating ratings. I believe that you need to show a decent contribution to the site also a competency in your contributions to be invited into the ranks of collaborators. There is an understanding that the collaborators are working to bring legitimacy to the site as a whole.
It's not a way to thank us, it's a way to lessen the impact of the less informed, the fan boys and the haters.
Edited by Roland113 - February 06 2015 at 18:20
|
-------someone please tell him to delete this line, he looks like a noob-------
I don't have an unnatural obsession with Disney Princesses, I have a fourteen year old daughter and coping mechanisms.
|
|
bhikkhu
Special Collaborator
Honorary Collaborator
Joined: April 06 2006
Location: AČ Michigan
Status: Offline
Points: 5109
|
Posted: February 06 2015 at 21:59 |
Svetonio wrote:
^ Thank you, Kati. If you had not posted that report, I would not have learned that King Crimson was in former Prog Archives' Art Rock section |
That was a huge undertaking. At the time Art Rock was basically the sub for bands who didn't fit anywhere else. So as you can image it was huge and consisted of many strange bedfellows. So the collabs went in, sorted it out and created three new subs. Even though the ones appointed to the task took the heaviest load, it did involve all of us as some artists did need to move to other existing subs. We had just done an audit of Symphonic as well and this all happened when I was new to the role. It was quite an education.
|
|
|
Kati
Forum Senior Member
Joined: September 10 2010
Location: Earth
Status: Offline
Points: 6253
|
Posted: February 07 2015 at 11:05 |
earlyprog wrote:
earlyprog wrote:
Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II, the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.
*Perhaps I just didn't 'get' the music, but repeated listening convinced me that this emperor in fact did not wear any clothes (someone just had to say so before it was realised by all).
|
This is directed at Kati who appears to continue her CS II crusade under a different banner in this offshoot thread of another thread http://www.progarchives.com/forum/forum_posts.asp?TID=101074.
I could be mistaking?
Hugs to you Kati
|
Hugs to me? Are you for real? I did not want to come back here but read this via email. I am an individual not a band. Stop saying my name and connecting me with a band who I happen to love very much!
|
|