Forum Home Forum Home > Progressive Music Lounges > Prog Music Lounge
  New Posts New Posts RSS Feed - A general survey about P.A. and RYM
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Topic ClosedA general survey about P.A. and RYM

 Post Reply Post Reply Page  <12345 8>
Author
Message
 Rating: Topic Rating: 1 Votes, Average 5.00  Topic Search Topic Search  Topic Options Topic Options
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: January 31 2015 at 12:08
Originally posted by Dean Dean wrote:

As the article stated, early votes and reviews for an album tend to come from people who like the album. This is perfectly reasonable since if an album doesn't move you then you will be less motivated to write anything about it.

Bands with a large fan (or the opposite!) base will receive skewed ratings for a longer period. A high rating average would imply a small standard deviation (had the rating followed a normal distribution due to the upper rating limit of 5 - perhaps an upper rating limit of 10 would reduce the skewness?). Using the mean favors the fans (or haters) while the median reduces the skewness effect i.e. disfavors the extreme views. Of course, you could encompass all by stating the mode, the mean, the average, the standard deviation and the skewness (and probably more measures) of an album's rating Sick. But to what purpose? who really cares about an albums rating as long as it just indicates how 'good' it is? So in the end does it really matter if PA use the average, mean or whatever to rank an album?
Back to Top
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: January 31 2015 at 12:24
Originally posted by Dean Dean wrote:

As I have said many times, the best indicator of a bands true rating is not the average, but shape of the rating distribution (the article also used this as an indicator), which does not take into account any reviewer/collab weighting

Please define 'average': aritmetic mean, median, mode?

And what do you mean by the 'shape of the rating distribution': the actual curve showing the distribution or some derived measures of that distribution?

Of course the shape of the rating distribution is the best indicator of an album's rating but how would you present this to those who visit PA's site unless by some derived measure(s, like the 'average', standard deviation etc) ? Notwithstanding this, it would be neat with a curve showing the distribution on the site  - but wouldn't it scare of potential PA members LOL


Edited by earlyprog - January 31 2015 at 12:25
Back to Top
Dean View Drop Down
Special Collaborator
Special Collaborator
Avatar
Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
Direct Link To This Post Posted: January 31 2015 at 12:36
Originally posted by earlyprog earlyprog wrote:

Originally posted by Dean Dean wrote:

As I have said many times, the best indicator of a bands true rating is not the average, but shape of the rating distribution (the article also used this as an indicator), which does not take into account any reviewer/collab weighting

Please define 'average': aritmetic mean, median, mode?
This was covered in my posts, the average used here is the arithmetic mean.

Originally posted by earlyprog earlyprog wrote:

And what do you mean by the 'shape of the rating distribution': the actual curve showing the distribution or some derived measures of that distribution?
The actual curve.
Originally posted by earlyprog earlyprog wrote:

Of course the shape of the rating distribution is the best indicator of an album's rating but how would you present this to those who visit PA's site unless by some derived measure(s, like the 'average', standard deviation etc) ? Notwithstanding this, it would be neat with a curve showing the distribution on the site  - but wouldn't it scare of potential PA members LOL
Yeah.
What?
Back to Top
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: January 31 2015 at 12:42
Originally posted by earlyprog earlyprog wrote:

Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II,  the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.

*Perhaps I just didn't 'get' the music, but repeated listening convinced me that this emperor in fact did not wear any clothes (someone just had to say so before it was realised by all).

This is directed at Kati who appears to continue her CS II crusade under a different banner in this offshoot thread of another thread http://www.progarchives.com/forum/forum_posts.asp?TID=101074. 

I could be mistaking?

Hugs to you Kati Hug


Edited by earlyprog - January 31 2015 at 13:18
Back to Top
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: January 31 2015 at 12:44
Originally posted by Dean Dean wrote:

Originally posted by earlyprog earlyprog wrote:

Originally posted by Dean Dean wrote:

As I have said many times, the best indicator of a bands true rating is not the average, but shape of the rating distribution (the article also used this as an indicator), which does not take into account any reviewer/collab weighting

Please define 'average': aritmetic mean, median, mode?
This was covered in my posts, the average used here is the arithmetic mean.

Originally posted by earlyprog earlyprog wrote:

And what do you mean by the 'shape of the rating distribution': the actual curve showing the distribution or some derived measures of that distribution?
The actual curve.
Originally posted by earlyprog earlyprog wrote:

Of course the shape of the rating distribution is the best indicator of an album's rating but how would you present this to those who visit PA's site unless by some derived measure(s, like the 'average', standard deviation etc) ? Notwithstanding this, it would be neat with a curve showing the distribution on the site  - but wouldn't it scare of potential PA members LOL
Yeah.

Thumbs Up
Back to Top
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: January 31 2015 at 13:05
Originally posted by Dean Dean wrote:

As the article stated.....

Honestly, I don't care much about an article from 2008. This is 2015.

That's why I inserted:



(I probably should read it... sorry I haven't Embarrassed)


Edited by earlyprog - January 31 2015 at 13:25
Back to Top
Dean View Drop Down
Special Collaborator
Special Collaborator
Avatar
Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
Direct Link To This Post Posted: January 31 2015 at 13:49
Originally posted by earlyprog earlyprog wrote:

Originally posted by Dean Dean wrote:

As the article stated.....

Honestly, I don't care much about an article from 2008. This is 2015.

That's why I inserted:



(I probably should read it... sorry I haven't Embarrassed)
You sir, are a waste of my time and energy. Read the damn article or stfu.
What?
Back to Top
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: January 31 2015 at 13:57
^And this is within your socalled PA 'regulation' LOL

Edited by earlyprog - January 31 2015 at 13:58
Back to Top
Andy Webb View Drop Down
Special Collaborator
Special Collaborator
Avatar
Retired Admin

Joined: June 04 2010
Location: Terria
Status: Offline
Points: 13298
Direct Link To This Post Posted: February 02 2015 at 22:03
Originally posted by Guldbamsen Guldbamsen wrote:

^So if one is looking for say progressive electronic they will have to wade through individual countries and study biographies to see if it's there? Doesn't compute to be honest.
I do however miss the opportunity of band listing by country. If we somehow could implement that as a feature on PA, I'd be a happy bunny

Back to Top
BrufordFreak View Drop Down
Collaborator
Collaborator
Avatar
Honorary Collaborator

Joined: January 25 2008
Location: Wisconsin
Status: Offline
Points: 7951
Direct Link To This Post Posted: February 03 2015 at 09:16
Very interesting! Thank you for posting that, Katie. It would seem that PA member/contributors are a bit more judicious in their ratings--probably due to their more serious commitment to "quality" versus "popularity." This is probably why I visit PA thirty times more often than RYM.
Drew Fisher
https://progisaliveandwell.blogspot.com/
Back to Top
tboyd1802 View Drop Down
Forum Senior Member
Forum Senior Member
Avatar

Joined: June 01 2012
Location: USA
Status: Offline
Points: 898
Direct Link To This Post Posted: February 04 2015 at 19:25
Originally posted by earlyprog earlyprog wrote:

[QUOTE=Dean]

Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II,  the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.


I think this is pretty fascinating stuff - and my geek apologizes for this. My take is that this is pretty common behavior for new releases, and I would posit that this behavior may be predictable. That is, if you measure the rating at some time point close to album release, you may be able to predict the rating that album eventually settles into. To test this though we would need snap shots of the ratings over time. Do we know if these exist?
Back to Top
Kazza3 View Drop Down
Forum Senior Member
Forum Senior Member
Avatar

Joined: November 29 2009
Location: Australia
Status: Offline
Points: 557
Direct Link To This Post Posted: February 05 2015 at 04:13
Yeah, you've posted a lot of interesting information, Dean, thanks!


I wonder if it wouldn't be interesting, if the average on the site is around 3.75, to conform the ratings output to a normal distribution with an average of 3, in the way that IQ distribution is made to always average (or maybe median?) at 100, with standard deviations of set sizes on either side. It'd make it clearer where albums sit in relation to the whole- a ratings system out of 5 where most albums are rated above 4 and most have more 5 star ratings than anything else begins to be less than useful. I know it's like that because this is a site for fans of a specific genre/style who are more than likely going to rate a majority of albums highly, but I'm not sure that's a good argument against the idea- if it's mostly for fans then they already know that most albums are pretty well-liked and it'd be more useful to see clearly where they sit against the whole. On the other hand, it'd obviously screw around with the relationship between the ratings people input and the actual rating the system comes up with, so that despite most people giving an album, say, 3 stars, it'd probably end up closer to 2, reflecting "Collectors/fans only" instead of the generally agreed-upon "Good but non-essential". Interesting anyway.
Back to Top
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: February 05 2015 at 04:41
Originally posted by tboyd1802 tboyd1802 wrote:

Originally posted by earlyprog earlyprog wrote:

[QUOTE=Dean]

Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II,  the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.


I think this is pretty fascinating stuff - and my geek apologizes for this. My take is that this is pretty common behavior for new releases, and I would posit that this behavior may be predictable. That is, if you measure the rating at some time point close to album release, you may be able to predict the rating that album eventually settles into. To test this though we would need snap shots of the ratings over time. Do we know if these exist?

Interesting question you pose there, tboyd1802. I would like to see PA have a statistics group that did this kind of research. With all these rating data available - and properly processed - PA could become a great source for presenting the evolution in peoples' prog tastes. Who knows, 20 years from now top 100 will look entirely different from now. I agree that snap shots of the ratings over time would be a step forward in the statistcis department. However, this kind of data processing it not performed in PA as far as I know.
Back to Top
earlyprog View Drop Down
Collaborator
Collaborator
Avatar
Neo / PSIKE / Heavy Teams

Joined: March 05 2006
Location: .
Status: Offline
Points: 2081
Direct Link To This Post Posted: February 05 2015 at 04:56
Originally posted by Kazza3 Kazza3 wrote:

Yeah, you've posted a lot of interesting information, Dean, thanks!


I wonder if it wouldn't be interesting, if the average on the site is around 3.75, to conform the ratings output to a normal distribution with an average of 3, in the way that IQ distribution is made to always average (or maybe median?) at 100, with standard deviations of set sizes on either side. It'd make it clearer where albums sit in relation to the whole- a ratings system out of 5 where most albums are rated above 4 and most have more 5 star ratings than anything else begins to be less than useful. I know it's like that because this is a site for fans of a specific genre/style who are more than likely going to rate a majority of albums highly, but I'm not sure that's a good argument against the idea- if it's mostly for fans then they already know that most albums are pretty well-liked and it'd be more useful to see clearly where they sit against the whole. On the other hand, it'd obviously screw around with the relationship between the ratings people input and the actual rating the system comes up with, so that despite most people giving an album, say, 3 stars, it'd probably end up closer to 2, reflecting "Collectors/fans only" instead of the generally agreed-upon "Good but non-essential". Interesting anyway.

Ratings that result in an average near the bounds (1 and 5) should have a narrow standard deviation provided a standard rating distribution which in reality is not the case as a long tail will be present at the opposite limit (1 for high averages, 5 for small averages). More elasticity in the ratings permitted, say up to 10 instead of 5 would allow the rating to become 'more' normally distributed. Upper and lower limits on the ranking is part of the problem. Of course, you could shift highly or low rated albums' distribution curve so that the deviation is allowed upwards or downwards; for instance, shifting the curve to enable a 5% (or less) fractile to coincide with the limit.


Edited by earlyprog - February 05 2015 at 04:57
Back to Top
Dean View Drop Down
Special Collaborator
Special Collaborator
Avatar
Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
Direct Link To This Post Posted: February 05 2015 at 05:24
The problem there Kazza is the sample sizes for most albums are too small to have any real statistical significance. As has already been noted albums with less than say 50 votes will have a disproportionate number of 5-star ratings. That inevitably shifts the site-average upwards because the number of albums with less than 50 votes is statistically significant. However, as the article in the OP observed, voting here often tends to follow a power distribution rather than a normal distribution, because all high-rated albums must inevitably have a high proportion of 5-star votes. I suspect it is more likely to be truncated normal distribution rather than a power distribution which, as Dan's post has suggested, is due to the limited bounds of a 5-star system.

Ratings are comparative - we rate an album compared to "something" even if we are not sure what that "something" is, or whether it is the same as everyone else's "something". Normalising the site average to "100" would be to assume that everyone rated an album relative to this site average, rather than the median (i.e., presumed average) of 3-stars. Yet the distribution shapes would suggest that most people rate albums relative to 5-stars. That is, "Essential" becomes the norm.

Opeth ~ Pale Communion average=4.25 from 522 ratings:
***** 49% #################################################
**** 32% ################################
*** 12% ############
** 4% ####
* 4% ####
This distribution is not uncommon, practically every other album in the all-time Top 100 has this shape because those albums score higher than the site average.

Once you start looking at albums that score around the site average then you start to see the normal distribution appear but even that is truncated, for example:

Pendragon ~ Believe average 3.54 from 339 ratings:
***** 20% ####################
**** 39% #######################################
*** 28% ############################
** 10% ##########
* 3% ###

King Crimson ~ Three of a Perfect Pair average 3.24 from 823 ratings:
***** 11% ###########
**** 31% ###############################
*** 41% #########################################
** 14% ##############
* 3% ###

So what can see here is a normal distribution (that would span more than 5 bins) sliding down the 1-5 star window. As we get down to the lower averages the truncation occurs at the bottom end:

ELP ~ Love Beach average 2.06 from 485 ratings:
***** 3% ###
**** 6% ######
*** 22% ######################
** 28% ############################
* 41% #########################################

However, I do not believe that having a wider set of ratings would change this scoring from 0 to 10 would still result in a disproportionate number of top scores.

Personally I prefer raw data to adjusted or manipulated data so I can see any anomalous voting (double peaks for example).

Also, I don't use the numbers to compare one album against another. Is Pale Communion that much better than Three Of A Perfect Pair? Is it better than "average"? Is it better than the "average" Opeth album even? The votes would suggest that it is as good as Blackwater Park...


Edited by Dean - February 05 2015 at 05:28
What?
Back to Top
Kazza3 View Drop Down
Forum Senior Member
Forum Senior Member
Avatar

Joined: November 29 2009
Location: Australia
Status: Offline
Points: 557
Direct Link To This Post Posted: February 06 2015 at 03:52
Originally posted by Dean Dean wrote:

Also, I don't use the numbers to compare one album against another. Is Pale Communion that much better than Three Of A Perfect Pair? Is it better than "average"? Is it better than the "average" Opeth album even? The votes would suggest that it is as good as Blackwater Park...

Yeah, of course the ratings are not the be all and end all, but they are a prominent feature of the site, and even if the current system is fine and we never feel a need to change it, it's interesting to think about. Thanks very much to you and earlyprog for your answers.
 
Back to Top
tboyd1802 View Drop Down
Forum Senior Member
Forum Senior Member
Avatar

Joined: June 01 2012
Location: USA
Status: Offline
Points: 898
Direct Link To This Post Posted: February 06 2015 at 17:57
Originally posted by earlyprog earlyprog wrote:


I would like to see PA have a statistics group that did this kind of research. With all these rating data available - and properly processed - PA could become a great source for presenting the evolution in peoples' prog tastes. Who knows, 20 years from now top 100 will look entirely different from now. I agree that snap shots of the ratings over time would be a step forward in the statistcis department. However, this kind of data processing it not performed in PA as far as I know.

If we could make this happen, count me in Tongue
Back to Top
Roland113 View Drop Down
Special Collaborator
Special Collaborator
Avatar
Honorary Collaborator

Joined: March 30 2008
Location: Pittsburgh, PA
Status: Offline
Points: 3841
Direct Link To This Post Posted: February 06 2015 at 18:16
Originally posted by earlyprog earlyprog wrote:


A minor change im the weighting factors of various reviewer groups can have a profound effect on the overall album ranking. Over the years I have seen sudden, abrupt changes in top 100 which can only be attributed to changes in the weights. I assume and accept that the PA team has favorable weightings in the ranking process after all this is one way to thank them for their efforts. Fans and haters are found in tail regions of an album's rating distribution and can be (are?) weighted disfavorably. The rating and reviewing system of PA is not transparent on this, at least not to me, but then I haven't searched this site in an attempt to find the weighting factors.

 
Just to clarify one point, my understanding is that collaborator ratings receive a higher weight not to thank us, but to trust that we are less prone to manipulating ratings.  I believe that you need to show a decent contribution to the site also a competency in your contributions to be invited into the ranks of collaborators.  There is an understanding that the collaborators are working to bring legitimacy to the site as a whole.  

It's not a way to thank us, it's a way to lessen the impact of the less informed, the fan boys and the haters.


Edited by Roland113 - February 06 2015 at 18:20
-------someone please tell him to delete this line, he looks like a noob-------

I don't have an unnatural obsession with Disney Princesses, I have a fourteen year old daughter and coping mechanisms.
Back to Top
bhikkhu View Drop Down
Special Collaborator
Special Collaborator
Avatar
Honorary Collaborator

Joined: April 06 2006
Location: AČ Michigan
Status: Offline
Points: 5109
Direct Link To This Post Posted: February 06 2015 at 21:59
Originally posted by Svetonio Svetonio wrote:

^ Thank you, Kati. If you had not posted that report, I would not have learned that King Crimson was in former Prog Archives' Art Rock section Confused

That was a huge undertaking. At the time Art Rock was basically the sub for bands who didn't fit anywhere else. So as you can image it was huge and consisted of many strange bedfellows. So the collabs went in, sorted it out and created three new subs. Even though the ones appointed to the task took the heaviest load, it did involve all of us as some artists did need to move to other existing subs. We had just done an audit of Symphonic as well and this all happened when I was new to the role. It was quite an education.


Back to Top
Kati View Drop Down
Forum Senior Member
Forum Senior Member
Avatar

Joined: September 10 2010
Location: Earth
Status: Offline
Points: 6253
Direct Link To This Post Posted: February 07 2015 at 11:05
Originally posted by earlyprog earlyprog wrote:

Originally posted by earlyprog earlyprog wrote:

Edit: one can argue that fan boys and haters cancel out in the statistics but often the former tend to be a larger group than the latter. In a more recent case, dare I say Corvus Stone II,  the ranking appeared to be suspiciously high in the beginning (in my view*) but then, as always, levelled out, as the collaborators' reviews started to clock in. Here there's always the risk that the 'higher weighted reviewers' abuse their weighting to rate an album a few notches lower (or higher) in order to 'restore order' in their (PA) universe.

*Perhaps I just didn't 'get' the music, but repeated listening convinced me that this emperor in fact did not wear any clothes (someone just had to say so before it was realised by all).

This is directed at Kati who appears to continue her CS II crusade under a different banner in this offshoot thread of another thread http://www.progarchives.com/forum/forum_posts.asp?TID=101074. 

I could be mistaking?

Hugs to you Kati Hug
Hugs to me? Are you for real? I did not want to come back here but read this via email.
I am an individual not a band. Stop saying my name and connecting me with a band who I happen to love very much!
 
Back to Top
 Post Reply Post Reply Page  <12345 8>

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.121 seconds.
Donate monthly and keep PA fast-loading and ad-free forever.