Ratings: Weighting is harming Prog Archives
Printed From: Progarchives.com
Category: Site News, Newbies, Help and Improvements
Forum Name: Help us improve the site
Forum Description: Help us improve the forums, and the site as a whole
URL: http://www.progarchives.com/forum/forum_posts.asp?TID=54519
Printed Date: July 18 2025 at 19:55 Software Version: Web Wiz Forums 11.01 - http://www.webwizforums.com
Topic: Ratings: Weighting is harming Prog Archives
Posted By: Uncle Spooky
Subject: Ratings: Weighting is harming Prog Archives
Date Posted: January 05 2009 at 05:24
Firstly, apologies, but I assume this has already been discussed to death, however I have to make my belated contribution to the discussion.
A bit of background about me first.
40 years old. Been listening to Prog since about age 12 (Yes Album and Journey to the Centre of the Earth started it off.) My collection just hit it's 500th Prog artist, covering artists from the 60's through to present day. I consider myself a knowledgeable Prog fan. What I don't consider myself is somebody capable of writing coherent or even interesting reviews for this site.
Since discovering this site it has become my main portal for prog information, new releases and filling gaps in my collection based on ratings and reviews. And I have, according to my page, sumbitted 200 ratings. I also buy lots of albums which are little known on the site and always make a point of voting to bring attention to them for the benefit of others. So I'm doing my bit... or so I thought.
Yesterday I submitted a rating for http://www.progarchives.com/album.asp?id=12817 - Rocket Scientist - Revolution Road . There were only 8 votes, mine was the 9th. I checked the before and after rating as there was a 0% change, not even a 0.01 change. So I did my research and discovered the weighting system that is currently in place and understood why my rating made no difference.
Which leads me on to my concern.
The weighting system currently in place means that ANY rating that contains a PA preferential member is biased towards that person's particular view (or those people). In the case of the Rocket Scientists there are enough preferential members in those 8 previous votes that my vote may NO difference at all. Which begs the first question, what's the point of having an open voting system if the weighting is so strongly working against those of us without writing skills or the time to do the writing?
This is actually more of a problem on those albums with few votes because it only takes 1 or 2 preferential members to vote to effectively lock down the rating to their bias. And this means that, particularly if an album is not liked by a reviewer, that it will get lost amongst the higher rated albums, even if it's a geniune Prog gem.
So the first problem is that it's pointless to rate an album once a preferential member has done so. The weighting makes the contribution pointless. Which means it's actually pointless at all to rate albums on PA unless you're a preferential member.
But worse, this means that PA's charts only reflect the biases (and I'm not using that in a derogatory sense) of the preferntial members, and means the general audience are missing lots of potential gems. Becuase as soon as a prefential member votes positively, or negatively, the album will appear/disappear from the charts. (Yes, you can filter etc, but that doesn't solve the fundamental issue). This for me, seriously undermines PA's usefulness as it is simply does not reflect the actual feelings of the entire user base.
What I want is a site that reflects the entire userbase, that allows those little known gems to become known by the wider audience and a site where one feels their contribution is useful.
Right now the weighting has fundamentaly broken PA's usefulness in that regard and should be addressed (removed weighting altogether, significantly lower the weights, or simply lock the rating to non-preferential users). We would then all benefit from a more honest and open rating system rather than the current biased and pointless-to-contribute-to system that exists right now.
Thanks for listening!
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Replies:
Posted By: Angelo
Date Posted: January 05 2009 at 06:14
Hello Mark,
good to see you are so concerned with the site. However, if you do a bit of math here, you'll find that things are definitely not as extreme as you describe here. Looking at the Rocket Scientists album, you can see that it has 9 reviews and rating, including yours. 2 people voted 3 stars, the other 6 voted 4 stars. Now if I exclude your 4 stars and give all ratings equal weight, the 8 people before you would have awarded the album a 4.12. With your rating included, it would become 4.11 - a difference of 1/100 of a star, or about 0.25 percent.
The weighted system results in a 3.86 - which is just as much a 4 star as a 4.11 would've been.
Apart from that the weighting system does serve a purpose. We select collaborators and prog reviewers based on their knowledge and understanding of Prog. At the same time anyone can add ratings and reviews. Even the biggest prog nitwit in the world can (and will) give 1 star or 5 stars based on the mood of the day. By applying the weighting system, the serious reviewing efforts of our collaborators balance out these extreme, and often unmotivated, ratings. In cases like Rocket Scientists' Revolution Road this doesn't show - because there's a lot of consensus on the album anyway, on others, like Selling England By The Pound, it does - because many people vote 1 or 5 stars there blindly, just because they hate Phil Collins or want to have Close To The Edge at the top of our albums chart.
Over time, the weighting has been well thought through, and it has been redesigned quite a few times to become what it is now, taking into account just about any angle. I wouldn't worry too much about it if I were you - just do the math and you'll see how it works out.
On a completely different note: we have people who are not even native Englishs speakers who manage to squeeze out very useful reviews. Don't be afraid to describe how you feel about an album in your own words. Three sentences is sometimes enough, and it will increase the weight of your contribution. I for one would be very curious to read why you gave only 1 start to Circa's 2007... 
With the amount of albums you rated personally, you could make Prog Reviewer by adding some motivation to your ratings - and as a reward get a peek on the inside of this site as well. 
------------- http://www.iskcrocks.com" rel="nofollow - ISKC Rock Radio I stopped blogging and reviewing - so won't be handling requests. Promo's for ariplay can be sent to [email protected]
|
Posted By: Finnforest
Date Posted: January 05 2009 at 06:17
Mark, your point would have merit if there were 3 or 4 sinister, caped Collabs rating the albums to the exlcusion of everyone else. But the fact is this site has tons of "weighted" Collabs and adding many more all the time, meaning if you hang around for any length of time and *contribute* content to the site, you will soon be a weighted member and have your vote rewarded a little bit for your experience. And yeah, it has pretty much been discussed to death, as you say. It's a great system. Feel free to go back and read other threads on the matter if you'd like more information on the reasons. Thanks!
------------- https://www.youtube.com/shorts/sQD8uhpWXCw" rel="nofollow - It's a beautiful day in the neighborhood...Road Rage Edition
|
Posted By: Angelo
Date Posted: January 05 2009 at 06:24
Hmm, I guess that's the short version.
------------- http://www.iskcrocks.com" rel="nofollow - ISKC Rock Radio I stopped blogging and reviewing - so won't be handling requests. Promo's for ariplay can be sent to [email protected]
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 07:51
Angelo wrote:
good to see you are so concerned with the site. |
You're very welcome. I suppose a bit more background could be useful; I was one of the founders of the Internet Movie Database and one of my responsibilities was managing the votes database including generation of results so I have a pretty good insight into what goes on and what is required.
However, if you do a bit of math here, you'll find that things are definitely not as extreme as you describe here.Looking at the Rocket Scientists album, you can see that it has 9 reviews and rating, including yours. 2 people voted 3 stars, the other 6 voted 4 stars. Now if I exclude your 4 stars and give all ratings equal weight, the 8 people before you would have awarded the album a 4.12. With your rating included, it would become 4.11 - a difference of 1/100 of a star, or about 0.25 percent. |
The above is just one example. But what if I had genuinely voted 1? My vote would still have been lost to the weighting thus reinforcing my point that voting is pointless unless you are part of the PA in-crowd.
The weighted system results in a 3.86 - which is just as much a 4 star as a 4.11 would've been. |
Well, you're actually talking about a 6% swing caused by weighting with this example. For cash strapped prog fans trying to make purchasing decisions this could be significant. But not only that you're not giving a true representation of the album compared to it's peers; you're giving a representation of the bias (again, used non-derogatorily) of the reviewer. Sure, once you get more collaborators voting the results will tend to gravitate towards a true representative rating. But it still doesn't stop non-preferential votes being useless. And it doesn't address my concern about albums with, let's say just one negative rating from a preferential voter being lost to scrutiny from prog fans due to a poor rating.
Apart from that the weighting system does serve a purpose. We select collaborators and prog reviewers based on their knowledge and understanding of Prog. |
But that selection process is clearly lacking as I haven't been approached ;) I wonder how many other genuine, knowledgeable and honest voters slip the net?
At the same time anyone can add ratings and reviews. Even the biggest prog nitwit in the world can (and will) give 1 star or 5 stars based on the mood of the day. By applying the weighting system, the serious reviewing efforts of our collaborators balance out these extreme, and often unmotivated, ratings. In cases like Rocket Scientists' Revolution Road this doesn't show - because there's a lot of consensus on the album anyway, on others, like Selling England By The Pound, it does - because many people vote 1 or 5 stars there blindly, just because they hate Phil Collins or want to have Close To The Edge at the top of our albums chart. |
Yup, that's called vote stuffing, or lazy voting, and it's very easy to weed out.
Over time, the weighting has been well thought through, and it has been redesigned quite a few times to become what it is now, taking into account just about any angle. I wouldn't worry too much about it if I were you - just do the math and you'll see how it works out. |
What it works out to is that there is no point in anybody except preferntial voters voting. So why not close the voting system completely?
On a completely different note: we have people who are not even native Englishs speakers who manage to squeeze out very useful reviews. Don't be afraid to describe how you feel about an album in your own words. Three sentences is sometimes enough, and it will increase the weight of your contribution. |
I have contributed a review or two in the past but they were rejected. Unfortunately I don't have the time to spend editing reviews to get them up to snuff. I also offered to submit high quality scanned artwork, properly colour corrected from my collection but that didn't get anywhere. Recently I made a new group submission, and that didn't get anywhere either. There are some fundamental issues with this site. Now that sounds like I'm bitching, but I'm not. I'm just frustraed that PA could be so much better than it currently is and key to that is not setting up a division between those on the inside and those on the outside but by embracing everybody who wishes to make the site a better place.
I for one would be very curious to read why you gave only 1 start to Circa's 2007...  |
Taken from my review on LastFM:
"It's no secret that I am a http://www.last.fm/music/Yes - Yes
fan - Yes are without doubt the best band in the world ;) Always have
been, always will be! :p So it's always with enthusiasm that I grab
anything Yes-related. Unfortunately, its not always a happy experience
and one Yes-related partnership which always disappoints me is anything
with Billy Sherwood and yet I keep hoping - perhaps I should heed Yes's
lyrics "Once bitten, twice shy"... This album is a compilation of all
the worst aspects of prog - its pompousness, its excess, its lack of
subtlety. Billy is like an amateur chef who has the recipe book in
front of him but lacks any finesse in his creations and each meal
leaves you gagging and trying to get the nasty taste out of your mouth.
Perhaps it's a little unfair to point the finger so firmly in Billy's
direction, but there is a body of work out there which suggests there
is some justification... Sometimes, Billy, less is more. Less is more."
I stand by my rating on this one :D
With the amount of albums you rated personally, you could make Prog Reviewer by adding some motivation to your ratings - and as a reward get a peek on the inside of this site as well.  |
The problem there is that while I'd love my ratings to be useful, I don't want it to be because I'm a "chosen one". The site as it stands is exclusionary. I understand the arguments that weighting collaborators gives better results, but it doesn't actually stand up to scrutiny. You can filter out junk voting transparently to ensure that only the good votes get through and you then end up with a more open and honest rating system.
How about this: add a filter to the chart report pages to turn off weighting? That would allow the "best" of both worlds...
Cheers,
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 07:52
Finnforest wrote:
Mark, your point would have merit if there were 3 or 4 sinister, caped Collabs rating the albums to the exlcusion of everyone else. But the fact is this site has tons of "weighted" Collabs and adding many more all the time, meaning if you hang around for any length of time and *contribute* content to the site, you will soon be a weighted member and have your vote rewarded a little bit for your experience. And yeah, it has pretty much been discussed to death, as you say. It's a great system. Feel free to go back and read other threads on the matter if you'd like more information on the reasons. Thanks!
|
See above. I'm not making any suggestions that there is collusion going on, I'm just pointing out, in what I hope is a reasonable manner, my concerns as a normal user of the site.
Cheers,
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Dean
Date Posted: January 05 2009 at 08:03
Angelo wrote:
Looking at the Rocket Scientists album, you can see that it has 9 reviews and rating, including yours. 2 people voted 3 stars, the other 6 voted 4 stars. Now if I exclude your 4 stars and give all ratings equal weight, the 8 people before you would have awarded the album a 4.12. With your rating included, it would become 4.11 - a difference of 1/100 of a star, or about 0.25 percent.
The weighted system results in a 3.86 - which is just as much a 4 star as a 4.11 would've been.
|
I think your abacus needs rebeading - (3+3+4+4+4+4+4+4)=30 ... 30/8=3.75 and (30+4)/9=3.78 
------------- What?
|
Posted By: Mr ProgFreak
Date Posted: January 05 2009 at 08:05
UncleSpooky wrote:
Yup, that's called vote stuffing, or lazy voting, and it's very easy to weed out.
|
I'm curious - how can you easily weed out "erratic" 1/5 star votes? It's all open to opinion - if someone wants to assign 1 star to Close to the Edge or 5 stars to Love Beach, then I see no way to change these votes in a way that doesn't feel like censorship.
BTW: Saw your votes at Progfreak.com ... nice! My website really focuses on the ratings, while this one values reviews more highly. I may be slightly biased because I run PF ... but on the other hand I was a PA collaborator for many years. My suggestion: Use both websites - and others too! They each have their special advantages and disadvantages.
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Raff
Date Posted: January 05 2009 at 08:08
I think the others addressed your concerns more than adequately. However, I'd like to put my own two cents in. As Finn and Angelo already said, everyone can become a Prog Reviewer (if my memory serves me right, the requisites are illustrated in the Site Rules) provided their reviews show some quality. You are a native speaker of English (something I am not, yet I became a PR after three months here), and from what I can see you are very good at expressing yourself in writing - which would mean you could become a PR very quickly.
Anyway, we are not "preferential" members, not "chosen ones", as you put it. We are just people who put in a lot of work on behalf of this site, and all for FREE. The higher weighting given to our reviews is a form of reward for all we are doing in order to make this site such an invaluable resource to lovers of prog rock. Do you think this is such a negative thing?
|
Posted By: Easy Livin
Date Posted: January 05 2009 at 08:22
The only other thing I would add is that it is open to everyone to join the ranks of Prog reviewer/Collaborator etc. Such appointments are simply a way of rewarding those who contribute most to the site.
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 08:32
Mr ProgFreak wrote:
UncleSpooky wrote:
Yup, that's called vote stuffing, or lazy voting, and it's very easy to weed out.
|
I'm curious - how can you easily weed out "erratic" 1/5 star votes? It's all open to opinion - if someone wants to assign 1 star to Close to the Edge or 5 stars to Love Beach, then I see no way to change these votes in a way that doesn't feel like censorship.
|
It's quite easy to spot lazy voters, whose votes, while valid, may actually be against the spirit of the vote, ie to generate accurate ratings for a given album. You may then decide it's better to remove a percentage of the lazy votes to remove the "damage" they do. Or you may choose to remove them all. Or not.
But this is different to assigning weighting to collaborators.
If you look at IMDb you will see they breakdown votes and show IMDb staff as a completely separate rating and often the IMDb rating is different from the community: for example http://www.imdb.com/title/tt0468569/ratings - Dark Knight there is whole 10% difference. If IMDb employed weighting the rank of Dark Knight would be very different...
This just shows that weighting the "staff" skews ratings according to their biases.
BTW: Saw your votes at Progfreak.com ... nice! My website really focuses on the ratings, while this one values reviews more highly. I may be slightly biased because I run PF ... but on the other hand I was a PA collaborator for many years. My suggestion: Use both websites - and others too! They each have their special advantages and disadvantages.
|
Heh, yeah I need to explore your site more. I love the links to CDBaby, Amazon etc for available albums. Makes purchasing very easy as long as I trust the votes :D
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 08:38
Raff wrote:
I think the others addressed your concerns more than adequately. |
Not really. But I do appreciate the reasonable comments that are being made.
Anyway, we are not "preferential" members, not "chosen ones", as you put it. |
Well, you clearly are, according to what has been stated above.
We are just people who put in a lot of work on behalf of this site, and all for FREE. The higher weighting given to our reviews is a form of reward for all we are doing in order to make this site such an invaluable resource to lovers of prog rock. Do you think this is such a negative thing?
|
Of course, for the reasons I've already stated. Giving preferential weighting to a proportion of your community biases the ratings towards that proportion's tastes and does not give a true representation of the subject you profess to care deeply about. Would you stop contributing to PA if the weighting was removed?
I'm now more convinced than ever that weightings are harming PA and excluding the community at large.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Mr ProgFreak
Date Posted: January 05 2009 at 08:58
Uncle Spooky wrote:
Mr ProgFreak wrote:
UncleSpooky wrote:
Yup, that's called vote stuffing, or lazy voting, and it's very easy to weed out.
|
I'm curious - how can you easily weed out "erratic" 1/5 star votes? It's all open to opinion - if someone wants to assign 1 star to Close to the Edge or 5 stars to Love Beach, then I see no way to change these votes in a way that doesn't feel like censorship.
|
It's quite easy to spot lazy voters, whose votes, while valid, may actually be against the spirit of the vote, ie to generate accurate ratings for a given album. You may then decide it's better to remove a percentage of the lazy votes to remove the "damage" they do. Or you may choose to remove them all. Or not.
|
I see what you mean. Of course it's easy to spot those votes, meaning that when you see one you intuitively suspect abuse. But it's really difficult to implement automatic countermeasures, since there is no clear (mathematical) way to identify abusive votes.
At Progfreak.com I use a simple trick: Instead of simply computing the mean value the system also computes the median. The mean value of those two is then used as the result for the album. The median causes the value to "gravitate" towards the consensus. This means that if for example there are 9 5-star votes and only one 1-star vote, the result will be much closer to the 5-stars than with just using the mean:
mean: (45+1)/10 = 4.6 median: 5
result: (5 + 4.6)/2 = 4.8
UncleSpooky wrote:
Heh, yeah I need to explore your site more. I love the links to CDBaby, Amazon etc for available albums. Makes purchasing very easy as long as I trust the votes :D
Mark
|
Thanks! I'm glad that someone mentions those links ... it takes me quite some time to add them to the database. Each link to CDBaby, Amazon.com and eMusic.com is added manually.
And about trusting the votes: It depends all on whether you trust the voters ... 
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Ivan_Melgar_M
Date Posted: January 05 2009 at 10:10
Uncle Spooky wrote:
Raff wrote:
[quote]Anyway, we are not "preferential" members, not "chosen ones", as you put it. |
Well, you clearly are, according to what has been stated above.
|
Well Uncle Spooky, Raff is being modest, I saw her waking every day at 5 am to join me (She lives in Italy I live in Peru with 7 hours of difference) to discuss what bands were going to be added or moved to other genres.
Not one day, 365 days a year.
In my case; I have added 460 bios searching for bands hardly any member (inccluding me) had heard, because they released a cassette in Indonesia in 1979 or a Swiss band that only recorded an LP that was sold among the people who went to a concert in 1974.
We are here when the administrators call us to do anything for free, so we deserve some extra prerrogatives.
But that's not all, Ratings used to be weighted much more close, the site had to place two administrators (Atkingani and Easy Livin) to check review by review and notice who was trying to manipulate the site succesfuly), since we are always here, it's easy to send us a PM and tell us to remove a review or check a rating, something that doesn't happen with the hundred of lurkers that come here (Not saying you are a lurker).
So, everything has a reason Uncle Spooky.
Thanks
Iván
-------------
|
Posted By: Finnforest
Date Posted: January 05 2009 at 10:17
That's right Ivan. The site has it's reasons and Mark and other are welcome to join in and collaborate in good spirit, or not. He's dead wrong in his assertions though that our system is harming, we have a system that is the most successful anywhere in the promotion of quality music, we turn on new people every day to great prog. And we even allow guys like Mark to speak out against the site publicly.
My kudos to PA. 
------------- https://www.youtube.com/shorts/sQD8uhpWXCw" rel="nofollow - It's a beautiful day in the neighborhood...Road Rage Edition
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 11:16
Ivan_Melgar_M wrote:
Uncle Spooky wrote:
Raff wrote:
Anyway, we are not "preferential" members, not "chosen ones", as you put it. |
Well, you clearly are, according to what has been stated above.
|
Well Uncle Spooky, Raff is being modest, I saw her waking every day at 5 am to join me (She lives in Italy I live in Peru with 7 hours of difference) to discuss what bands were going to be added or moved to other genres.
Not one day, 365 days a year.
|
|
I'm certainly not dismissing peoples' efforts here. I am simply pointing out that the decision to include weighting has the effect of *excluding* your public userbase as well as undermining the accuracy of the ratings and charts by biasing the results towards collaborator's preferences.
Just because people are willing to donate their time to a cause doesn't make automatically make them experts. It simply means they are generous, industrious, etc etc.
In my case; I have added 460 bios searching for bands hardly any member (inccluding me) had heard, because they released a cassette in Indonesia in 1979 or a Swiss band that only recorded an LP that was sold among the people who went to a concert in 1974. |
That's really fantastic. People like you make this the great place it is. I absolutely salute you.
We are here when the administrators call us to do anything for free, so we deserve some extra prerrogatives. |
Preferential access to the site/ability to edit/etc for sure. Weighting votes? No. Nobody deserves that privilege. Do you honestly feel your opinion is 10 times (I think that's the current weighting) more valid than somebody elses?
But that's not all, Ratings used to be weighted much more close, the site had to place two administrators (Atkingani and Easy Livin) to check review by review and notice who was trying to manipulate the site succesfuly), since we are always here, it's easy to send us a PM and tell us to remove a review or check a rating, something that doesn't happen with the hundred of lurkers that come here (Not saying you are a lurker). |
But it *is* possible to automate the process of testing ratings for stuffing and flagging up suspicious patterns for admins to look it.
As I've said above, if the public vote is of no interest, stop accepting them.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 11:18
Finnforest wrote:
That's right Ivan. The site has it's reasons and Mark and other are welcome to join in and collaborate in good spirit, or not. He's dead wrong in his assertions though that our system is harming, we have a system that is the most successful anywhere in the promotion of quality music... |
According to whom?
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 11:28
So far, the only honest justification I've seen for weighting is as a reward to the work of collaborators. That's very incestuous and exclusionary, and as I've argued damages the accuracy of the ratings.
Do collaborators *genuinely* feel they have a more valuable opinion than the users of the site?
If not, something needs to change...
Regards,
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Queen By-Tor
Date Posted: January 05 2009 at 11:30
I'm sure this has already been mentioned, but it's also somewhat of a
precautionary method from people who join the site only to sl*g a
couple of records with 1 star review and give 5s to a couple others.
I'm also sure that this has been mentioned, but this is one of the only
'non-Amazon' sites that allows ANYONE to post a review, if this were
any other webzine you wouldn't even have that opportunity.
Should our opinions be worth more? The collaborators all started off in
your same position and proved themselves as people who the website
wants to represent opinions. I've always thought this way, so instread
of bitching about it in my early days I simply worked to be the best so
that my opinions would be valued. Now they are.
You say that the ratings aren't accurate? Please give an example of where they are not.
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 11:59
King By-Tor wrote:
Should our opinions be worth more? The collaborators all started off in
your same position and proved themselves as people who the website
wants to represent opinions. I've always thought this way, so instread
of bitching about it in my early days I simply worked to be the best so
that my opinions would be valued. Now they are. |
Yeah, that's fine it a) one has the time and inclination b) one has the money to buy albums to rewview c) one has the skill to write etc.
If an eminent scientist were to post on wiki-pedia, would there contribution be deemed less worthy simply because they didn't contribute extensively?
You say that the ratings aren't accurate? Please give an example of where they are not.
|
OK pop pickers, let's take a look at a chart topper from - http://www.progarchives.com/top-prog-albums.asp?ssubgenres=12&ssubgenres=42&ssubgenres=41&ssubgenres=28&ssubgenres=30&ssubgenres=18&ssubgenres=36&ssubgenres=4&salbumtypes=1&syears=2008&scountries=&sminratings=5&smaxratings=0&sminavgratings=3&smaxresults=100&x=77&y=6#list - 2008
Beardfish: Sleeping in Traffic Part 2
PA Rating: 4.39/5 from 43 ratings, 8 of which are collaborators and 3 were non-collabs with reviews, the rest are just ratings.
So, PA rating of 4.39/5 Basic average of 4.0/5
Or a 9% upswing due to weighting.
Remove collaborators entirely: 3.88/5
A 13% downswing from the PA rating due to removal of 8/43 ratings.
If we remove the extreme voting from non-collabs (just in case they're stuffing) the average drops to 3.56/5
How come collabs are so seemingly consistently positive about an album compared to the rest of their audience? Have they no critical faculty? Are they such fans of prog they can see no bad? Are they worried about their collaborator status being removed if they aren't positive?
Any way you look at it, with a basic average or removal of collaborators, Beardfish is very unlikely to be the best album of the year in those categories.
So, a) the information presented is currently skewed and b) potentially better albums are being penalised.
Or in otherwords, the PA community is being misled.
Does PA care about true representation at all?
Cheers,
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Queen By-Tor
Date Posted: January 05 2009 at 12:02
okay, quickly and before I read the rest of your post, you're saying
that people who have no idea how to write a good review should be able
to have the same weighting as someone who spends countless hours (338
reviews at 30 minutes to 1 hour per review aint easy) listening to
music and writing to improve his technique?
Hate to sound pretentious, but why should Joe Blow's poorly written
1-star review of an album be worth more than my well thought, well
written, worth-while semi-essay 4-star review of an album? Answer me
that.
|
Posted By: Queen By-Tor
Date Posted: January 05 2009 at 12:03
Your example is a poor one. Beardfish also won 5th place in the top albums for this year, and if you'd heard it you'd know that it was worth the rating. I gave it a 4 - oh good god no!!
|
Posted By: Queen By-Tor
Date Posted: January 05 2009 at 12:06
But you're right. The best thing that we can do is eliminate this problem. Let's go the way of the other websites and only let collaborators write for the site - if you want to write a review then please submit your resume and review, our newly appointed editors will read through and make their judgments. If you're lucky then your review should be online by the beginning of February.
That's much better now, isnt it?
|
Posted By: rushfan4
Date Posted: January 05 2009 at 12:11
A well-written review is a far more important review then an assigned number with no written explanation as to how that rating was derived. Non-collaborator reviews with words are also given more weighting than reviews without words; first, because there is some substance backing up the number rating, and second, it hopefully encourages people to write reviews instead of just clicking a number with no substance to back it up.
-------------
|
Posted By: Finnforest
Date Posted: January 05 2009 at 12:12
King By-Tor wrote:
But you're right. The best thing that we can do is eliminate this problem. Let's go the way of the other websites and only let collaborators write for the site - if you want to write a review then please submit your resume and review, our newly appointed editors will read through and make their judgments. If you're lucky then your review should be online by the beginning of February.
That's much better now, isnt it?
|
Right on Mike, this is one of the most democratic sites you will find, sometimes so democratic that our impassioned staff are at each other's throats in the pursuit of making it better. According to me.
Meanwhile, we have the same old whiner argument rolled out again by a guy who's posted here.....how many times??
------------- https://www.youtube.com/shorts/sQD8uhpWXCw" rel="nofollow - It's a beautiful day in the neighborhood...Road Rage Edition
|
Posted By: Queen By-Tor
Date Posted: January 05 2009 at 12:20
Six musicians, one exquisite albumFormed and conceived as a ''progressive'' side-project by Parallel or 90 Degrees front man
Andy Tillison, The Tangent eventually took on a life of it's own and became a full blown band. Good
thing too, because if they'd stuck to a one album project we would never have received this effort
from the band. Not As Good As The Book represents a very personal record for the lead
singer/keyboardist, with the entire album focusing on events in his life or his reactions to the
modern world. As such the album has a very sarcastic feel to it at points, but a very emotional and
attached feel at others. However, even with this mix of messages the album never gets lost. Whether
he be commenting on warfare and media reactions or telling the story of himself and a friend trying
to get into a club in Soho the album always manages to keep a familiar feel, and the songs segue
perfectly from one to the next, never jaunting the audience, but bringing them along for the ride. Though not a concept record in of itself, the album is tied together in the form of a book. That's
right, for Not As Good As The Book Andy actually published a near 100-page novel to go along
with it, harking back to the days when albums weren't simply groups of songs to be downloaded from
iTunes, but an experience which one could actually sit down and enjoy the entire way through. It
would be redundant to actually make comments on the book itself since it has nothing to do with the
actual performances on the record, but it certainly does tie all the songs together. Not to mention
that old and young proggers alike should be able to get plenty of kicks from all the prog references
in the book itself. Tillison may not be the greatest novelist to have ever lived, but he certainly
knows how to entertain. The music itself on the album is very reminiscent of progressive records of old, while maintaining a
modern sound. The opening synth riff right off the top of A Crisis In Midlife gives that away
immediately. The album is rather keyboard heavy, but what prog head is going to turn that down,
really? The keys always press the album along in a satisfactory manner, and this is likely because
(as he's said before) Tillison really builds the songs from the ground up. Start with the keyboards
and start to layer everything on top of it. Thus, everything feels like it's in the right place at
the right time, long songs like The Ethernet manage to express an idea without becoming
redundant and short rockers like the heavy and cynical instrumental Celebrity Puree and
Bat Out Of Basildon come off as excellent music instead of a hard rock song out of place on
an otherwise very progressive record. Of course all the other instruments are there when you need
them. The heavy guitars, the powerful sax and the graceful flute all appear exactly when you'd want
them to. The two disc set is split heavily between discs, and one could even go so far as to call them
separate albums. They maintain feel and consistency enough to be able to let them run together
without having the album feel long to listen to, but they really are separate entities, as suggested
by Tillison's naming of them. The first disc [A Crisis In Midlife] is full of the shorter songs.
There's a kind of light feel to these songs thanks to pieces like the somewhat humorous Lost In
London 25 Years Later (in which Andy and friend Ian find themselves face to face with a Soho
pimp), with its calm progression leading up to outright heavy parts, and the formally mentioned
Bat Out Of Basildon which is fairly uplifting thanks to lines like ''He's only as old as his
helmet, and he only got it last week''. The second disc is where prog heads will likely turn, these are the 'epics'. This disc holds a mere
two songs, and two wonderful songs they are. They heavy, brooding, and socially aware Four Egos,
One War makes wonderful use of everything The Tangent is good at, its good when it's quiet and
when it's loud it's spine chilling. It's hard to not get worked up listening to the lyrics of the song
too (I'd quote them, but there's far to many good lines to make reference to), it really makes you
want to take to the streets with 'Stop the war!' signs. At the shrill scream of ''Throwing metal at
the sky!'' it's hard not to feel a chill down your spine (no wonder they chose that line as the
title of the second disc). The second track The Full Gamut is a hugely personal song for
Tillison, reflecting on the end of his long marriage between the time of this album and the band's
previous. The emotion comes across very well in the song and Andy gets a similar effect in this song
when he starts into the ''This is not a rehearsal'' lines of the song. Surprisingly heavy for a song
of its nature, this one does not let go oft he energy started on the first track. Honestly, either one of these discs packaged on their own would have been great, putting them
together only brings the album to another level. As for the topic of the album, it's not aimed at the norm. Yes, it covers a large area (the size of
New York...?), but it's mostly for the people Tillison's age. A Crisis In Midlife indeed, and
a lot of the older audience of prog heads should appreciate that quite a bit with a lot of songs
being aimed at the majority market (16-30 year olds). However, if you're young are you going to miss
the point of the album? Not likely. Sure, we may not be able to relate to the album in completely,
but it does give a strange insight into a different generation, and that's something not a lot of
artists are able to capture. Besides, now I find myself looking forward to that 'crisis' in midlife
so I can put on this album and say to myself ''ah, that's what he meant!''. Besides, since the music
on this album is just so enjoyable it's easy to overlook not being the target audience for once.
Kind of refreshing actually. It may only be halfway through the year, but I don't think it's premature to say that this is a
must have album. It takes a bit for this album to grow on you, but after three or four spaced
out listens you might find yourself coming back to it every single day needing more. 5 Soho Jazz clubs out of
5! It doesn't get much better than this. It's going to be very, very hard to top this album for my
album of the year.
Not as good as previous works....
I am one of the few fans of this group who has been disappointed by this album?
I just had few listens but it doesn 't give me any thrill and all the songs appear very proggy but
with no special feeling generation...
Sorry to say, but the atmospheres which made A Place in The Queue a number one, have disappeared...
Uncle Spooky, should these two reviews have the same weighting - really?
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 12:25
King By-Tor wrote:
okay, quickly and before I read the rest of your post, you're saying
that people who have no idea how to write a good review should be able
to have the same weighting as someone who spends countless hours (338
reviews at 30 minutes to 1 hour per review aint easy) listening to
music and writing to improve his technique? |
Absolutely. In a democracy everybody's vote is worth the same.
Hate to sound pretentious, but why should Joe Blow's poorly written
1-star review of an album be worth more than my well thought, well
written, worth-while semi-essay 4-star review of an album? Answer me
that.
|
Because somebody's rating has nothing to do with their ability to write, or inclination to do so. That doesn't make they vote worth any less.
I didn't realise this was a website for aspiring writers to hone their craft. I thought it was a place for lovers of Progressive music?
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Slartibartfast
Date Posted: January 05 2009 at 12:27
Maybe we should just eliminate the reviews all together and have only ratings.
------------- Released date are often when it it impacted you but recorded dates are when it really happened...
|
Posted By: rushfan4
Date Posted: January 05 2009 at 12:29
There are already two websites for that: Rate Your Music, and http://www.progfreak.com - www.progfreak.com .
-------------
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 12:29
King By-Tor wrote:
Your example is a poor one. Beardfish also won 5th place in the top albums for this year, and if you'd heard it you'd know that it was worth the rating. I gave it a 4 - oh good god no!!
|
Oh please, that's insulting. I own the Beardfish albums, Sane Day, Sleeping in Traffic, Pt1 and Sleeping, Pt2.
Anyway, that was the very first example I picked and it amply demonstrated the problem of assigning weighting to collaborators.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 12:31
rushfan4 wrote:
A well-written review is a far more important review then an assigned number with no written explanation as to how that rating was derived. Non-collaborator reviews with words are also given more weighting than reviews without words; first, because there is some substance backing up the number rating, and second, it hopefully encourages people to write reviews instead of just clicking a number with no substance to back it up.
|
Writing well has absolutely nothing to do with understanding the subject you are writing about. I could cobble together a reasonable article about how to make a cake but that doesn't make me an expert.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Slartibartfast
Date Posted: January 05 2009 at 12:31
rushfan4 wrote:
There are already two websites for that: Rate Your Music, and http://www.progfreak.com - www.progfreak.com . |
I was about to mention that site.  Mr. ProgFreak would very much like you to drop by and rate like crazy. 
------------- Released date are often when it it impacted you but recorded dates are when it really happened...
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 12:32
Finnforest wrote:
Meanwhile, we have the same old whiner argument rolled out again by a guy who's posted here.....how many times?? |
Again with the insults. And again from the collaborators...
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 12:34
King By-Tor wrote:
Six musicians, one exquisite album
Not as good as previous works....
I am one of the few fans of this group who has been disappointed by this album?
I just had few listens but it doesn 't give me any thrill and all the songs appear very proggy but
with no special feeling generation...
Sorry to say, but the atmospheres which made A Place in The Queue a number one, have disappeared..
|
Uncle Spooky, should these two reviews have the same weighting - really?
[/QUOTE]
Yes, indeed. Who's to say that the expressive and effusive writer of the 5 star review isn't blinded by his love for the band and that the less able to express himself 2 star rater hasn't seen through all the hype and nailed the rating?
Get enough votes and things will shake out. Weight the votes and you'll never get the accurate picture.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: rushfan4
Date Posted: January 05 2009 at 12:35
Uncle Spooky wrote:
rushfan4 wrote:
A well-written review is a far more important review then an assigned number with no written explanation as to how that rating was derived. Non-collaborator reviews with words are also given more weighting than reviews without words; first, because there is some substance backing up the number rating, and second, it hopefully encourages people to write reviews instead of just clicking a number with no substance to back it up.
|
Writing well has absolutely nothing to do with understanding the subject you are writing about. I could cobble together a reasonable article about how to make a cake but that doesn't make me an expert.
Mark
|
But it does show what thought and effort was put into the number that was assigned. A five-star review that describes the excellence of the album as it brings on memories of Close to the Edge is going to mean a lot more to me than a five-star review that describes the excellence of the album because it brings on memories of Fantomas' Delirium Corda. The first review tells me the reader that I will be interested; the second review tells me the reader that I will not be interested. A review with no words tells me nothing.
-------------
|
Posted By: Queen By-Tor
Date Posted: January 05 2009 at 12:36
Uncle Spooky wrote:
King By-Tor wrote:
Six musicians, one exquisite album
Not as good as previous works....
I am one of the few fans of this group who has been disappointed by this album?
I just had few listens but it doesn 't give me any thrill and all the songs appear very proggy but
with no special feeling generation...
Sorry to say, but the atmospheres which made A Place in The Queue a number one, have disappeared..
|
Uncle Spooky, should these two reviews have the same weighting - really?
|
Yes, indeed. Who's to say that the expressive and effusive writer of the 5 star review isn't blinded by his love for the band and that the less able to express himself 2 star rater hasn't seen through all the hype and nailed the rating?
Get enough votes and things will shake out. Weight the votes and you'll never get the accurate picture.
Mark [/QUOTE]
Oh god.
Please - in the first review I (the writer) clearly stated why the album deserved wuch a high rating while the other person just said "pass" without saying what is wrong with the album.
|
Posted By: Dean
Date Posted: January 05 2009 at 12:37
Uncle Spooky wrote:
Finnforest wrote:
Meanwhile, we have the same old whiner argument rolled out again by a guy who's posted here.....how many times?? |
Again with the insults. And again from the collaborators...
Mark
|
The forum and the archive are two seperate items - many people who regularily review and rate albums do not spend hours of the day posting on the forum.
Personally, I would like to hear more views of non-collaborators on this issue.
------------- What?
|
Posted By: Slartibartfast
Date Posted: January 05 2009 at 12:40
By the way, I hope I haven't come across as insulting, if I have then you are just an empty headed animal food trough wiper...
------------- Released date are often when it it impacted you but recorded dates are when it really happened...
|
Posted By: Dean
Date Posted: January 05 2009 at 12:46
Uncle Spooky wrote:
OK pop pickers, let's take a look at a chart topper from - http://www.progarchives.com/top-prog-albums.asp?ssubgenres=12&ssubgenres=42&ssubgenres=41&ssubgenres=28&ssubgenres=30&ssubgenres=18&ssubgenres=36&ssubgenres=4&salbumtypes=1&syears=2008&scountries=&sminratings=5&smaxratings=0&sminavgratings=3&smaxresults=100&x=77&y=6#list - 2008
Beardfish: Sleeping in Traffic Part 2
PA Rating: 4.39/5 from 43 ratings, 8 of which are collaborators and 3 were non-collabs with reviews, the rest are just ratings.
So, PA rating of 4.39/5 Basic average of 4.0/5
Or a 9% upswing due to weighting.
Remove collaborators entirely: 3.88/5
A 13% downswing from the PA rating due to removal of 8/43 ratings.
If we remove the extreme voting from non-collabs (just in case they're stuffing) the average drops to 3.56/5
Cheers,
Mark
|
I'm more interested in hearing how we can eliminate spurious ratings - frankly, without building up a database of voting trends for each reviewer against various genre-types I cannot see how this can be implemented. Simply eliminating all 1 and 5 star reviews like you demonstrated in your example is far from accurate and skews the results even more than by weighting the results in favour of collaborators who have demonstrated (by peer review) that their ratings are an accurate reflection of their opinion and not biased towards vote-rigging.
The algorithms used have undergone several changes and improvements since they were first included into the site and we are always open to methods for improving them. Unfortunately simple un-weighted algorithms proved to be non-viable in the past due to excessive vote-rigging by spurious voters.
------------- What?
|
Posted By: rushfan4
Date Posted: January 05 2009 at 12:52
BTW, this argument does come up about once every 3 or 4 months or so. Here is a link to one of the more recent threads discussing this issue. You will probably see that the arguments for and against in this thread pretty much parallel what was said in the previous thread. http://www.progarchives.com/forum/forum_posts.asp?TID=51697&PID=3000648#3000648 - http://www.progarchives.com/forum/forum_posts.asp?TID=51697&PID=3000648#3000648
-------------
|
Posted By: Slartibartfast
Date Posted: January 05 2009 at 12:54
Well, isn't the whole weighting thing about compensating for spurious ratings anyway? I'm one of those freaks who posts on the forum and does reviews. 
------------- Released date are often when it it impacted you but recorded dates are when it really happened...
|
Posted By: Dean
Date Posted: January 05 2009 at 12:55
Yes 
------------- What?
|
Posted By: Henry Plainview
Date Posted: January 05 2009 at 13:00
Dean wrote:
Uncle Spooky wrote:
Finnforest wrote:
Meanwhile, we have the same old whiner argument rolled out again by a guy who's posted here.....how many times?? |
Again with the insults. And again from the collaborators...
Mark
|
The forum and the archive are two seperate items - many people who regularily review and rate albums do not spend hours of the day posting on the forum.
Personally, I would like to hear more views of non-collaborators on this issue. |
You're right, it's been a bit of a star pile on.
The collabs could be weighted less, I guess--although in a way, doesn't the fact that their reviews are really long weight themselves? ;-) Ratings only should be almost worthless. I must strongly object to any notion that this is a democracy. Every website is a benevolent oligarchy, or at least mostly benevolent ;-), and you can embrace your Glorious Leaders or leave.
I am more upset that the regular reviews are pushed off to the side, even when there isn't a collab review. I'm sure there were reasons, and I know it's probably not going to get changed any time soon, if ever, but it seems to me to be a bit of a slap in the face of the normal reviewer.
------------- if you own a sodastream i hate you
|
Posted By: Dean
Date Posted: January 05 2009 at 13:04
Henry Plainview wrote:
I am more upset that the regular reviews are pushed off to the side, even when there isn't a collab review. I'm sure there were reasons, and I know it's probably not going to get changed any time soon, if ever, but it seems to me to be a bit of a slap in the face of the normal reviewer. |
I'm with you on that one, but that's how the site owner has chosen to display them. As Admins and Collabs we have control over the content, not the layout.
------------- What?
|
Posted By: Henry Plainview
Date Posted: January 05 2009 at 13:10
Dean wrote:
Henry Plainview wrote:
I am more upset that the regular reviews are pushed off to the side, even when there isn't a collab review. I'm sure there were reasons, and I know it's probably not going to get changed any time soon, if ever, but it seems to me to be a bit of a slap in the face of the normal reviewer. |
I'm with you on that one, but that's how the site owner has chosen to display them. As Admins and Collabs we have control over the content, not the layout. |
Yeah, I know it's up to Max and not you, and Max is probably doesn't care about my opinion of his layout choices, but I'm not awake enough now for math, although it's not like I'm ever awake enough, so I was just contributing in the only way I could. 
Although if you're planning a mutiny, dean, you have my sword. ;-)
------------- if you own a sodastream i hate you
|
Posted By: Easy Livin
Date Posted: January 05 2009 at 13:19
mailto:M@x - M@x is aware of the shortcomings in the way the reviews are presented, and is working on improvements.
On the ratings issue, bear in mind that the weightings are just that, weightings. No ratings are excluded.
IMDB is an excellent resource and the founders etc. are to be congratulated on its development. I suspect however that it is more commercially orientated that this one, which is a genuine fans' site. Our sole objective here is the appreciation of prog and the widening of its appeal.... our two objectives here are.... (sorry wandered off into Monty Python there). The offering of incentives must be a legitimate way towards achieving that goal.
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 13:20
rushfan4 wrote:
But it does show what thought and effort was put into the number that was assigned. A five-star review that describes the excellence of the album as it brings on memories of Close to the Edge is going to mean a lot more to me than a five-star review that describes the excellence of the album because it brings on memories of Fantomas' Delirium Corda. The first review tells me the reader that I will be interested; the second review tells me the reader that I will not be interested. A review with no words tells me nothing. |
It shows thought and effort was put into the *writing*; that has nothing to do with understanding what you are writing about...
Look at David Ike - writes thousands of pages about lizards circling the Earth and running our society. Is there an ounce of truth in what he writes? I'm willing to bet not.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 13:22
King By-Tor wrote:
Oh god.
Please - in the first review I (the writer) clearly stated why the album deserved wuch a high rating while the other person just said "pass" without saying what is wrong with the album.
|
Irrelevant.
I can rate an album honestly and accurately without saying a single word. Everything I need to say is there in the rating.
If people disagree, let their votes do the talking, not their writing skills.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 13:25
Slartibartfast wrote:
Well, isn't the whole weighting thing about compensating for spurious ratings anyway? |
But as I've argued weighting is not the correct way to do it because the ratings skew towards weighted raters' preferences and obscure the true rating.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Henry Plainview
Date Posted: January 05 2009 at 13:30
Easy Livin wrote:
mailto:M@x - M@x is aware of the shortcomings in the way the reviews are presented, and is working on improvements. |
I'm glad to hear that!
IMDB is an excellent resource and the founders etc. are to be congratulated on its development. I suspect however that it is more commercially orientated that this one, which is a genuine fans' site. Our sole objective here is the appreciation of prog and the widening of its appeal.... our two objectives here are.... (sorry wandered off into Monty Python there). The offering of incentives must be a legitimate way towards achieving that goal. |
That is true. We also have a much better community than IMDB. ;-)
Uncle Spooky wrote:
King By-Tor wrote:
Oh god.
Please - in the first review I (the writer) clearly stated why the album deserved wuch a high rating while the other person just said "pass" without saying what is wrong with the album.
|
Irrelevant.
I can rate an album honestly and accurately without saying a single word. Everything I need to say is there in the rating.
If people disagree, let their votes do the talking, not their writing skills.
Mark
|
Not at all, because your rating is meaningless if nobody else knows why you think that. ------------- if you own a sodastream i hate you
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 13:35
Easy Livin wrote:
mailto:M@x - M@x is aware of the shortcomings in the way the reviews are presented, and is working on improvements.
On the ratings issue, bear in mind that the weightings are just that, weightings. No ratings are excluded.
|
And weightings are used to smother what are seen as sabotage votes? But in the process smother valid votes too. Surely much better to pro-actively work to remove the sabotage and allow all votes their full impact? That's the only route to accuracy.
Our sole objective here is the appreciation of prog and the widening of its appeal.... our two objectives here are.... (sorry wandered off into Monty Python there).
And yet, because of the weightings what PA is actually doing is pushing the collaborators' view as to what is good Prog at the expense of the general concensus. This is a fundamental problem. When I display a chart, eg best of 2008, I don't want to see the opinions of a few collaborators but the genuine concensus view of the Prog loving community.
[quote]The offering of incentives must be a legitimate way towards achieving that goal. |
Sure, but at what expense?
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Ivan_Melgar_M
Date Posted: January 05 2009 at 13:38
Uncle Spooky wrote:
Do you honestly feel your opinion is 10 times (I think that's the current weighting) more valid than somebody elses?
[ |
In first place, that's not how it works:
Ratings without reviews have weight of 1: I believe you won't even ask why, we don't know if the person who rated it, even heard the album or just rated because he/she loves or hates the band. We've seen people who come here and rate (for example) all Pink Floyd albums with 5 stars, then they go and rate all Yes and Genesis albums with 1 star......Do you think Close to the Edge (I don't like Yes) is worth 1 star in any universe? Or Foxtrot? I guess you won't complain about this.
Ratings with reviews: Are worth 3 times, because the person has taken the time to listen the album and rate it, we can know if the person at least heard the album. But still it's not secure, when Dream Theater released Octavarium, we received like 50 reviews from non usual members,all of them described the album as the best thing since sliced bread...Well not my taste but it's OK. But a Collaborator noticed this guys were rating "Elements of Persuasion" by LaBrie, as if it was Octavarium.......So we must take precautions
Reviews by Collaborators weight 10, which means 3 1/3 times the review of a non Collaborator: Why? Because we know them, this are persons who have earned a place writing at least 30 coherent reviews before becoming Collaborators or worked on a team checking an analyzing albums that I'm sure you (and probably I) haven't heard in our lives, so their knowledge is improved.
3.3333 times is not an abuse, it's a rational precaution we take, BTW a rating without review by a Collaborator or even by mailto:M@x - M@x is worth 1, because that's a flat average.
Now lets go to the central piece, this is not a Democracy, but we are by far the most democratic place of the big Prog ones, lets see the biggest:
GEPR: I write reviews for them, but you have to send then to Fred Trafton, and in the best scenario you have to wait 3 months, I sent a review of the Peruvian band "Laghonia" several months ago,. but the last update was March 27, 2008. That's not all, he evaluates the review and adds it if he decides it's good enough...Here in Prog Archives you see your review 5 seconds after you posted it and nobody decides if it's good or not, we just add it.
Progressive Ears: Even though they are an excellent Forum, their database is very small, and the reviews have to be approved also.
Proggnosis: Another good site,. as far as I know only the staff writes reviews
DPRP: You have to be evaluated, make your review according to their standards and sign a compromise of exclusivity with them....For God's sake, it's for free and you have to be exclusive. And that's not all, they have just published:
Many thanks to all who responded for our call for New Reviewers. Due to the good response, we've now closed the application page and will be in touch with those who applied soon.
Bob Mulvey Reviews Editor
http://www.dprp.net/vacancies/ - http://www.dprp.net/vacancies/
So not only you have to apply, be accepted, wait, etc, it's their system, I respect but because it works for them, but here you are allowed to publish as many reviews as you want without any limit and you still complain?
Iván
-------------
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 13:39
Henry Plainview wrote:
Not at all, because your rating is meaningless if nobody else knows why you think that. |
So every chart view on the site is worthless? If so, why include them?
I grant you that reviews can be very helpful to offer insight into the album, to offer commentary on it etc, but when you just want a list of the top 50 best symphonic prog albums of 2008 so that you can fill up your shopping cart with the best prog has to offer this year wouldn't you rather it was based on accurate and honest ratings (so that you're not about to waste your money) and not ones distorted by some misguided conception that those who write reviews are better able to accurately rate albums than those who do not?
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 13:40
Henry Plainview wrote:
That is true. We also have a much better community than IMDB. ;-) |
Hehe, well that's a topic for another time perhaps!
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 05 2009 at 13:44
Ivan_Melgar_M wrote:
[QUOTE=Uncle Spooky] Now lets go to the central piece, this is not a Democracy, but we are by far the most democratic place of the big Prog ones, lets see the biggest: |
But why not go that one step further, blaze the trail and become the most democratic Prog site around? ;)
There's really nothing to lose, and everything to gain.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: rushfan4
Date Posted: January 05 2009 at 13:47
It is kind of funny. I use IMDB all of the time for looking up the names of various actors and actresses in movies or TV shows that I have seen when I think that I know them but I can't place where I know them from. I also like using it as a resource for finding out other movies and TV shows that favorite actors and actresses have appeared in so that I can later track down those movies or shows to watch. For all of the times that I have used IMDB as a resource I had no idea that there was a system in place for anyone to rate the movies. 
-------------
|
Posted By: Ivan_Melgar_M
Date Posted: January 05 2009 at 13:48
We tried it Uncle Spooky and didn't worked, it was chaotic, two administrators had to work 24/7 in precaution of manipulation, and a lot of reviews were deleted, today hardly a review is deleted unless it's insulting.
If we are NŞ 1 place in the web for Progressive Rock pages.....is because we are doing things well.
BTW: Music is not a democracy, if it was we would be listenin Gangsta Rap because millions would chose them instead of us.
Iván
-------------
|
Posted By: Dean
Date Posted: January 05 2009 at 13:50
Uncle Spooky wrote:
Slartibartfast wrote:
Well, isn't the whole weighting thing about compensating for spurious ratings anyway? |
But as I've argued weighting is not the correct way to do it because the ratings skew towards weighted raters' preferences and obscure the true rating.
Mark
|
I'm not convinced you have demonstrated that with any degree of confidence; without analysing the content of the Collabs review with the numerical rating value, and taken that in relation to his other ratings and reviews. Even then it is difficult (nay impossible) to measure any bias or preference - for example we all know of a well known Collab who dislikes Dream Theater with a passion - yet he manages to review and rate Scenes From a Memory pt 2 with 4-stars - is that an indication of bias or personal preference? No. Compare that to some one who gives every DT album 1-star and every King Crimson album 5-star ratings-only. With ratings-only we cannot gauge the reviewers intent, only record their final vote, unless we fully analyse each and every rating they make across the board..
Weighting skews the results towards the Collaborators votes because that is what it is designed to do - and they are only effective on albums with a low total number of ratings to prevent nefarious people hyping-up an album - the weighting on albums with hundreds of ratings is less relevant.
------------- What?
|
Posted By: Padraic
Date Posted: January 05 2009 at 13:55
Plus any skew towards "collabs preferences" doesn't mean much - the tastes and preferences of the all the collaborators on this site span the prog spectrum.
|
Posted By: Slartibartfast
Date Posted: January 05 2009 at 14:10
Posted By: Finnforest
Date Posted: January 05 2009 at 14:53
Mark, forgive the "whiner" comments, apologies for that.
It's just that we go through some of these same issues over and over again. Folks who have contributed little or nothing to the site come by and tell Collabs, who have worked unbelievably hard over many long hours, that what we are doing is wrong and worth little. Ya know, that is what the real "insult" here is, my friend. Give the site a little credit please for the good it does rather than harping about your perceived injustices. And that's just what they are, perceived by you. There is nothing wrong with the way Max has set up his site for the majority of users who frequent the place and LOVE it.
I'll extract myself at this point and let the calmer heads take it from here.  Cheers!
------------- https://www.youtube.com/shorts/sQD8uhpWXCw" rel="nofollow - It's a beautiful day in the neighborhood...Road Rage Edition
|
Posted By: Windhawk
Date Posted: January 05 2009 at 15:54
Hmmm, this is a topic that really haven't engaged me previously - I couldn't care less for how my ratings are weighted or not. Never thought about it before I became a collab, never thought about it after either.
The weighted system sees to it that releases with few ratings will be skewed in favor of the collabs. And if I understand the system correctly; only if they write a review.
And if I have understood that one correctly, I don't see much of a problem.
------------- Websites I work with:
http://www.progressor.net http://www.houseofprog.com
My profile on Mixcloud: https://www.mixcloud.com/haukevind/
|
Posted By: popeyethecat
Date Posted: January 05 2009 at 16:22
Sorry if I'm not contributing much to the debate here but despite not being a Collaborator I personally think that the weighting is a good idea! I would not like the idea of my votes having the same bearing as those of someone far more experienced when it comes to evaluating and rating music. As long as people are picked properly I think it's a great idea
-------------
|
Posted By: Logan
Date Posted: January 05 2009 at 17:58
Weightings have particular value due to the potential for abuse, I think -- particularly when it comes to devaluing ratings without reviews. And we once had a senior member who rated a huge number of albums in a short time based on fuzzy memory. When one writes a review, I think it would be normal to put more effort into thinking about the rating.
One idea might be -- only if M@X was interested of course and I see no reason why he would be would be -- to offer preferences for the way we want to see the ratings: a) the way it is now b) collab and non-collab reviewers given equal weight with ratings without reviews having the same low-weight c) all ratings given equal weight -- with or without reviews. However, I suspect that that would lead to more people trying to bump up and down albums with ratings only.
Incidentally, as an almost non-reviewer I don't think I should have greater weight, but then I am conservative with ratings and I would rather not rate/ review albums already reviewed if I were to do them. Really, what I want is the ability to review without rating at all since I have some qualms with rating albums. Truth be told, I rarely read reviews at this site, but the average rating of artists' albums often has got me into looking at album reviews (if there are reviews -- I tend to research quite obscure albums).
Where I do find ratings most useful with or without reviews is for checking out an individual's reviews list. There are some prolific raters, who share very similar tastes to my own, whose lists I've found very useful as a reference/ guide.
------------- Watching while most appreciating a sunset in the moment need not diminish all the glorious sunsets I have observed before. It can be much like that with music for me.
|
Posted By: Angelo
Date Posted: January 05 2009 at 18:01
Uncle Spooky wrote:
Henry Plainview wrote:
Not at all, because your rating is meaningless if nobody else knows why you think that. |
So every chart view on the site is worthless? If so, why include them?
I grant you that reviews can be very helpful to offer insight into the album, to offer commentary on it etc, but when you just want a list of the top 50 best symphonic prog albums of 2008 so that you can fill up your shopping cart with the best prog has to offer this year wouldn't you rather it was based on accurate and honest ratings (so that you're not about to waste your money) and not ones distorted by some misguided conception that those who write reviews are better able to accurately rate albums than those who do not?
Mark |
If there was a system to guarantee accurat and honest ratings on a site with as many visitors and raters as this one, you might have had a point here. As pointed out earlier, but sadly misinterpreted in this thread: the written reviews explain the rating given - and reviews written by those who have proven insight rather than literary skills are weighted more, because they become the collaborators. We even used to have a collaborator who spent years writing reviews in his native language, pull them through babelfish or google translator and then post the automatic english translation. We only removed the ones that were completely 'lost in translation' and those he happily rewrote. The guy had virtually NO English writing skills, but his little finger knows more about prog than you and I together. Thus - in the end, the weighting serves two purposes: minimze the effect of lazy voting, and honour what prog fans apparently need: a knowledgeable explanation of a rating, while still allowing disagreement amongst reviewers. After all - it's the end result we are talking about.
Then again, the point is moot I guess - this discussion has occured many times, and so far we only ended up running in circles like in this one.
EDIT: final note - if you read the descriptions of the star system we have, only the whole stars have a real meaning. Each and every example Ive seen in this thread rounds to the same star regardless of the approach taken, so in the end - if M@X had decided not to show the decimals, we never would've had this discussion.
------------- http://www.iskcrocks.com" rel="nofollow - ISKC Rock Radio I stopped blogging and reviewing - so won't be handling requests. Promo's for ariplay can be sent to [email protected]
|
Posted By: debrewguy
Date Posted: January 05 2009 at 18:35
Damn, I went and reviewed all those Ange & Klaatu albums before I became a collab ! Why wasn't I told my opinion didn't matter ? Oh well, back to the world juniors hockey final game ...
------------- "Here I am talking to some of the smartest people in the world and I didn't even notice,” Lieutenant Columbo, episode The Bye-Bye Sky-High I.Q. Murder Case.
|
Posted By: kenethlevine
Date Posted: January 05 2009 at 20:49
I guess the argument being advanced here could be applied much more effectively against music critics for newspapers, books or websites where only a small number of people get to assess the value of a recording, and indeed influence its success. At least here we provide a broad range of reviewers, and visitors can find their favourites - those they trust or whose taste more closely matches their own - and apply their own personal weighting system. So, while I do understand the argument for equal weighting, and ultimately believe that the current weighting system is more sensible, I think that individual reviews provide value to the visitor, who need not care whether a particular review was done by a site reviewer or a newbie.
|
Posted By: RaúlGuate
Date Posted: January 05 2009 at 21:46
I'm giving my opinion from the perspective of a non-collab:
I don't if the problem can be fixed with the mathematics of the ratings, but I felt that my reviews became trivial when the layout was changed.
I think what UncleSpooky's point comes from the greater context of the discontent of being a non-collab. I tend to lurk around the forums a lot and I've seen that the site indeed isn't at all a democracy, but has the structure of a business (because it is actually a business). Max & ProgLucky are the owners. They are the ones that do the most work (because they have invested their money and are expecting revenue). Admins are like "big managers". The "middle managers" are the collabs and reviewers. Because their work is free, their retribution (salary) is in the weighings.
The discontent in the non-collabs lies in the gap of "unretributed" work that one has to do to become a collaborator. Some non-collabs do some work, but it isn't enough to get retribution... Guess we should form a union  
Seriously, though, I think everything is ok as long as everyone is having fun in this process. We shouldn't take it too seriusly. But personally I'd like to be the breach between collabs and non-collabs gradually diminished. (By bringing the old layout, so I can resume the reviewing! )
|
Posted By: Petrovsk Mizinski
Date Posted: January 05 2009 at 23:08
Finnforest wrote:
King By-Tor wrote:
But you're right. The best thing that we can do is eliminate this problem. Let's go the way of the other websites and only let collaborators write for the site - if you want to write a review then please submit your resume and review, our newly appointed editors will read through and make their judgments. If you're lucky then your review should be online by the beginning of February.
That's much better now, isnt it?
|
Right on Mike, this is one of the most democratic sites you will find, sometimes so democratic that our impassioned staff are at each other's throats in the pursuit of making it better. According to me.
|
Not wanting to sound like an ass, but compared to some of the other forum boards/sites I go, which have the various levels/positions of people, PA isn't anywhere near as democratic as some of the other sites I've seen to be entirely honest. I'm not saying this as an attack on PA, it's just how I feel. Personally, when I was just a senior member, I was fine with that because you realize there are people in place because they know how to get the sh*t done. And now, I'm a collab, which as the last post in this thread before be pointed out, is somewhere in between a normal member and the webmasters, and I'm happy with where I am, because I understand the stuff I need to do, know what I have to do in my given role and let others who know better and have greater expertise in their area do their stuff without people really getting in their way. You know, it would be cool and all to own this site, be the webmaster and stuff, but geez, I wouldn't have a clue what to do anyway, that's why we have M@X and Proglucky.
-------------
|
Posted By: cobb2
Date Posted: January 06 2009 at 01:39
Dean wrote:
The forum and the archive are two seperate items - many people who regularily review and rate albums do not spend hours of the day posting on the forum.
Personally, I would like to hear more views of non-collaborators on this issue. |
I frequent the forum (don't take any notice of the low post rating, I am hopeless at remembering passwords). To tell the truth I don't even go to the archives sections. Ever since I have been here there has always been bias involved in the rating system, under different guises and this has always turned me off delving further. After 40 years of listening to prog I am quite capable of recognising what I like when I hear it, without needing a rating (flawed) rating system to point me in the right direction. What I do need is a site that has easily accessible new release info and this site doesn't. So for this I use another.
But I do love the forum...
|
Posted By: Failcore
Date Posted: January 06 2009 at 02:13
I personally think the weighting system should just be less drastic, but I've already mentioned this before and been shot down. What I think could be done is only ranked members can vote on albums newer than 6 months old. Most of the reactionary bs is at release time, right? Then after 6 months, open it to everyone, you could still have a 2:1 ratio IMO and be fairly democratic, but the aforementioned 3.33:1 is waaaay too much.I however, like the less waiting of ratings w/o reviews. If someone writes a few paragraphs about an album, you know they have to give at least 1/2 of a crap. Above all tho, even if I'm not listened to, I encourage the owners of this site to experiment and monkey with things. Btw, Beardfish owns.
|
Posted By: Uncle Spooky
Date Posted: January 06 2009 at 02:50
Dean wrote:
Uncle Spooky wrote:
Weighting skews the results towards the Collaborators votes because that is what it is designed to do |
Yes, that is apparent.
[quote] and they are only effective on albums with a low total number of
ratings to prevent nefarious people hyping-up an album - the weighting
on albums with hundreds of ratings is less relevant. |
Define "low". As I showed with the Beardfish album, with 43 votes, there's a large skew caused by weighting towards collabs. It's even worse with albums with low votes that contain collab reviews.
And as I've already said it's possible to automatically monitor voting patterns for sabotage and flag up suspect entries. Sabotage voters can be transparently blocked, or percentages of votes can be dropped for all entries to account for sabotage and so on. It doesn't take 24/7 effort once implemented.
Skewing for the in-crowd is parochial and disingenuous.
And if non-collab ratings are distrusted so much, why allow them? It seems to me you should only allow ratings with reviews, that way you get to decide who is worthy?
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: someone_else
Date Posted: January 06 2009 at 02:51
The weighting system is not that bad. I only hope that it can deal with sometimes badly written templates, which become quite annoying after seeing nine similar "reviews" on the front page.
|
Posted By: Uncle Spooky
Date Posted: January 06 2009 at 02:53
NaturalScience wrote:
Plus any skew towards "collabs preferences" doesn't mean much - the tastes and preferences of the all the collaborators on this site span the prog spectrum.
|
To a certain extent, but as my Beardfish experiment showed the weighting is skewing the ratings. How many genuinely better albums are being lost in the mix because weightings are artificially elevating lesser albums? Or even pushing good albums down should a collab take a dislike to it?
I really don't see how a site that claims to care about Prog can sanction such moves?
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Atavachron
Date Posted: January 06 2009 at 03:00
who the bloody hell cares about "ratings".. I mean c'mon, the important things are the information via review, samples and links to other sources, and an excellent forum for communication
|
Posted By: Uncle Spooky
Date Posted: January 06 2009 at 03:05
Finnforest wrote:
Mark, forgive the "whiner" comments, apologies for that. |
No problem.
It's just that we go through some of these same issues over and over again. Folks who have contributed little or nothing to the site come by and tell Collabs, who have worked unbelievably hard over many long hours, that what we are doing is wrong and worth little. |
Again, I understand. I've done my time on the front line and appreciate how stressful it can be and how easy it is to feel under appreciated, etc but the truth is if the site wasn't so important I wouldn't be here putting my case.
Ya know, that is what the real "insult" here is, my friend. Give the site a little credit please for the good it does rather than harping about your perceived injustices. |
Heh, and now you're being insulting again  That anybody cares enough to stand up in front of you guys and risk the flak should be taken as a compliment... Look, I am being totally reasonable in my arguments. I've not presented anything in a whiny, aggressive, lame manner. Just reasoned observation. I appreciate that this has possibly been discussed before, but that shouldn't stop people speaking their mind in a reasonable way when something is perceived as wrong. USA invades IRAQ? It's happening as planned, why should anybody stand up and speak out against it? Women don't get to vote? Whatever? etc. OK, these are slightly different extremes, but the principle remains. If a reasonable answer is given then people shut up and go away, but so far the only honest answer I've seen is that the system is designed to give collaborators weighting. The inference being that non-collabs are at best mistrusted, at worst seen as saboteurs with hidden agendas. I find this unacceptable and have presented an argument for why. Discussion is nothing to be feared unless the answers are painful to give... Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Easy Livin
Date Posted: January 06 2009 at 03:45
This debate has probably gone about as far as it can go without simply becoming a yes it is/no it isn't exchange. Unless there are any further points which have not yet been made, it is probably time to agree to differ and move on.
|
Posted By: Uncle Spooky
Date Posted: January 06 2009 at 03:48
Angelo wrote:
If there was a system to guarantee accurat and honest ratings on a site with as many visitors and raters as this one, you might have had a point here. |
Well, I have already highlighted IMDb as a model? They use no weighting (or didn't when I was in charge).
Thus - in the end, the weighting serves two purposes: minimze the effect of lazy voting, and honour what prog fans apparently need: a knowledgeable explanation of a rating, while still allowing disagreement amongst reviewers. After all - it's the end result we are talking about. |
Well, here you are talking about two different things? Ratings and reviews are separate entities with different information to convey...
Then again, the point is moot I guess - this discussion has occured many times, and so far we only ended up running in circles like in this one. |
Indeed.
EDIT: final note - if you read the descriptions of the star system we have, only the whole stars have a real meaning. Each and every example Ive seen in this thread rounds to the same star regardless of the approach taken, so in the end - if M@X had decided not to show the decimals, we never would've had this discussion.
|
But of course decimals must be used if charts are to be meaningful. And ratings have to be accurate if charts are to be meaningful...
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 06 2009 at 03:55
Easy Livin wrote:
This debate has probably gone about as far as it can go without simply becoming a yes it is/no it isn't exchange. Unless there are any further points which have not yet been made, it is probably time to agree to differ and move on. |
Fair enough.
I'd like to thank everybody for their input and insights and for taking the time to discuss this issue.
My suggestions:
Drop Weighting, or at least significantly lower the weightings to remove the gross skewing that is currently occurring.
Implement vote stuffing code to weed out sabotage. Smothering votes with weighting doesn't actually solve the problem. It needs to be tackled properly.
Failing that, some possible options:
Do as IMDb does and display the basic averages, non-weighted ratings, non-collab ratings etc. There's nothing to be scared of here and the more ways you can present information the more interesting it is and the more valuable the site becomes.
Offer a filter on the chart pages to remove weighting from the result sets. Again, the more ways people can sift information, the more interesting the site becomes.
Embrace the userbase! Don't become so insular that all non-collabs are viewed with suspicion. That will lead to PA's downfall.
Keep on Proggin'!
Regards,
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Mr ProgFreak
Date Posted: January 06 2009 at 04:19
I found this on the IMDB page:
"IMDb publishes weighted vote averages rather than raw data averages.
Various filters are applied to the raw data in order to eliminate and
reduce attempts at 'vote stuffing' by individuals more interested in changing
the current rating of a movie than giving their true opinion of it.
The exact methods we use will not be disclosed. This should ensure that
the policy remains effective. The result is a more accurate vote average." I wonder what they are doing. As far as I'm concerned, it's censorship - for example, they might remove votes which differ a lot from the established average, or lessen the weight of people who constantly submit ratings which differ from consensus. If there was a method to reliably detect abuse, they would *not* need to keep it secret. BTW: I like IMDB and will continue to use it - my point is that whenever a website which accepts ratings does something to prevent abuse, it will also offend some members. It's a compromise between having a totally fair and democratic system with abusive ratings in it which distort the averages, and having a totally isolated system where only known members are allowed to contribute.
Maybe for the archives it would suffice for M@x to also show the unweighted averages, and be a little more transparent about the algorithms used.
|
Posted By: Dean
Date Posted: January 06 2009 at 04:24
Uncle Spooky wrote:
Dean wrote:
Uncle Spooky wrote:
Weighting skews the results towards the Collaborators votes because that is what it is designed to do |
Yes, that is apparent. |
glad we sorted that out then 
Uncle Spooky wrote:
[quote] and they are only effective on albums with a low total number of ratings to prevent nefarious people hyping-up an album - the weighting on albums with hundreds of ratings is less relevant. |
Define "low". As I showed with the Beardfish album, with 43 votes, there's a large skew caused by weighting towards collabs. It's even worse with albums with low votes that contain collab reviews. |
Low is something below an average - I have no I care what the average is, low is not 900 votes, average is probably 100 or so. I don't accept that weighting giving a 9% difference on 43 votes is "a large skew" - what you don't know (and cannot tell) is how skewed the results are without weighting.
I don't know about you, but when people give any album a rating that is below the average for that album I don't automatically see sabotage, but someone who simply didn't like it, so I'd like to know what they didn't like about it.
Uncle Spooky wrote:
And as I've already said it's possible to automatically monitor voting patterns for sabotage and flag up suspect entries. Sabotage voters can be transparently blocked, or percentages of votes can be dropped for all entries to account for sabotage and so on. It doesn't take 24/7 effort once implemented.
|
You've already said it is possible to automatically monitor voting patterns for sabotage - I've asked for details on how this can be done - on 21,000 members - considering that a lot of "sabotage" is done using multiple accounts with proxy IP addresses or dynamically allocated IP address - it is difficult enough keeping track of people who set up multiple accounts with fixed IP addresses. Beardfish was a poor example - look at Pendragon. I know that Pure has been sabotaged and I'm fairly confident that Sleeping In Traffic has not - please examine the ratings for these two albums and tell me where the sabotage is. I can assure you that simple analysis of voting-trends will not find or reveal it.
Uncle Spooky wrote:
Skewing for the in-crowd is parochial and disingenuous.
|
I really don't get the "in-crowd" and the "parochial and disingenuous" jibes. But I guess I'm on the inside looking out.
Uncle Spooky wrote:
And if non-collab ratings are distrusted so much, why allow them? It seems to me you should only allow ratings with reviews, that way you get to decide who is worthy?
Mark
|
Because this is a multinational site where we insist the reviews are written in English - ratings-only allows non-English speakers the opportunity to share in the rating of their favourite Prog albums. It would be parochial (though not disingenuous) for us exclude these voters.
Unfortunately that opens up the site for abuse by people who want to hype their favourites, bash they're pet-hates and attempt to manipulate the Top-XX charts. We have seen this enough times to know it happens on a regular basis, and not just for popular or contentious albums.
Regretably that penalises honest rater-onlys such as yourself.
Of course the weighting system does not prevent people who can write a mere 100 words on a particular release from abusing the system, but it is more difficult to do that consistently and not get caught-out.
------------- What?
|
Posted By: Finnforest
Date Posted: January 06 2009 at 05:16
Uncle Spooky wrote:
Finnforest wrote:
Mark, forgive the "whiner" comments, apologies for that. |
No problem.
It's just that we go through some of these same issues over and over again. Folks who have contributed little or nothing to the site come by and tell Collabs, who have worked unbelievably hard over many long hours, that what we are doing is wrong and worth little. |
Again, I understand. I've done my time on the front line and appreciate how stressful it can be and how easy it is to feel under appreciated, etc but the truth is if the site wasn't so important I wouldn't be here putting my case.
Ya know, that is what the real "insult" here is, my friend. Give the site a little credit please for the good it does rather than harping about your perceived injustices. |
Heh, and now you're being insulting again  That anybody cares enough to stand up in front of you guys and risk the flak should be taken as a compliment... Look, I am being totally reasonable in my arguments. I've not presented anything in a whiny, aggressive, lame manner. Just reasoned observation. I appreciate that this has possibly been discussed before, but that shouldn't stop people speaking their mind in a reasonable way when something is perceived as wrong. USA invades IRAQ? It's happening as planned, why should anybody stand up and speak out against it? Women don't get to vote? Whatever? etc. OK, these are slightly different extremes, but the principle remains. If a reasonable answer is given then people shut up and go away, but so far the only honest answer I've seen is that the system is designed to give collaborators weighting. The inference being that non-collabs are at best mistrusted, at worst seen as saboteurs with hidden agendas. I find this unacceptable and have presented an argument for why. Discussion is nothing to be feared unless the answers are painful to give... Mark |
No Mark, there is no insult here. You just don't like the fact that not everyone buys your theory that PA is going to crash and burn if we don't follow your advice. To the contrary, the site is doing quite well and the reasons for Max's set-up are solid. But don't play the victim today--i didn't "insult" you in this post. The injustice as you see it is a perception issue, an opinion. Not a fact. Pointing that out after 5 pages of your argument does not merit the "black eye" emoticon. You've been treated well here by all despite my defensiveness over the work of our Collabs. I've seen no one truly attack you, I wonder if that would be the case if you waltzed into PE or similar prog site and proclaimed their ratings useless. Thanks.
------------- https://www.youtube.com/shorts/sQD8uhpWXCw" rel="nofollow - It's a beautiful day in the neighborhood...Road Rage Edition
|
Posted By: Windhawk
Date Posted: January 06 2009 at 05:18
Interesting. When even IMDB has gone over to using weighted ratings. I would assume they have their reasons for that - and the crown argument of the thread starter appears to be somewhat busted here now.
A continud discussion as to how much or not a weighting should be might be appropriate - but if the admins calculations are correct here and the difference is in the 10-15% range at max; what's the problem?
As far as I know, when people are looking around to buy music they will look it up in a number of places; and read several reviews as well before deciding - at least when shopping on the net. Most will seek out samples too these days.
As ratings go, they show an indication of popularity in terms of broadness of appeal and the general appeal amongst the scope of those who have it. And so far in life I don't think I've ever encountered people buying an album based on ratings alone...
------------- Websites I work with:
http://www.progressor.net http://www.houseofprog.com
My profile on Mixcloud: https://www.mixcloud.com/haukevind/
|
Posted By: Mr ProgFreak
Date Posted: January 06 2009 at 06:17
^ I think that in the case of IMDB they also use the reviews to identify raters who can be trusted. They also have that feature of "rating reviews". Of course that can be used to to compute a "trust level" for reviewers - together with other factors, like for example whether people are consistently submitting trustworthy ratings over an extended period of time. Most of the manipulative votes come in "bursts".
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Dean
Date Posted: January 06 2009 at 06:41
^ IMDb also only use ratings from regular reviewers when computing their Top-100 ... and they give no indication of what constitutes a "regular reviewer".
------------- What?
|
Posted By: Mr ProgFreak
Date Posted: January 06 2009 at 06:49
^ yes, I remember reading about that. Apparently your ratings become more important if you submit reviews over an extended period of time. That makes a lot of sense to me, and maybe I will implement something like that at PF some day. However, I would make it more transparent, and I also think that I would limit the range of weights to the factor of 2 or maybe 3.
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Angelo
Date Posted: January 06 2009 at 11:33
I'm with stupid, err, I mean Bob - this has entered the yes no stage, so I'm off to warmer places (it's -9 C here now - only people like Peter enjoy a cold beer at those temperatures)
------------- http://www.iskcrocks.com" rel="nofollow - ISKC Rock Radio I stopped blogging and reviewing - so won't be handling requests. Promo's for ariplay can be sent to [email protected]
|
Posted By: Uncle Spooky
Date Posted: January 09 2009 at 04:45
Windhawk wrote:
Interesting. When even IMDB has gone over to using weighted ratings. I would assume they have their reasons for that - and the crown argument of the thread starter appears to be somewhat busted here now. |
Just to clear up confusion here, IMDb's "weighting" here refers to active vote stuffing/lazy voting filters and the usual statistical methods for weighting individual entries across larger samples, not assigning weight to individuals.
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Uncle Spooky
Date Posted: January 09 2009 at 04:47
Dean wrote:
^ IMDb also only use ratings from regular reviewers when computing their Top-100 ... and they give no indication of what constitutes a "regular reviewer". |
This simply means that voters have to pass a certain threshold of number of votes cast before they are included in the Top charts. Again, no weighting is applied to those included in the top charts.
Cheers,
Mark
------------- http://last.fm/user/Mark_H" rel="nofollow - http://last.fm/user/Mark_H
|
Posted By: Dean
Date Posted: January 09 2009 at 05:55
Uncle Spooky wrote:
Windhawk wrote:
Interesting. When even IMDB has gone over to using weighted ratings. I would assume they have their reasons for that - and the crown argument of the thread starter appears to be somewhat busted here now. |
Just to clear up confusion here, IMDb's "weighting" here refers to active vote stuffing/lazy voting filters and the usual statistical methods for weighting individual entries across larger samples, not assigning weight to individuals.
Mark
Dean wrote:
^ IMDb also only use ratings from regular reviewers when computing their Top-100 ... and they give no indication of what constitutes a "regular reviewer". |
This simply means that voters have to pass a certain threshold of number of votes cast before they are included in the Top charts. Again, no weighting is applied to those included in the top charts. Cheers, Mark |
There is no confusion - neither site uses a simple arithmetic average of all votes cast. IMDb has the luxury of large sample sizes so statistical weighting has a reasonable level of confidence. Unfortunately we do not have large sample sizes so statistical analysis would be so inaccurate as to be meaningless. If we applied IMDb methods then most albums would be have zero ratings and many people who submitted ratings-only would be excluded completely. The system isn't perfect, but we do try to include everybody's opinion.
However, both sites do use the same Bayesian algorithm when computing the Top 100. 
------------- What?
|
Posted By: debrewguy
Date Posted: January 09 2009 at 21:04
But does this mean that some albums reviewed a hundred times or so are not as good or bad as they're rated ? And if so, how do we move another hundred people to review the same album to see if the previous hundred reviewers got it all wrong ? And having done that, would we get still another hundred people to review the reviews and the albums and vote on which set of reviewers is kinda right ? Heck, let's save time, me & T rate the RIO/Avant-Garde; Rocktopus takes care of the prog metal, Sean Trane does the Neo, Mandrakeroot does Raga rock, and admin strip all Symph albums of their ratings so we can start all over, then we get Baldfriede to handle the crossover, with Raff eliminating the eclectic & jazz fusion genres until the Electronic prog lovers notice that Kraft has split from Werk. Then, after our 11th beer, me & VB admit that the site is really a put on by the staff of Kerrang.
------------- "Here I am talking to some of the smartest people in the world and I didn't even notice,” Lieutenant Columbo, episode The Bye-Bye Sky-High I.Q. Murder Case.
|
Posted By: Mr ProgFreak
Date Posted: January 10 2009 at 01:58
^  
Of course you have a point - we shouldn't take this all too serious. However, when a website implements a system which gives different weights to the votes depending on the users's status ... in that case I think it's important for the website to try to be transparent about the algorithm. Especially when people submit their rating and the new album average does not change in the expected way, there should be some way for them to find out how it works.
Which reminds me that I should add/update those explanations at PF too ... 
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Atavachron
Date Posted: January 10 2009 at 02:15
ahh, the Bayesian algorithm, use it every day
|
Posted By: Mr ProgFreak
Date Posted: January 10 2009 at 02:59
^ actually I'm wondering who brought that up ... I'm pretty sure that PA doesn't use Bayesian filters. You could not apply them to ratings ... only reviews, but PA is monitoring them manually.
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Dean
Date Posted: January 10 2009 at 05:23
Mr ProgFreak wrote:
^ actually I'm wondering who brought that up ... I'm pretty sure that PA doesn't use Bayesian filters. You could not apply them to ratings ... only reviews, but PA is monitoring them manually. |
Not Bayesian Filters, you were the only person to mention Filters. 
Bayesian Weighting is not filtering:
br = ( (avg_num_votes * avg_rating) + (this_num_votes * this_rating) ) / (avg_num_votes + this_num_votes)
------------- What?
|
Posted By: Mr ProgFreak
Date Posted: January 10 2009 at 05:31
^ and now you introduced "Bayesian Weighting" ... 
Actually "Weighted Mean" or "Weighted Average" means something different - it means applying weights to all the ratings. Maybe M@x should remove the link on the charts page to http://en.wikipedia.org/wiki/Weighted_average#Example - http://en.wikipedia.org/wiki/Weighted_average#Example .
The thing you're describing ... I've never heard it being referred to as "Bayesian", but I guess you're right. The principle is explained here: http://en.wikipedia.org/wiki/Bayesian_average - http://en.wikipedia.org/wiki/Bayesian_average , so that's the link which should be used on the charts page. 
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Dean
Date Posted: January 10 2009 at 06:25
^ True on all counts - I originally used the term Bayesian Algorithm, which can apply to either filtering or weighting, however since I said it was used to calculate the Top 100 it infers weighting.
Yes, the weighted averages link should be removed - it applied to the previous algorithm used to calculate individual album averages and is no longer used. Bayesian weighting is only used to calculate chart position and not the displayed average value, which is why CTTE has a lower average than WYWH but has a higher chart position.
Of course any statistical probablity based system is doomed to failure on the small sample populations we have here. Analysis of an album with only 6 votes is meaningless, even the a straight arithmetic mean is pointless - if 3 people love it and 3 people hate it that does not make the album "average", quite the reverse in fact. No amount of weighting will give a meaningful number because there isn't one. Even for albums with 900 votes the average tells you nothing because it does not take into account your personal taste or predilection.
The best computer to analyse a set of ratings is still the human brain, the numbers are just numbers.
------------- What?
|
Posted By: Mr ProgFreak
Date Posted: January 10 2009 at 06:35
Dean wrote:
^ True on all counts - I originally used the term Bayesian Algorithm, which can apply to either filtering or weighting, however since I said it was used to calculate the Top 100 it infers weighting.
Yes, the weighted averages link should be removed - it applied to the previous algorithm used to calculate individual album averages and is no longer used. Bayesian weighting is only used to calculate chart position and not the displayed average value, which is why CTTE has a lower average than WYWH but has a higher chart position.
|
It would be interesting for the users to see the bayesian average along with the arithmetic mean, but from my own website I can say that it's a bit difficult to implement. However I'll try to do that.
Dean wrote:
Of course any statistical probablity based system is doomed to failure on the small sample populations we have here. Analysis of an album with only 6 votes is meaningless, even the a straight arithmetic mean is pointless - if 3 people love it and 3 people hate it that does not make the album "average", quite the reverse in fact. No amount of weighting will give a meaningful number because there isn't one. Even for albums with 900 votes the average tells you nothing because it does not take into account your personal taste or predilection.
The best computer to analyse a set of ratings is still the human brain, the numbers are just numbers. |
Well, I think that the numbers are quite useful. Of course they don't represent the "true" rating of the album ... there is no such thing. As far as I'm concerned, ratings are useful because they enable the system to provide suggestions - even if only two people rate something highly, I might want to check it out.
BTW: I already thought of what you're describing in the highlighted section. At PF I'm calculating the standard deviation for each album, and http://progfreak.com/home/music.xhtml?path=albums/controversial - here you can see the album with the highest values. For large numbers of ratings with about equal "haters" and "lovers", it might even make sense to tweak the resulting average in some way. At PF I'm doing that by also considering the median value in the resulting average.
------------- https://tagyourmusic.org/users/Mike" rel="nofollow - https://tagyourmusic.org/users/Mike
|
Posted By: Dean
Date Posted: January 10 2009 at 07:15
Mr ProgFreak wrote:
Dean wrote:
if 3 people love it and 3 people hate it that does not make the album "average", quite the reverse in fact. |
Well, I think that the numbers are quite useful. Of course they don't represent the "true" rating of the album ... there is no such thing. As far as I'm concerned, ratings are useful because they enable the system to provide suggestions - even if only two people rate something highly, I might want to check it out. |
Doubly so if those two people have similar tastes to you.
Mr ProgFreak wrote:
BTW: I already thought of what you're describing in the highlighted section. At PF I'm calculating the standard deviation for each album, and http://progfreak.com/home/music.xhtml?path=albums/controversial - here you can see the album with the highest values. For large numbers of ratings with about equal "haters" and "lovers", it might even make sense to tweak the resulting average in some way. At PF I'm doing that by also considering the median value in the resulting average.
|
The question then is which way to tweak the average. Do you tweak it in favour of the "lovers" or "haters"? Common sense says towards the "lovers" ... (a low rating by a "hater" is in effect a high rating  ) ... but the problem there is what if the low ratings were from people who love the genre/artist but hate the album.
Standard deviation does give more information - we could flood the page with numbers, but that is a distraction which would open us to even more criticism by people who would not appreicate what the numbers mean. We do plot the distributions on each album page - people should be using that graph to draw their own conclusions rather than concentrating on the individual scores (sorry they don't display properly here, but in essence the 3.74 rating is better explained by the 44% of people who gave the album 4-stars):
------------- What?
|
Posted By: Desoc
Date Posted: April 10 2009 at 09:20
Well, I realize that this thread has been inactive for some weeks now, but I feel the need to make the question reappear, partly because I feel the debate was largely inconclusive.
I didn't join the crowd the last time around, but I must admit that the debate puzzled me. I have the deepest respect for most of the collaborators and the time and effort they put into this site. But this thread was a curious showcase.
Regarding the debate
To my eyes, the debate consisted mainly of non-collaborators (in particular, but not limited to one single person) that was questioning a particular (and very visible and impactable) feature of the site, against a massive load of collaborators who (with a couple of exceptions) went right down in the trenches to defend their privileges. I don't think these privileges are the reason they are active, so the total reaction was peculiar, and it certainly stopped me from engaging in the debate.
Well, this is not meant to be a rant against collabs, whose efforts - as I said - I admire. But this thread leaves the impression that there is a certain defiance against the common people here, which is an impression that gains noone, regardless of its accuracy. (Something similar can be found in this thread: http://www.progarchives.com/forum/forum_posts.asp?TID=55758 - http://www.progarchives.com/forum/forum_posts.asp?TID=55758 and this: http://www.progarchives.com/forum/forum_posts.asp?TID=55741&PN=2 - http://www.progarchives.com/forum/forum_posts.asp?TID=55741&PN=2 ) I think Mark had valid points, and I was surprised at how he was met. Take it as friendly advice.
Reviews vs ratings
I believe that collabs in general write better reviews than non-collabs. Thus, I think the exhibition of their reviews should reflect this. For my part, the frontpage feed could consist of collab reviews only. And collabs should be rewarded manyfold for their efforts in various ways.
But ratings are an entirely other issue. Being a good reviewer doesn't mean that your opinion is more qualified. And what is the point of the rating system? First and foremost it is to show the standing of an album amongst the community at large. As such, the current system must be said to be misleading.
Possible changes?
When I say that the debate was largely inconclusive, I refer to the fact that most of the defendants where people who "gained" on the current system, and those few who raised voices were (with a couple of exceptions) not. But there were a few concrete proposals that hardly anyone commented.
I'm curious to know if displaying different averages is something that could be considered, or alternatively why not. Even if the current system remains the "standard", would anything be lost if people could additionally chose basic average, non-weighted ratings, collab-only ratings etc?
And what about having a filter on the chart pages that removes the weighting? Or is that technically impossible?
A specific example
I'd like to close with an example of how the current system works. Let me attract your attention to this album: http://www.progarchives.com/album-reviews.asp?id=12217 - http://www.progarchives.com/album-reviews.asp?id=12217
This album had a rating of 4,67 or something and was on the top 100 chart if minimum # of ratings was lowered. Then one collab gave it a 1-star rating, without review, and it dropped like a rock. First of all, this shows that it's dead wrong that it's the quality of reviews that matters. Secondly, it shows that this system doesn't primarily reward collabs - it primarily punishes the other 17 raters, many of them with well-crafted reviews. And this would have been the case also if the collab in question wrote a review alongside.
So in conclusion, if the site owners feel that weighting by mixing reviews and ratings is important for giving incentives to writing reviews, then I will hold my peace. But then my advice as a regular user would be to at least treat all reviews equal, and rather give other kinds of bonuses to collabs. At any rate, the current weighting is - sorry - ridiculously biased.
|
|