Curious finding in charts - Progressive Rock Music Forum

Author

Message

Topic Search

Topic Options

Marty McFly View Drop Down

Special Collaborator

Honorary Collaborator

Joined: March 23 2009
Location: Czech Republic
Status: Offline
Points: 3968

Topic: Curious finding in charts
Posted: February 22 2012 at 08:32

I was looking through Neo Prog top DVD's charts and found many errors (release that is higher than it should be from its ratings). Among the most weird ones are these three depicted on the screenshot:

EDIT: Probably needless to say, but I'll say it anyway - 45 should be 46, 46 should be 47 and 47 should be 45. All albums have reviews. And why I think it should be like I am saying ? Well, that's my understanding of how charts should work - the higher rating AND the more ratings, the higher the entry should be. Some complex equation of these two ... maybe number of reviews is accounted too, maybe just having a review (with its weighted ratio) is enough, I don't know.

(Full) Link to this list is here:

http://www.progarchives.com/top-prog-albums.asp?ssubgenres=18&salbumtypes=2&syears=&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=100&x=73&y=6#list

But this is just a one problem of many I am encountering every time I use charts.

Here, you have basic link from main page:

http://www.progarchives.com/top-prog-albums.asp?salbumtypes=1#list

OK ? We all know that ... Well, when I trim the genres to just Symphonic and Neo Prog (yeah, with CTRL), it will show me Symph and Neo - BUT then the weird thing happens. It will automatically select Tech/Extreme Prog Metal, Eclectic, Experimental and Heavy Prog from some reason I can oblivious to.

I just don't get it ... this bug is here as long as I know, maybe 3 years at least ?

Is there something that can be done about that, so that charts would work like they should ?

Edited by Marty McFly - February 22 2012 at 08:35

There's a point where "avant-garde" and "experimental" becomes "terrible" and "pointless,"

-Andyman1125 on Lulu

Even my

Dean

Special Collaborator

Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575

Posted: February 22 2012 at 11:14

Chart position and Average value are calculated using two different algorithms - ~~the numeric score takes collaborator weighting into account whereas the chart position does not.~~

/edit: correction the collab/review weighting is used for both.

Edited by Dean - February 22 2012 at 14:54

What?

Marty McFly View Drop Down

Special Collaborator

Honorary Collaborator

Joined: March 23 2009
Location: Czech Republic
Status: Offline
Points: 3968

Posted: February 22 2012 at 12:08

Well, Pendragon album has 4 reviews in total (3 of them collab - 4-4-3 stars), while Fish and Haze have just 1 (4 and 3 stars respectively).

Not sure if it changes anything.

There's a point where "avant-garde" and "experimental" becomes "terrible" and "pointless,"

-Andyman1125 on Lulu

Even my

Dean

Special Collaborator

Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575

Posted: February 22 2012 at 13:13

~~As I said - collaborator weighting is not used to calculate chart position~~, so

Underneath the Average rating, which is the value calculated using Collaborator and Review weighting you will see the QWR average ... or Query Weighted Rating, this is used to calculate chart position and uses an algorithm like this:

br = ( (avg_num_votes * avg_rating) + (this_num_votes * this_rating) ) / (avg_num_votes + this_num_votes)

Which when you pick any two DVDs you can easily solve to find the avg_num_votes and avg_rating score for this site, which I have and they are:

avg_num_votes = 15.9779

avg_rating = 3.9979

What this means is that any disc that has a rating higher than the site average rating will be weighted down, and any disc that has a rating lower will be weighted up. Similarly, any disc that has fewer ratings than the site average number will be weighted up and those with more will be weighted down.

Weighted averages are prone to producing counter-intuitive results (such as these here)... as you said yourself in the OP ... "the more ratings, the higher the entry should be" - that's the intuitive result and that would be a false result, the weighted algorithm corrects that "popularity" skew.

/edit: collab/review weighting average is used in this formula, not the arithmetic average

Edited by Dean - February 22 2012 at 14:55

What?

TheGazzardian View Drop Down

Prog Reviewer

Joined: August 11 2009
Location: Canada
Status: Offline
Points: 8443

Posted: February 22 2012 at 13:28

Hey Marty, the number used to calculate the position, as Dean said, is the weighted average. You can see that number next to each album, under the average and number of ratings, where it says QWR =. And by those numbers, the list you showed is in the correct order.

Marty McFly View Drop Down

Special Collaborator

Honorary Collaborator

Joined: March 23 2009
Location: Czech Republic
Status: Offline
Points: 3968

Posted: February 22 2012 at 13:38

OK, this explained a lot to me, I completely (unfortunately) forgot about QWR.

So even this explain a lot (little rated ones getting boosted up and high rated ones being beaten down, which is a case of Steven Wilson's new album as we can see + [correct me if I am wrong] high ratings are disadvantage and low ratings can work as advantage to an extent, or this switched logic works only on number of ratings, not on its values?) ...

but it's not just against my intuition, it just doesn't make sense. Well, obviously it does, because there are equations that calculates it.

But what I mean is simply that 46 shouldn't be higher than 47. It has far less ratings, less reviews, lesser rating (by almost whole point, which in our 4 point system is a lot). If it's not a bug, then it is not right. Why I think so ? My logic is simple - as it is now, having such album higher than my Pendragon example doesn't help anyone. I don't know any of these three DVD's in question, but logic, intuiting and common sense all tells me that it should be differently.

There's a point where "avant-garde" and "experimental" becomes "terrible" and "pointless,"

-Andyman1125 on Lulu

Even my

Triceratopsoil View Drop Down

Forum Senior Member

Joined: April 03 2010
Location: Canada
Status: Offline
Points: 17995

Posted: February 22 2012 at 14:05

I think that's why the top lists, by default, filter out releases with fewer than X ratings

Dean

Special Collaborator

Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575

Posted: February 22 2012 at 14:14

What the QWR is calculating is not the average score of the album from all the votes it received, but the average score of the album relative to all the votes for all albums.

for example - if one album has an average of 4.5 another has 4.2 then the mean of the two would be 4.35, which would be fine if both albums received the same number of votes - however if the first had 5 votes and the second 10 votes then the real average would have been (4.5*5+4.2*10)/15 = 4.3 and the average number of votes cast per album would be 7.5.

So what weighted averages do is calculate the average score for an album in relation to that 4.3 score and that 7.5 average number of votes, so when we get another album that has an average of 5 over 1 vote we can plug it into the forumula and calculate it's relative position - ie ( (7.5 * 4.3) + (1 * 5) ) / (7.5 + 1) = 4.38, rather than "5" and would slot this new album inbetween the two rather than it becoming the top album.

What?

Dean

Special Collaborator

Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575

Posted: February 22 2012 at 14:53

CORRECTION

Sorry Marty, I've just calculated the numbers for these three albums and I've discovered I have told you a big fat fib. The collaborators weightings are used to calculate the chart position with the weighted average formula. So the reason why Pendragon has a lower position is the lower percentage of collaborator/review ratings (11.8%) compared to Fish (12.5%) and Haze (100%).

I'm going to have to think about whether that is "the right thing to do" or not...

Edited by Dean - February 22 2012 at 15:58

What?

Author	Message Topic Search Topic Options Post Reply Create New Topic Printable Version Translate Topic
Marty McFly Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Special Collaborator Honorary Collaborator Joined: March 23 2009 Location: Czech Republic Status: Offline Points: 3968	Topic: Curious finding in charts Posted: February 22 2012 at 08:32
	I was looking through Neo Prog top DVD's charts and found many errors (release that is higher than it should be from its ratings). Among the most weird ones are these three depicted on the screenshot: EDIT: Probably needless to say, but I'll say it anyway - 45 should be 46, 46 should be 47 and 47 should be 45. All albums have reviews. And why I think it should be like I am saying ? Well, that's my understanding of how charts should work - the higher rating AND the more ratings, the higher the entry should be. Some complex equation of these two ... maybe number of reviews is accounted too, maybe just having a review (with its weighted ratio) is enough, I don't know. (Full) Link to this list is here: http://www.progarchives.com/top-prog-albums.asp?ssubgenres=18&salbumtypes=2&syears=&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=100&x=73&y=6#list But this is just a one problem of many I am encountering every time I use charts. Here, you have basic link from main page: http://www.progarchives.com/top-prog-albums.asp?salbumtypes=1#list OK ? We all know that ... Well, when I trim the genres to just Symphonic and Neo Prog (yeah, with CTRL), it will show me Symph and Neo - BUT then the weird thing happens. It will automatically select Tech/Extreme Prog Metal, Eclectic, Experimental and Heavy Prog from some reason I can oblivious to. I just don't get it ... this bug is here as long as I know, maybe 3 years at least ? Is there something that can be done about that, so that charts would work like they should ? Edited by Marty McFly - February 22 2012 at 08:35
	There's a point where "avant-garde" and "experimental" becomes "terrible" and "pointless," -Andyman1125 on Lulu Even my

Dean Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Special Collaborator Retired Admin and Amateur Layabout Joined: May 13 2007 Location: Europe Status: Offline Points: 37575	Posted: February 22 2012 at 11:14
	Chart position and Average value are calculated using two different algorithms - ~~the numeric score takes collaborator weighting into account whereas the chart position does not.~~ /edit: correction the collab/review weighting is used for both. Edited by Dean - February 22 2012 at 14:54
	What?

Marty McFly Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Special Collaborator Honorary Collaborator Joined: March 23 2009 Location: Czech Republic Status: Offline Points: 3968	Posted: February 22 2012 at 12:08
	Well, Pendragon album has 4 reviews in total (3 of them collab - 4-4-3 stars), while Fish and Haze have just 1 (4 and 3 stars respectively). Not sure if it changes anything.
	There's a point where "avant-garde" and "experimental" becomes "terrible" and "pointless," -Andyman1125 on Lulu Even my

Dean Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Special Collaborator Retired Admin and Amateur Layabout Joined: May 13 2007 Location: Europe Status: Offline Points: 37575	Posted: February 22 2012 at 13:13
	~~As I said - collaborator weighting is not used to calculate chart position~~, so Underneath the Average rating, which is the value calculated using Collaborator and Review weighting you will see the QWR average ... or Query Weighted Rating, this is used to calculate chart position and uses an algorithm like this: br = ( (avg_num_votes * avg_rating) + (this_num_votes * this_rating) ) / (avg_num_votes + this_num_votes) Which when you pick any two DVDs you can easily solve to find the avg_num_votes and avg_rating score for this site, which I have and they are: avg_num_votes = 15.9779 avg_rating = 3.9979 What this means is that any disc that has a rating higher than the site average rating will be weighted down, and any disc that has a rating lower will be weighted up. Similarly, any disc that has fewer ratings than the site average number will be weighted up and those with more will be weighted down. Weighted averages are prone to producing counter-intuitive results (such as these here)... as you said yourself in the OP ... "the more ratings, the higher the entry should be" - that's the intuitive result and that would be a false result, the weighted algorithm corrects that "popularity" skew. /edit: collab/review weighting average is used in this formula, not the arithmetic average Edited by Dean - February 22 2012 at 14:55
	What?

TheGazzardian Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Prog Reviewer Joined: August 11 2009 Location: Canada Status: Offline Points: 8443	Posted: February 22 2012 at 13:28
	Hey Marty, the number used to calculate the position, as Dean said, is the weighted average. You can see that number next to each album, under the average and number of ratings, where it says QWR =. And by those numbers, the list you showed is in the correct order.

Marty McFly Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Special Collaborator Honorary Collaborator Joined: March 23 2009 Location: Czech Republic Status: Offline Points: 3968	Posted: February 22 2012 at 13:38
	OK, this explained a lot to me, I completely (unfortunately) forgot about QWR. So even this explain a lot (little rated ones getting boosted up and high rated ones being beaten down, which is a case of Steven Wilson's new album as we can see + [correct me if I am wrong] high ratings are disadvantage and low ratings can work as advantage to an extent, or this switched logic works only on number of ratings, not on its values?) ... but it's not just against my intuition, it just doesn't make sense. Well, obviously it does, because there are equations that calculates it. But what I mean is simply that 46 shouldn't be higher than 47. It has far less ratings, less reviews, lesser rating (by almost whole point, which in our 4 point system is a lot). If it's not a bug, then it is not right. Why I think so ? My logic is simple - as it is now, having such album higher than my Pendragon example doesn't help anyone. I don't know any of these three DVD's in question, but logic, intuiting and common sense all tells me that it should be differently.
	There's a point where "avant-garde" and "experimental" becomes "terrible" and "pointless," -Andyman1125 on Lulu Even my

Triceratopsoil Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Forum Senior Member Joined: April 03 2010 Location: Canada Status: Offline Points: 17995	Posted: February 22 2012 at 14:05
	I think that's why the top lists, by default, filter out releases with fewer than X ratings

Dean Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Special Collaborator Retired Admin and Amateur Layabout Joined: May 13 2007 Location: Europe Status: Offline Points: 37575	Posted: February 22 2012 at 14:14
	What the QWR is calculating is not the average score of the album from all the votes it received, but the average score of the album relative to all the votes for all albums. for example - if one album has an average of 4.5 another has 4.2 then the mean of the two would be 4.35, which would be fine if both albums received the same number of votes - however if the first had 5 votes and the second 10 votes then the real average would have been (4.55+4.210)/15 = 4.3 and the average number of votes cast per album would be 7.5. So what weighted averages do is calculate the average score for an album in relation to that 4.3 score and that 7.5 average number of votes, so when we get another album that has an average of 5 over 1 vote we can plug it into the forumula and calculate it's relative position - ie ( (7.5 * 4.3) + (1 * 5) ) / (7.5 + 1) = 4.38, rather than "5" and would slot this new album inbetween the two rather than it becoming the top album.
	What?

Dean Members Profile Send Private Message Find Members Posts Add to Buddy List Reviewer page Special Collaborator Retired Admin and Amateur Layabout Joined: May 13 2007 Location: Europe Status: Offline Points: 37575	Posted: February 22 2012 at 14:53
	CORRECTION Sorry Marty, I've just calculated the numbers for these three albums and I've discovered I have told you a big fat fib. The collaborators weightings are used to calculate the chart position with the weighted average formula. So the reason why Pendragon has a lower position is the lower percentage of collaborator/review ratings (11.8%) compared to Fish (12.5%) and Haze (100%). I'm going to have to think about whether that is "the right thing to do" or not... Edited by Dean - February 22 2012 at 15:58
	What?