Forum Home Forum Home > Site News, Newbies, Help and Improvements > Report bugs here
  New Posts New Posts RSS Feed - missing Albums on top 2014 list?!
  FAQ FAQ  Forum Search   Events   Register Register  Login Login

Topic Closedmissing Albums on top 2014 list?!

 Post Reply Post Reply Page  <12
Author
Message
Mind_Drive View Drop Down
Forum Senior Member
Forum Senior Member
Avatar

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 374
Direct Link To This Post Posted: May 18 2014 at 17:16
hey Dean,

thanks for reading through all my complainments Wink

the QWR is in the right order but it doesnīt seem to be correct in every case.

letīs compare:
PFMs per un amico and anglagards hybris in the overall toplist:
PFMs album has 4.42 and 1041 ratings and 96 reviews ranking at nr 14
Anglagards album has 4.42 and 1144 ratings and 130 reviews ranking at nr 15

although i donīt have studied the rating system to a degree, that i could be sure this is not correct -
my feeling says, that they should rank the other way round, you donīt think so?

i mean it would only make sense to me if the weight of reviews would have an impact on the QWR and PFMs albm had significantly more collaborator reviews.
but i think the reviews weight is only used to calculate the average rating and from then its only the average rating and the number of ratings in relation to the total number of ratings (and maybe overall average rating) to calculate the QWR.

in the higher areas of the toplists there seem to be only slightly wrong placed albums
but if you look into the unpopular parts of this list e.g:
http://www.progarchives.com/top-prog-albums.asp?ssubgenres=&salbumtypes=1&syears=2014&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=250&x=67&y=3#list

you will encounter lots of questionable orders of albums (the QWR ofc here is coherent, too)...
152: Lascailleīs Shroud 3.04 - 4 ratings
153: Not Overwise Specified 3.29 - 7 ratings
154: DaccorD 3.58 - 31 ratings

i hope you can get my point and i donīt bring too much confusion to you right now Wink




Edited by Mind_Drive - May 18 2014 at 17:31
It's just a ride... <3
Back to Top
Sponsored Links


Back to Top
HolyMoly View Drop Down
Special Collaborator
Special Collaborator
Avatar
Retired Admin

Joined: April 01 2009
Location: Atlanta
Status: Offline
Points: 25951
Direct Link To This Post Posted: May 18 2014 at 18:07
I wish I had time right now to delve into this, because I love maths.  But on the surface, I don't think it necessarily follows that two albums with an identical average (e.g. PFM and Anglagard example above) should be ranked by which one has more reviews.  Perhaps the distribution of ratings (and weightings) has some bearing on it - the "standard deviation" concept used often in statistics.  I don't know though, so I'll shut up for now.
My other avatar is a Porsche

It is easier for a camel to pass through the eye of a needle if it is lightly greased.

-Kehlog Albran
Back to Top
Dean View Drop Down
Special Collaborator
Special Collaborator
Avatar
Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575
Direct Link To This Post Posted: May 18 2014 at 19:42
Originally posted by Mind_Drive Mind_Drive wrote:

hey Dean,

...
i hope you can get my point and i donīt bring too much confusion to you right now Wink


The only confusion is you said you understood QWR but kept saying there was a bug.

The QWR formula is:

QWR = (NR + nr) / (N + n)

where QWR is the weighted rating, R is the average rating of all albums, N is the average number of votes per album for all albums, r is the album average and n is the number of ratings.

The effect of calculating the QWR is two-fold: 
  1. As the number of ratings surpasses N, the weighted rating QWR approaches the actual average r. ie QWR  (nr) / (n)
  2. The closer "n" is to zero, the closer QWR  gets to R. ie  QWR  (NR) / (N)
So, in simpler terms, albums with very few ratings/votes will have a rating weighted towards the average across all albums, while albums with many ratings/votes will have a rating weighted towards their own average rating. 

Therefore once the number of ratings for an album gets really big (ie n is much larger than N) then the actual value of the average rating r will have more effect on the formula than the chart average rating R and the QWR value will approach the actual average rating value r.

QWR is a prediction, it is a prediction of what the average ratings would be if all the albums had similar numbers of ratings so it allows us to compare an album with 2 ratings with one that has 2000 ratings, therefore it is more related to probability.
Originally posted by Mind_Drive Mind_Drive wrote:

thanks for reading through all my complainments Wink

the QWR is in the right order but it doesnīt seem to be correct in every case.

letīs compare:
PFMs per un amico and anglagards hybris in the overall toplist:
PFMs album has 4.42 and 1041 ratings and 96 reviews ranking at nr 14
Anglagards album has 4.42 and 1144 ratings and 130 reviews ranking at nr 15

although i donīt have studied the rating system to a degree, that i could be sure this is not correct -
my feeling says, that they should rank the other way round, you donīt think so?
No, not really. The numbers displayed are not accurate enough to make that kind of judgement call. The Average rating value and the QWR value for each album is calculated to more decimal places (dp) accuracy than what is displayed on the chart. So while both are displayed with an average of 4.42 (to 2 dp), their actual average ratings will be different - one could be 4.420001 and the other 4.4249999 for example. 

[example 2: The Abscinded Universe has an unweighted average of 3.25 and a weighted average of 3.04, but if you compute the weighted value to its full-precision it is 3.04347826086957]

To explain that further, consider the two-fold effect of QWR (above)

In both these cases the number of ratings (1041 and 1144) is significantly larger than N, which I guess is somewhere around the 20 mark. This is effect (1): because >> N the QWR values are very close to the Average rating values r for these albums. 

This suggests that if we could see the averages to more decimal places, we would see that Per Un Amico has a slightly higher average than Hybris.

For example, assuming that N =17 and R=4.2 (I'm not saying they are, this is just an example), if we back-calculate the values of r from the displayed values of QWR for each album we get:

Per Un Amico  QWR = 4.3899, therefore r=4.423539 ... which rounds to 4.42 to 2dp
Hybris  QWR = 4.3895, therefore r=4.423269 ... which rounds to 4.42 to 2dp

Originally posted by Mind_Drive Mind_Drive wrote:


i mean it would only make sense to me if the weight of reviews would have an impact on the QWR and PFMs albm had significantly more collaborator reviews. 
but i think the reviews weight is only used to calculate the average rating and from then its only the average ratingand the number of ratings in relation to the total number of ratings (and maybe overall average rating) to calculate the QWR.

in the higher areas of the toplists there seem to be only slightly wrong placed albums
but if you look into the unpopular parts of this list e.g:
http://www.progarchives.com/top-prog-albums.asp?ssubgenres=&salbumtypes=1&syears=2014&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=250&x=67&y=3#list

you will encounter lots of questionable orders of albums (the QWR ofc here is coherent, too)...
152: Lascailleīs Shroud 3.04 - 4 ratings
153: Not Overwise Specified 3.29 - 7 ratings
154: DaccorD 3.58 - 31 ratings

Now these examples are the result of effect (2) of the QWR formula because the number of ratings for each is very small (ie n < N) so the QWR values approach the value R and this value now has more effect on the result. 

As I said earlier, the site average for the number of ratings per album is around 20, so while 7 ratings is small and does approach zero, it is still quite close to N, (more so than the 1041 ratings for Per Un Amico for example). This means that we are close to the situation where R and r have equal effect (as would be the case if n=N).

While D'AccorD III is > N, Projective Instruments and The Abscinded Universe are < N. This means that for  D'AccorD III r has more effect on the QWR and for Projective Instruments and The Abscinded Universe R has more effect.

You can think of this as the value of acting like a fulcrum on a see-saw: values above move down and values below move up, which leads to a counter-intuitive result. (ie it looks wrong to you).

This is why the main charts do not show albums that have n < N.

The problem with any statistical analysis of album ratings on the PA is that a relatively small population that rate any one album. This means that very slight changes in the data can have a dramatic (and subsequently meaningless) effect on the result. The charts are just for fun and should not be taken seriously.



Edited by Dean - May 18 2014 at 20:02
What?
Back to Top
Guldbamsen View Drop Down
Special Collaborator
Special Collaborator
Avatar
Retired Admin

Joined: January 22 2009
Location: Magic Theatre
Status: Offline
Points: 23098
Direct Link To This Post Posted: May 19 2014 at 06:22
Dean to the rescue!

Thanks. Like I said, to me this is mostly gibberish.
“The Guide says there is an art to flying or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”

- Douglas Adams
Back to Top
Mind_Drive View Drop Down
Forum Senior Member
Forum Senior Member
Avatar

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 374
Direct Link To This Post Posted: May 19 2014 at 07:14
thank you Dean!! you restored my sanity and my trust in the archives LOL

i was not aware of this formular and you explained it super nice!

especially the following sentences where strinking:
"QWR is a prediction, it is a prediction of what the average ratings would be if all the albums had similar numbers of ratings so it allows us to compare an album with 2 ratings with one that has 2000 ratings, therefore it is more related to probability."

"...which leads to a counter-intuitive result."

itīs indeed counter-intuitive but now i got it Wink

thanks again for your efford!

prog on! Smile

It's just a ride... <3
Back to Top
 Post Reply Post Reply Page  <12

Forum Jump Forum Permissions View Drop Down

Forum Software by Web Wiz Forums® version 11.01
Copyright ©2001-2014 Web Wiz Ltd.

This page was generated in 1.352 seconds.

Donate monthly and keep PA fast-loading and ad-free forever.