missing Albums on top 2014 list?! - Progressive Rock Music Forum

Author

Message

Topic Search

Topic Options

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Topic: missing Albums on top 2014 list?!
Posted: May 19 2014 at 07:14

thank you Dean!! you restored my sanity and my trust in the archives LOL

i was not aware of this formular and you explained it super nice!

especially the following sentences where strinking:
"QWR is a prediction, it is a prediction of what the average ratings would be if all the albums had similar numbers of ratings so it allows us to compare an album with 2 ratings with one that has 2000 ratings, therefore it is more related to probability."

"...which leads to a counter-intuitive result."

it´s indeed counter-intuitive but now i got it Wink

thanks again for your efford!

prog on! Smile

It's just a ride... <3

Guldbamsen View Drop Down

Special Collaborator

Retired Admin

Joined: January 22 2009
Location: Magic Theatre
Status: Offline
Points: 23098

Posted: May 19 2014 at 06:22

Dean to the rescue!

Thanks. Like I said, to me this is mostly gibberish.

“The Guide says there is an art to flying or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”

- Douglas Adams

Dean

Special Collaborator

Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575

Posted: May 18 2014 at 19:42

Mind_Drive wrote:

hey Dean,

...

i hope you can get my point and i don´t bring too much confusion to you right now Wink

The only confusion is you said you understood QWR but kept saying there was a bug.

The QWR formula is:

QWR = (NR + nr) / (N + n)

where QWR is the weighted rating, R is the average rating of all albums, N is the average number of votes per album for all albums, r is the album average and n is the number of ratings.

The effect of calculating the QWR is two-fold:

As the number of ratings surpasses N, the weighted rating QWR approaches the actual average r. ie QWR → (nr) / (n)
The closer "n" is to zero, the closer QWR gets to R. ie QWR → (NR) / (N)

So, in simpler terms, albums with very few ratings/votes will have a rating weighted towards the average across all albums, while albums with many ratings/votes will have a rating weighted towards their own average rating.

Therefore once the number of ratings for an album gets really big (ie n is much larger than N) then the actual value of the average rating r will have more effect on the formula than the chart average rating R and the QWR value will approach the actual average rating value r.

QWR is a prediction, it is a prediction of what the average ratings would be if all the albums had similar numbers of ratings so it allows us to compare an album with 2 ratings with one that has 2000 ratings, therefore it is more related to probability.

Mind_Drive wrote:

thanks for reading through all my complainments Wink

No, not really. The numbers displayed are not accurate enough to make that kind of judgement call. The Average rating value and the QWR value for each album is calculated to more decimal places (dp) accuracy than what is displayed on the chart. So while both are displayed with an average of 4.42 (to 2 dp), their actual average ratings will be different - one could be 4.420001 and the other 4.4249999 for example.

[example 2: The Abscinded Universe has an unweighted average of 3.25 and a weighted average of 3.04, but if you compute the weighted value to its full-precision it is 3.04347826086957]

To explain that further, consider the two-fold effect of QWR (above)

In both these cases the number of ratings n (1041 and 1144) is significantly larger than N, which I guess is somewhere around the 20 mark. This is effect (1): because n >> N the QWR values are very close to the Average rating values r for these albums.

This suggests that if we could see the averages to more decimal places, we would see that Per Un Amico has a slightly higher average than Hybris.

For example, assuming that N =17 and R=4.2 (I'm not saying they are, this is just an example), if we back-calculate the values of r from the displayed values of QWR for each album we get:

Per Un Amico QWR = 4.3899, therefore r=4.423539 ... which rounds to 4.42 to 2dp
Hybris QWR = 4.3895, therefore r=4.423269 ... which rounds to 4.42 to 2dp

Mind_Drive wrote:

i mean it would only make sense to me if the weight of reviews would have an impact on the QWR and PFMs albm had significantly more collaborator reviews.
but i think the reviews weight is only used to calculate the average rating and from then its only the average ratingand the number of ratings in relation to the total number of ratings (and maybe overall average rating) to calculate the QWR.

in the higher areas of the toplists there seem to be only slightly wrong placed albums
but if you look into the unpopular parts of this list e.g:
http://www.progarchives.com/top-prog-albums.asp?ssubgenres=&salbumtypes=1&syears=2014&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=250&x=67&y=3#list

you will encounter lots of questionable orders of albums (the QWR ofc here is coherent, too)...
152: Lascaille´s Shroud 3.04 - 4 ratings
153: Not Overwise Specified 3.29 - 7 ratings
154: DaccorD 3.58 - 31 ratings

Now these examples are the result of effect (2) of the QWR formula because the number of ratings for each is very small (ie n < N) so the QWR values approach the value R and this value now has more effect on the result.

As I said earlier, the site average for the number of ratings per album is around 20, so while 7 ratings is small and does approach zero, it is still quite close to N, (more so than the 1041 ratings for Per Un Amico for example). This means that we are close to the situation where R and r have equal effect (as would be the case if n=N).

While D'AccorD III is > N, Projective Instruments and The Abscinded Universe are < N. This means that for D'AccorD III r has more effect on the QWR and for Projective Instruments and The Abscinded Universe R has more effect.

You can think of this as the value of N acting like a fulcrum on a see-saw: values above N move down and values below N move up, which leads to a counter-intuitive result. (ie it looks wrong to you).

This is why the main charts do not show albums that have n < N.

The problem with any statistical analysis of album ratings on the PA is that a relatively small population that rate any one album. This means that very slight changes in the data can have a dramatic (and subsequently meaningless) effect on the result. The charts are just for fun and should not be taken seriously.

Edited by Dean - May 18 2014 at 20:02

What?

HolyMoly

Special Collaborator

Retired Admin

Joined: April 01 2009
Location: Atlanta
Status: Offline
Points: 26133

Posted: May 18 2014 at 18:07

I wish I had time right now to delve into this, because I love maths. But on the surface, I don't think it necessarily follows that two albums with an identical average (e.g. PFM and Anglagard example above) should be ranked by which one has more reviews. Perhaps the distribution of ratings (and weightings) has some bearing on it - the "standard deviation" concept used often in statistics. I don't know though, so I'll shut up for now.

My other avatar is a Porsche

It is easier for a camel to pass through the eye of a needle if it is lightly greased.

-Kehlog Albran

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Posted: May 18 2014 at 17:16

hey Dean,

thanks for reading through all my complainments Wink

the QWR is in the right order but it doesn´t seem to be correct in every case.

let´s compare:
PFMs per un amico and anglagards hybris in the overall toplist:
PFMs album has 4.42 and 1041 ratings and 96 reviews ranking at nr 14
Anglagards album has 4.42 and 1144 ratings and 130 reviews ranking at nr 15

although i don´t have studied the rating system to a degree, that i could be sure this is not correct -
my feeling says, that they should rank the other way round, you don´t think so?

i mean it would only make sense to me if the weight of reviews would have an impact on the QWR and PFMs albm had significantly more collaborator reviews.
but i think the reviews weight is only used to calculate the average rating and from then its only the average rating and the number of ratings in relation to the total number of ratings (and maybe overall average rating) to calculate the QWR.

in the higher areas of the toplists there seem to be only slightly wrong placed albums
but if you look into the unpopular parts of this list e.g:
http://www.progarchives.com/top-prog-albums.asp?ssubgenres=&salbumtypes=1&syears=2014&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=250&x=67&y=3#list

you will encounter lots of questionable orders of albums (the QWR ofc here is coherent, too)...
152: Lascaille´s Shroud 3.04 - 4 ratings
153: Not Overwise Specified 3.29 - 7 ratings
154: DaccorD 3.58 - 31 ratings

i hope you can get my point and i don´t bring too much confusion to you right now Wink

Edited by Mind_Drive - May 18 2014 at 17:31

It's just a ride... <3

Dean

Special Collaborator

Retired Admin and Amateur Layabout

Joined: May 13 2007
Location: Europe
Status: Offline
Points: 37575

Posted: May 18 2014 at 12:55

Mind_Drive wrote:

hey, thanks for your detailed answer!

i think i really understand the complex rating system, which i think is pretty awesome
but still: DTs album is still there when i look at the list and some strange rankings in other places i don´t feel that they are correct..

i guess i just have to deal with it and as i said it´s just my perfectionism thats speaks here^^ so most of you probably wont bother about some misplaced albums - and so should i Wink

what counts is the music and wow! i can´t tell how much i love the archives for it has enriched my life so much with so much inspiration for new music...

thanks so far! Smile

Hi Constantin,

Hmm.... I'm confused. I've read all your posts in this thread and I don't see what the problem is or why you think it is a bug.

I've looked at the chart and everything looks okay to me. The albums are ranked by their QWR rating, and that value is shown in the listing.

Can you explain exactly what you think the problem is.

What?

DamoXt7942 View Drop Down

Forum & Site Admin Group

Avant/Cross/Neo/Post Teams

Joined: October 15 2008
Location: Okayama, Japan
Status: Offline
Points: 17459

Posted: May 18 2014 at 06:34

Sorry I'm not familiar with PA rating system (and of course QWR ... oh my, that gives me headache like David LOL

), but let me say we Admins sometimes kick some ratings for hyping or manipulating out, but upon such a case, the rating system cannot usually work precisely ... there are a couple of errors or specifications also in PA, actually. Unhappy

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Posted: May 17 2014 at 17:32

hey, thanks for your detailed answer!

i think i really understand the complex rating system, which i think is pretty awesome
but still: DTs album is still there when i look at the list and some strange rankings in other places i don´t feel that they are correct..

i guess i just have to deal with it and as i said it´s just my perfectionism thats speaks here^^ so most of you probably wont bother about some misplaced albums - and so should i Wink

what counts is the music and wow! i can´t tell how much i love the archives for it has enriched my life so much with so much inspiration for new music...

thanks so far! Smile

It's just a ride... <3

Guldbamsen View Drop Down

Special Collaborator

Retired Admin

Joined: January 22 2009
Location: Magic Theatre
Status: Offline
Points: 23098

Posted: May 17 2014 at 17:19

Thanks Andy.-)

I get a head ache just looking at that stuff.
If anyone is interested in Max Ernst or Jack Kerouac though, then I can be of service. If it's to do with algorithms and codes, then I'm out:-P

“The Guide says there is an art to flying or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”

- Douglas Adams

Andy Webb

Special Collaborator

Retired Admin

Joined: June 04 2010
Location: Terria
Status: Offline
Points: 13298

Posted: May 17 2014 at 17:07

Ok - if this was a test I would definitely get partial credit for this answer, as I'm not 100% sure as to what the algorithm for the QWR is, but I'm fairly certain I have the general idea behind it, so I can conjecture a good guess. I think I covered all my bases there when all the math I'm about to do is horribly wrong.

Take a look here for a microcosmic look at how the QWR is calculated, kind of.

There are a few data points you need to keep in mind when fiddling with these numbers. This query, of the best prog albums of 1955 (lol) is very small with a small number of ratings for only 2 albums. We need to keep in mind the average number of ratings for all albums in this query (that being 8 ((13+3)/2)), the average rating for the query (4.04), and the rating and number of ratings for each album. Got those? Ok, lemme explain some stuff first -

The QWR is a method that PA uses to ensure that albums with a couple of 5 star reviews from collabs don't become the number one album on the list. When the top of the list says it uses the weighted average theory, it's kind of a rough weighted average - that is, the average number of ratings and average ratings are literally a 'weight' that holds down higher ascending highly rated albums, as you can see by Miles Davis and Milt Jackson having a QWR that's .02 lower than it's actual rating and even lower that the weighted average of the two (4.065).

So, with those four numbers in mind, think of it this way - if we weight the average of each individual album with the average of the aggregate query (8 average ratings, 4.04 average score), we can get a fairly fair QWR. One equation I found when doing some research on this topic was this:

(NxR)+(nxr)/(N+n)

N = average number of ratings

R = average score

n = individual number of ratings

r = individual score

If you punch in the four numbers for our 1955 example, it works, to a degree. The actual QWR varies slightly which could be due to either aggregation issues or slight adjustments to the curve, which is what I suspect it is. Max has tweaked the algorithm countless times over the years, and the elimination of ratings abusers has left some of the stats a little scarred.

When you apply this to your issue with 2013, of course we first come to the issue that it's difficult to ascertain the N and R for all of 2013, and even when solving the nasty systems of equations you can make with two albums, the numbers are uncertain. I got N=1193 and R=3.8608 for one combination of albums, which is clearly not correct seeing as SWilson's TRTRTS only has 995 reviews. My assumption (from the opinion of someone who's math education only goes up to Calc II) is that the N and R follow a curve that's probably logarithmic in some sense. When I dug through the source code in attempt to find the .js query that solves all this I couldn't find anything, so Max would have to be the one to ask about that. And interestingly enough, I think Dream Theater's album, the one that you said was #150 with 400+ ratings, has been booted from the list. That's another thing about this list - it's very volatile.

For the 2014 list and the album with 2 stars chilling in the top 150, I'll have to ask Max about that - it may just be a computational error or something was deleted that used to be there. Even with the formula and the curve that shouldn't be there.

I don't think anyone admits the system is perfect. With the dozens of little tweaks to the formula Max has made over the past ten years, there are bound to be errors in the variables.

Hope this answered your general question - I love spelunking through the caves of this site's history, so please say so if something was left untouched.

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Posted: May 17 2014 at 09:47

ok guys this problem is not solved yet! i am still convinced that we have huge bugs in the rating system!

It's just a ride... <3

Guldbamsen View Drop Down

Special Collaborator

Retired Admin

Joined: January 22 2009
Location: Magic Theatre
Status: Offline
Points: 23098

Posted: May 17 2014 at 05:51

No problem:-)

Next time I'm on my laptop, I will probably be able cook something up for you. Though, I'm guessing, not until next week. Denmark's biggest karneval launches on Saturday, so the weekend is definitely out for me:-)

“The Guide says there is an art to flying or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”

- Douglas Adams

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Posted: May 17 2014 at 05:43

thanks for you answer!
i also tend to think that i just got something wrong but i wont be satisfied until i figure out or someone tells me what exatcly is right about a higher qwr with 1 rating 2 stars compared to a lower qwr with 31 ratings and 3,69 stars.. sorry Wink

but you gave your best!

i hope some1 else is willing to explain this issue to me annoying pedant Tongue

It's just a ride... <3

Guldbamsen View Drop Down

Special Collaborator

Retired Admin

Joined: January 22 2009
Location: Magic Theatre
Status: Offline
Points: 23098

Posted: May 17 2014 at 05:19

I am on my useless phone for the next couple of days, but I actually wanted to do a search of Dean's posts on this particular matter, because I know I have read him on numerous occasions explaining it to perfection. Btw Dean is a retired admin on PA.
I think he would say that there is no bug, and the way you look at the rating algorithms basically is faulty. Sorry, but that is my analysis;-)

Maths interest me about as much as parking lots, so I have never looked into the mechanics of PA's rating system. Tell you the truth, most of the time I feel it's more of a hassle than anything else. The whole "ratings' thang:P
I guess it's nice to have some kind of musical bearing on PA users - you can always look at his/her ratings and read a little bit of personality from it. It's only when we get to the 'sportsmanship' part of the deal where the wheels start falling off the cart. Those people who suddenly decide to equate music with sports, and then proceed to go for the top slot, the wonderful gold medal on PA!!!!

Yep, completely ridiculous, but you'd be surprised to learn just how many who actually do this, whether intentionally or unintetionally. Some literally do believe that you can put sports and music in the same bag, and that, I feel, is a frightening thought. Music is supposed to be something else right? At least that's what I hope it is.

Edited by Guldbamsen - May 17 2014 at 05:45

“The Guide says there is an art to flying or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”

- Douglas Adams

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Posted: May 17 2014 at 04:51

ok but in the above mentioned cases they must have directly altered the QWR? did you look up the obvious spots i linked? it still seems like a bug to me...

and your right, ratings are not that important especially when it comes to the albums one personally loves most..
and yet i still like to see what most of you like and my sense of perfection wants a conclusive toplist Wink

It's just a ride... <3

Guldbamsen View Drop Down

Special Collaborator

Retired Admin

Joined: January 22 2009
Location: Magic Theatre
Status: Offline
Points: 23098

Posted: May 17 2014 at 04:29

I am not the most proficient in PA ratings system, but I do know that we erase a lot of bogus accounts, rating manipulators and the likes, but before the affected release receives a new rating after we've fondled with it, it still looks warped.
Anyway don't get too hung up on ratings - most of us aren't

“The Guide says there is an art to flying or rather a knack. The knack lies in learning how to throw yourself at the ground and miss.”

- Douglas Adams

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Posted: May 17 2014 at 04:05

yes, i know that.
but the weighted rating system applies before the qwr. e.g: 4.09 | 2 ratings | 1 reviews | 50% 5 stars

only then the QWR is calculated wich depends on average ratings, number of ratings compared to all the other albums in the list - the weight of reviews doesn´t play a role in this calculation anymore because it was already used to get to the average rating.

please look at the example of syndone, panic room and flor de loto - nr 134,135 and 136 now in this list
http://www.progarchives.com/top-prog-albums.asp?ssubgenres=&syears=2014&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=250&x=78&y=8#list

again: panic room has MORE average rating, More ratings and MORE reviews(if it would even count) than syndone´s album! then comes flor de loto with 1 lousy rating of 2 stars - after this album there are some 40 albums ALL of which have again ofc more average rating (between 2,79 and 3,63) then this album on position 136.
you can find more examples of this if you search the list in the later positions...

does noone see here a bug? or am i just too dumb now to get that there is everything ok in the list?^^

ok now i am totally doubting my brains!
i always thought the higher the rating the better and the more ratings/reviews the better (becuase its more representative)
looking in the all time toplist:
red (nr 8) and animals (nr 9) both 4.52, but red has less ratings
get to nr 11 to 15 (all 4.42):
number of ratings goes down from 11 to 12 to 13 to 14.. but 15 has more ratings (and way more reviews, dont tell me this is to take into account here anymore! ;) ) than position 14.

am i on to something or just plain stupid?
thank you so far... :)

It's just a ride... <3

someone_else View Drop Down

Forum Senior Member

VIP Member

Joined: May 02 2008
Location: Going Bananas
Status: Offline
Points: 23996

Posted: May 17 2014 at 01:56

A rating without a review counts for 1.

A rating with a review by a non-collab PA member counts for 10.

A rating with a review by a PA collaborator counts for 20.

For example, an album with 10 reviews, of which 4 are written by collabs, and 40 ratings without a review, puts in more weight than an album with 2 reviews and 132 ratings.

Then there is the Query Weighted Ratings system. There is a link to a Wiki entry about this on top of the ranking page. I have to study it sometime because I don't know the ins and outs of it yet. But the Query Weighted Rating may differ depending on the ratings of the albums participating in the list.

Mind_Drive View Drop Down

Forum Senior Member

Joined: September 06 2009
Location: Germany
Status: Offline
Points: 389

Posted: May 16 2014 at 12:11

sorry that im bothering you again but

lets look at the example of panic rooms incarnate (3.69 - 31 ratings) nr 133 on this list:
http://www.progarchives.com/top-prog-albums.asp?ssubgenres=&syears=2014&scountries=&sminratings=1&smaxratings=0&sminavgratings=1&smaxresults=250&x=78&y=8#list

so why is it, that Syndone´s Odysseás is nr 132 with less ratings (22) and less average points (3.68) ?
nr 134 is btw another album wich has 1 rating and 2 stars oO i just.. i am confused
is there something totally screwed up in the later parts of the toplists?

or take for example 2013:
nr149: 3,35 - 36 ratings
nr150: 3,46 - 432 ratings.... what?
http://www.progarchives.com/top-prog-albums.asp?ssubgenres=&salbumtypes=1&syears=2013&scountries=&sminratings=0&smaxratings=0&sminavgratings=1&smaxresults=250&x=74&y=8#list

do i just dont get the weighted mean? (weight of review/collaborators review should have been already taken into account for the average rating, no?)
help!^^

Edited by Mind_Drive - May 16 2014 at 16:09

It's just a ride... <3

someone_else View Drop Down

Forum Senior Member

VIP Member

Joined: May 02 2008
Location: Going Bananas
Status: Offline
Points: 23996

Posted: May 16 2014 at 10:07

^You're welcome