Ratings, my new method

This was originally posted in the collaborator's discussion forum, Micky felt general members should have a say on this as well.

Okay I basically worked on this all day and came up with some good stuff.

I basically took the formula mentioned above:

Of course some of this is inaccurate because I made up the minimum votes required to be in the top 100 and the arithmetic mean of the entire database. I feel 50 for a minimum number of required votes and a 3.5 mean seemed right. All these variables can be changed depending on what the collaborators feel is best.

By having the 3.5 thrown in there it's a little bit like Easy Livin's proposal, but not too radical. If we determine the average of every album on the site, we get an idea of what rating everything might come close to. This way it edges an album's overall rating toward that mean, but only just a little. This is mainly most effective for new albums and albums without a lot of ratings. It also helps balance the rating against biased votes (which is what we're trying to solve here right?) So, from what I can see we're lessening that problem, but also improving upon what we have already.

Another VERY positive thing I noticed is that new albums with only a few reviews will actually start out with lower ratings and climb as they grow more popular. Personally, I think this is a huge improvement over the previous system where albums would always start with a 5 and drop over time.

I think this is a great improvement on accuracy because now albums with only 5 reviews won't have a ridiculous rating. Say for example a new album only has 5 ratings. 2 of those are by collabs, 3 are not. 3 people rated it a 5 including one collab. 2 people gave it a 4, one of those also a collab. With this new method the album will only come to a rating of 3.66 instead of a weighted average of 4.55. Does this make more sense? What are your thoughts?

I also posted this in response to someone who had some concerns about the treatment of more obscure albums and less popular genres.

I do agree with that there are tons of albums out there that are just too obscure or inaccessible to get the recognition they deserve. However, I do not feel this is as much of a concern when it comes to ratings. Anyone delving into the obscure will realize that these albums won't have as high of a rating, and that's understandable. Because the truth is, the opinions of 200 people is just more trustworthy than the opinions of 10. People who have been exploring prog for a while will understand this.

As I've told Micky earlier, the ratings serve their main purpose to those new to prog, and the new proggers will be more likely to get into the more popular stuff before the obscure stuff. The ratings should reflect this. If we get tons of zeuhl and RIO stuff with ratings a good deal higher than everything else, the newbies will be thinking "huh?". You understand what I'm getting at?

Also as far as the algorithm goes, the number of ratings is one of the factors incorporated into. A minimum number of ratings for an album to be in the top 100, which also reflects the overall rating of the album. I have that number set at 50 at the moment. Depending on what number we agree on, this can be changed. The way it works is, the further below this number the amount of ratings is, the lower the overall rating will be. So if this number was set to 25 instead of 50, the overall rating would increase.

The thing about this new method is, every album can be compared solely by its rating, and a list can be compiled this way. There is no extra algorithm needed to compile a top 100, as the new method makes it so everything can be sorted purely by its rating. One of the improvements of it, in my opinion.

EDIT: I thought about this a little more and there are a few things that can be done, concerning your worries.

First of all, we can have not just one rating, but two. Their will be the overall rating displayed, which uses the new method, depending on number of reviews, average, etc. But we can also have the arithmetic mean displayed. By having two ratings displayed, people can see both sides. This way everyone is happy.

Secondly, I'm not sure about this, but as you said, some genres are more popular than others. Perhaps we can have a slightly different rating method for each genre. For example: I have it set to 50 ratings to be included in the top 100 (which factors into the overall rating). However, for Zeuhl, which is significantly less popular than symphonic prog, 50 ratings is much more impressive than 50 ratings in symph prog. Am I right? So perhaps, the amount needed for top 100 placement could vary between genres. Say it would be 50 for symph prog, while 30 for zeuhl, considering the popularity difference.

What do you think of this?

2ND EDIT:

I did some editing to the formula. I think you might like this, personally I find it pretty awesome. What I did was I set up separate values to input into the minimum # of ratings field for each genre. I came up with these numbers through their popularity. I figured this out by averaging the amount of votes for the top 20 albums of each genre. Through this I could see that symphonic prog was significantly more popular than every other genre, while indo prog was by far the least. I gave symph prog a value of 50, while everything else's values were decided through diving their averages by that of symph prog, then multiplying by 50.

For example: Symph prog is 50, Canterbury - 11, Eclectic - 27, Prog Metal - 24, Zeuhl - 5, and so on...

This basically means the symphonic prog genre is about 10 times more popular than the zeuhl genre, at least on this site. This is taken into consideration now when coming up with the overall score, so each album's rating can be compared to that of its genre. Thus no genre is favored over another, and each is considered equally when it comes to determining a rating.

I think each album's overall rating could be determined using this method, while for top 100 lists everything should stick to my previous method (the minimum # of ratings value staying at 50.)

Here's a visible representation of what I have going on ATM:



Anyone wishing to mess around with it themselves, I uploaded the excel file here



Edited by Fight Club - December 18 2007 at 12:42

Love what you have here Matt    I'll pm Pat to have a looksie if he hasn't broke away for the holidays .. as I will do in a couple of days myself

really like it...

back to work I go... I will look at that formula closer tonight.. got caught up in genre team business last night and didn't get to it.

y'know FC, I was ready to dismiss this whole ridiculous idea of yours but frankly results don't lie, and those new adjustments for the top albums are much more inline with reality IMO

very interesting, good work

Atavachron wrote: y'know FC, I was ready to dismiss this whole ridiculous idea of yours but frankly results don't lie, and those new adjustments for the top albums are much more inline with reality IMO very interesting, good work

I'm still tweaking it, but I think so too. The thing is, I have two separate rating methods, one to apply everything into a top 100, and then a genre specific method. Since each genre receives a different treatment I think it makes sense to have a different rating method. It's not too different, just takes in the amount of ratings differently.

For example: since RIO is a lot less popular than symphonic prog, the amount of ratings is a less important factor in the overall rating.

Also, that's not a COMPLETE list of top albums. A lot are probably still missing, as I only figured out the ratings to those that are there.

1800iareyay wrote: Ugh, you just had to make me look at math on my break didn't you?  Seriously though, this looks great as far as I can tell, but I have one question? Where do the ratings without reviews fit in? Maybe this could also be the solution to the issue Bob brought up about the spam ratings. Maybe if the ratings without reviews didn't count toward the overall rating, it would cease the pointless ratings. We could allow people to rate without reviews, but it would explicitly state that it would ahve no effect on teh overall score unless submitted with a written review. Keep up the good work Matt.

Well the way I have it now, ratings w/out reviews count just as much as non collab ratings with reviews. I did it this way, because at the moment I can't actually SEE all the ratings, nor can I see all the non collab reviews alone.

So in my method I gave collab/prog reviewer ratings a weight of 3 with everything else a weight of 1. The way I figured it out was so I didn't have to count all the ratings on a page I highlighted the block of collab reviews and opened the source code in firefox. I then copied that to a word document and told it to fine 5stars.gif, 4stars, and so on so I could tell how many of each there was. I can't do this with the other reviews ratings, because to see all non collab reviews, I also have to show all the collab reviews (therefore I cannot tell them apart in Word). And I just can't see non review ratings in general at all anymore

Of course if we can all decide on the specifics of this new method later. Obviously I'm sure whoever does this rating stuff has access to all the information we need. Then we can decide on weights between ratings, the overall mean, and stuff like that. Some of the factors I have in right now are still my own speculation/decisions.

This is better than the currant top list algorithm...

micky wrote: I like that a lot....  curious as to what the example album list (from above) might like like now.. using that latest tweek...

Haha... Well it'll take me a while to go through that sh*t load of albums again, but I'll see.

Only thing is, the minimum number of reviews (previously ratings) might have to be changed now since there will be a lot less ratings. Also, what about all the people who have rated TONS of albums, but never reviewed? All that time will be wasted now.


Edited by Fight Club - December 18 2007 at 18:14

I think that you have to remember that many of the ratings without reviews may have been done by non-English speaking members of the site who aren't comfortable enough with their English skills to write reviews in English.  It wouldn't be fair to exclude their reviews because they don't have written reviews.