Ratings, my new method

The Whistler wrote: That first post sounds like math. ... I hate math. I didn't start writing reviews so I could do math.

All you need to know is that it's about 10000000X better than the current formula

I'm trying to get this new algorithm as perfect as possible, so what I'd like to do is gather any concerns or problems anyone has with the rating system in general. Things you like and things you don't like, what should the ratings take into account, etc.

List your opinions here, all of them. I'm listening, and I'll address everything I can.

Something that came to mind was in regards to when general reviewers become reviewers or special collabs.  I think that I read somewhere that the additional weighting for reviewers and special collabs is only on their reviews after they become reviewers or special collabs.  The reviews that were done prior to becoming reviewers and special collabs still had the weighting as general reviewers.  I don't know if this is correct or not, but I thought that I read this somewhere before.  If so, I am not sure how this might effect your algorithms.

Easy Livin wrote: Would it be possible for those of us who are unfamiliar with statistical formulae to explain the logic behind each part of the formula, and the effect it has on the overall calculation in layman's terms?   Have you looked at what effect the proposed formula would have on albums with just over the minimum number of reviews required to enter the the top 50/100 (whatever)?   Does the formula automatically exclude those with less than the minimum number?   What purpose does the minimum number serve in the formula? (I guess I covered that in the first question).   Is this a recognised (i.e. published) formula used in other fields? (Our present one is).   Finally, just to clarify, Collabs reviews do not count as 10 times other members, they count as about 3.3 times. The "10 times" is the relationship to Ratings without Reviews (RwR), and is in response to the concerns frequently expressed about these. Ratings with reviews by non-collabs are weighted by 3 times when compared to RwR. Ratings without reviews will always have a place in the calculations, but their influence is small compared to written reviews. It would be as well to assume that these ratios will remain the same meanwhile.   Cheers!

The formula I came up with somewhat of a modified version of the ranking formula IMDB.com (Internet Movie Database) uses, which I tweaked a good bit to fit into Prog Archives.

IMDB's formula looks like this which can be found on their wikipedia page
 It expands upon credibility theory, explained in detail here

where:

= Weighted Rating

= average for the movie as a number from 0 to 10 (mean) = (Rating)

= number of votes for the movie = (votes)

= minimum votes required to be listed in the Top 250 (currently 1300)

= the mean vote across the whole report (currently 6.7)

I changed m and C to fit into prog archives, where I have the minimum as 50 since Prog Archives is a lot less popular than IMDB (unfortunately ). It could be set as an extra option to exclude any albums with less than 50 votes from the top 100, but the formula does not work that way. Instead the rating of anything having less than 50 votes will be affected more negatively than something in which the votes exceed 50.

As of right now, my method has the 50 vote aspect set only for a total top 100 list. I calculated separate "m"s for each genre, based on its popularity compared to the whole.

For example: symphonic prog is the most popular therefore, I set it's m at 50. Krautrock and RIO are significantly less popular so their m's are set at 4 and 7. This makes it so each album can be equally compared in their respective genre without suffering due to a genre's obscurity or lack of popularity.

However, I also have the top 100 rating set in as well, so each album gets two ratings. One for its genre, and one for the whole site. This way not only can everything be compared only within its genre, but all together for a more accurate top 100. This also makes it so each album can be compared solely by its rating having the highest rated albums at the top, with the lowest rated at the bottom.

As far as C goes, I find that this works out a lot of balance issues as well. C is the arithmetic mean of every rated item on the site. While IMDB has a C of 6.8, I made ours 3.5 since: firstly - I do not have access to the information required to figure out the actual total mean, and secondly - our rating scale is out of 5, and 3.5 seemed reasonable. If this formula is put into effect, of course C will probably be a little different.

What C does is it draws the overall rating somewhat towards the average rating of everything. This helps balance out the hater and fanboy ratings that either greatly drag an album down or raise it completely out of proportion. C does not affect albums with a lot of ratings as much as say, new albums with only 5 ratings.

I noticed this serves a few purposes solving some issues I have noticed people complaining about in the past. For one thing, it works wonders on newly released albums. Say a prog metal album is just released and 5 people decide to rate it, 3 giving it 5 stars, 2 - 4 stars. 2 people accompany their 5 star rating with a review, while 1 person accompanies their 4. With our current rating system, this album would probably have a really high rating somewhere around 4.6. Of course no one would take it seriously, because it has so few ratings. With the upgraded formula, this album would only receive an overall rating of 3.70 and a 3.86 within its genre. Personally I think this represents reality much better than the current system as the "C" variable takes into account what the rating is "probably closer to".

Ultimately, the more an album is rated, the more "essential" it will become, while its new it will still be considered merely "good".

Also, I know about the Collab reviews being 3.3 times that of other members reviews. But they are 10 times that of ratings without reviews, are they not? I have these current weights set into my present formula, and they can be changed simply with the click of a button.

Does this answer all of your questions?

Easy Livin wrote: Cheers FC, I'll need to try to digest that! The reason I mentioned the 1/3/10 weightings is because your original post seemed to imply that there was a differential of 10 throughout, and that you thought this excessive.   A question for others reading. Can you follow the proposal OK, or would you like it explained in more basic terms? (I don't mean that to sound as patronising as it might appear, I'm wondering if it's just me who's having trouble getting to grips with it!)   Rushy - No, when a member is upgraded, all their reviews count as Collaborator reviews.

Anything else you would like to know, Bob? And what is your overall opinion of all this?

Easy Livin wrote: My perception (which may be totally wrong) is that to some extent it starts with the desired answer and works out what the question is

Sorry, but I'm not quite sure what you mean by this.

The only thing "tweaked" in the formula must be C and m. Except for that, everything is as far as I can see and think of identical to IMDb. And I do think IMDb knows quite a lot about this stuff, it’s only the largest rating site online... I think this formula is great! It should be implemented as fast as possible!

By the way, what is wrong with tweaking a formula for the best results anyway? I do know you are worried this sort of thinking will lead less popular and well-known albums to lose terrain, anyhow it is not the case here, and like this site is now, albums with few votes, ergo being less popular and well-known than albums with more votes, should lose terrain. Anyhow, as I said, this formula is not of the kind that needs much tweaking at all.

Edited by Sofagrisen - December 22 2007 at 06:18

MikeEnRegalia wrote: Easy Livin wrote: I do want to understand it as I know M@x is always open to suggestions in this field. We have discussed the algorithm in the past at great length, with Fitzcaralldo and MikeER contributing a lot.   I'd be interested to hear what Mike thinks of this proposal compared to the current and previous formulae. I've been very busy this week and didn't have the time to check it out ... but I will have a look soon. Finally there's someone else who takes rating/ranking methods as seriously as I do!

Amen to that! I'm also going to be extremely busy in the following weeks, so I won't be on much. But I'm willing to put as much time as necessary into this at the beginning of the new year.

PS. Happy Holidays everyone!

 Let's keep the discussion going.

OK, well I finally have some free time after my long and extremely busy last week or so! So how's everything going here? I guess I didn't miss much! Alright, so what else do we need to discuss here?

In response to Mike, I'm not sure about testing the formula on your site, because I'm not very familiar with it. I made it pretty specific to progarchives, so I don't think it would reflect much similarity between how it would affect ratingfreak and progarchives.