Solved – How to estimate a “confidence” in a score / rating

ratingvariance

I'm sure there's a more technical term for what I'm looking for, but I don't know what it is (it's not confidence intervals, I think)

I'm trying to make a system where people get ratings (from 1 to 5 stars) over time, and I want to combine all those ratings into one individual score for each person. Now, this score alone is not a very good thing, I'd like to have a second value that shows how "reliable" / "accurate" this score is.

Let's say that:

  • If someone gets 1 rating of "3", his score is "3", but we're not very sure of that.
  • If someone gets 10 ratings of "3", we're very confident he's a "3". Let's say that more than 10 ratings that are identical are not a stronger signal than 10 ratings, there's a point where we're "confident enough". Let's call this 100% confident.
  • If someone gets 2 x"1", and 2 x "5", the average is also a 3, but the person is all over the place, and I want to reflect this by giving this score a very low confidence value.

So, I'm pretty much looking for something that reflects variance, but also reflects how many "consistent" ratings someone has got. Or how "stable" he is at his score.

Can anyone think of either an algorithm for doing something like this, or point me in the direction of tools that you'd use for such a thing?

Thank you!
Daniel

Best Answer

Evan Miller's how not to sort by average rating has pretty good advice for the general problem you are trying to solve.

Related Question