I'm a new guy here. Hopefully, I'm asking this question to right forum.
Problem:
We have data of a group of people (P1, P2, P3). They rank their expertise (1-10, where higher number is better) in a list of components (G1, G2, G3).
P1 P2 P3
--------------
G1 | 8 4 7
G2 | 7 3 7
G3 | 9 6 5
Also, we have some data regarding work done by each person in each component. Example:
For P1,
W WD
----------
G1 | 0 0
G2 | 2 0
G3 | 8 2
where W is total work allotted to user P1, and WD is actual work done. W >= WD >= 0.
We have similar data for P2, and P3 users.
Point to note: The user might have some level of expertise regardless of work done in a component. Example: P1 has ranked himself 8/10 even though he has not been given any task in G1 component (W = 0). P1 also has ranked himself 7/10 for G2 even though he has not finished any task in that component (WD = 0).
Now, we want to calculate effective rating of all users relative to the group of users, not self-ranking, considering their work data and self-ranking.
Can anyone suggest some mechanism to achieve this?
Thanks much in advance!
Best Answer
It sounds like you have multiple sources of information about measurements of interest. The aim is to combine the multiple sources so as to maximise measurement accuracy. In general you could adopt a formula that weights the different sources of information based on the confidence you ascribe to the source.
It also seems that the confidence in the source might vary. E.g., the more work data you have on someone, perhaps the more weight this information would be given for that individual. You may also have evidence on the accuracy of an individual's self ratings.
Thus, a simple strategy would to develop a weighted composite of available information where the size of the weight is relative to the informativeness of the source.
This question on Bayesian rating systems might also be relevant.