What are the available options if I want to perform a scoring task on a set of observations that: a) have a set of variables connected to them and, b) each round I get new information about the success of the latest round.
What I mean is, I start with a normal scoring task: Using the available training data I want to rank each observation in a new dataset. This part is straightforward and can naturally be performed by a range of different methods. However, the nature of problem is that I get feedback from this ranking when see how observations react. Specifically, when a number of observations are picked from those with the highest score, the feedback consists of these observation either reacting positively or negatively to them being picked. This information can in turn be used for the next round, which is performing a new scoring and picking the next observations with the highest scores.
What mathematical methods can be used to process this feedback information? How should/could the feedback be used in conjunction with the original (background) information? What R packages are there for these types of on-line machine learning problems?
Best Answer
It sounds like you want reinforcement learning. I'm having a little trouble parsing the exact details of your specific problem, but perhaps it could be cast in the framework of a Multi-Armed Bandit problem?