If I understand correctly what you are aiming for, you might want to rearrange your data a little. You have for each year set of explaining variables in your example:
- Number of people who said it was good based on rumours
- Number of people who said it was ok based on rumours
- Number of people who said it was bad based on rumours
And the same for those where it was based on tasting. Which makes six columns for each year and if there were none saying something the value is zero. So you would have six times the years as input in this case and what you are predicting is 1,0 for profit/no-profit.
What it means is that "year" as such is not a predictor, but "number of people who did A in year X" is.
The bottom line is:
As soon as you use a portion of the data to choose which model performs better, you are already biasing your model towards that data.1
Machine learning in general
In general machine learning scenarios you would use cross-validation to find the optimal combination of your hyperparameters, then fix them and train on the whole training set. In the end, you would evaluate on the test set only to get a realistic idea about its performance on new, unseen data.
If you would then train a different model and select the one of them which performs better on the test set, you are already using the test set as part of your model selection loop, so you would need yet a new, independent test set to evaluate the test performance.
Neural networks
Neural networks are a bit specific in the sense that their training is usually very long, thus cross-validation is not used very often (if training would take 1 day, then doing 10 fold cross validation already takes over a week on a single machine). Moreover, one of the important hyperparameters is the number of training epochs. The optimal length of the training varies with different initializations and different training sets, so fixing number of epochs to one number and then training on all training data (training+validation) for this fixed number is not very reliable approach.
Instead, as you mentioned, some form of early stopping is used: Potentially, the model is trained for a long time, saving "snapshots" periodically, and eventually the "snapshot" with the best performance on some validation set is picked. To enable this, you have to always keep some portion of the validation data aside2. Therefore, you will never train the neural net on all of the samples.
Finally, there are plenty of other hyperparameters, such as the learning rate, weight decay, dropout ratios, but also the network architecture itself (depth, number of units, size of conv. kernels, etc.). You could potentially use the same validation set which you use for early stopping to tune these, but then again, you are overfitting to this set by using it for early stopping, so it does give you a biased estimate. Ideal would be, however, using yet another, separate validation set. Once you fix all the remaining hyperparameters, you could merge this second validation set into your final training set.
To wrap it up:
- Split all your data into
training + validation 1 + validation 2 + testing
- Train network on
training, use validation 1 for early stopping
- Evaluate on
validation 2, change hyperparameters, repeat 2.
- Select the best hyperparameter combination from 3., train network on
training + validation 2, use validation 1 for early stopping
- Evaluate on
testing. This is your final (real) model performance.
1 This is exactly the reason why Kaggle challenges have 2 test sets: a public and private one. You can use the public test set to check the performance of your model, but eventually it is the performance on the private test set that matters, and if you overfit to the public test set, you lose.
2 Amari et al. (1997) in their article Asymptotic Statistical Theory of Overtraining and Cross-Validation recommend setting the ratio of samples used for early stopping to $1/\sqrt{2N}$, where $N$ is the size of the training set.
Best Answer
Training on multiple records/observations from the same user/subject is ok, but you want your test data independent of your training data.
For example, you might imagine two approaches to constructing a test set (eg. for cross-validation):
If the records of subjects aren't independent, the two approaches may be extremely different, and one should almost certainly do the latter, selecting subjects at random to place in test set.
What can go wrong with record-wise test set construction?
To take an extreme example, imagine that all the records for each subject were exactly the same and each subject has numerous records. Then with record-wise validation, you'd be training on the test set! If your algorithm overfit the data, you'd get amazing performance on the test set but horrible performance when you actually see new, independent data.
Training and testing on the same set of users can give horribly misleading results that will not predict out of sample performance on new users.
Another example, here's a recent paper that discusses how record-wise cross-validation can go totally wrong in the clinical context: http://biorxiv.org/content/early/2016/06/19/059774.full.pdf+html