entropymachine learning
I have 4 class and four sets of features extracted from the huge data extracted from Real Time data acquisition system.
.
In above table, there are 4 sets of the features for each class. (C1, C2, C3 and C4). But when calculating information gain, information gain comes same for each features as for all attribute as there is only one value of each attribute for each class.
Can someone please help me in calculating a information gain for building decision tree.
Thanks
DDas
I am not sure there is a golden method to calculate the IG threshold.
I assume you have a two column vector (or map or dictionary, etc.) that contains words in col1 and their IG's in col2. Why not sort descending by IG, take top 10, 100, 1000, etc. and plot something like this:
This will give you a visually intuitive threshold point for your IG.
Further information provided on this link.
Any criteria which splits the data instances into disjunct regions might be used to compute any of the entropy related criteria. So, the split forms regions, and the criteria is computed on regions. Thus, for computing criteria it does not matter how do you find regions. And anyway, finding regions is a greedy approach, and as a consequence an heuristic.
The idea is that, at least for CART-like trees, this kind of splits are considered useless because if those regions makes sense in the perspective of the criteria used, it will be discovered anyway, by splitting on multiple nodes. Thus, the idea to have a split like $(x<3) or (x>7)$ was abandoned. Another reason is that it is computationally unfeasible.
Best Answer
In order to use Information Gain, you have to use a sort of sliding feature.
In this example, it's using GiniSplit but you can apply Entropy / InformationGain. You essentially sort the data ascending. Then for every distinct value, you create a split (Less than or equal to value vs. greater than value) and calculate the InformationGain on that split. Finally, choose the split that improves InformationGain the most.