Articles → MACHINE LEARNING → Information Gain In Machine Learning

Information Gain In Machine Learning

This article describes information gain in Machine Learning.

Purpose

Information Gain tells us which characteristic or property helps us best split a group into smaller, more organized groups.

If the value of Information Gain is higher, the attribute is a good splitter or vice versa.

Consider the following table: -

ID	Outlook	Temperature	Humidity	Windy	Play Tennis
1	Sunny	Hot	High	False	No
2	Sunny	Hot	High	True	No
3	Overcast	Hot	High	False	Yes
4	Rain	Mild	High	False	No
5	Rain	Cool	Normal	False	No
6	Rain	Cool	Normal	True	No
7	Overcast	Cool	Normal	True	Yes
8	Sunny	Mild	High	False	Yes

We have 8 instances, and the target variable is Play Tennis with values Yes or No. In the above dataset: -

The entropy of the dataset is: -

The option Sunny has three instances, i.e., one yes and two no.

The option overcast has two instances, i.e., two yes and zero no.

Entropy (Overcast)=0

The option rain has three instances, i.e., zero yes and three no.

Entropy (Rain)=0

Query/Feedback