Articles → MACHINE LEARNING → Decision Tree Algorithm In Machine Learning

Decision Tree Algorithm In Machine Learning

This article describes the Decision Tree Algorithm in Machine Learning.

Purpose

The Decision Tree Algorithm is a supervised learning algorithm. It divides the dataset into branches based on feature (column or attribute in the dataset) values until it makes a decision.

Example

Consider the following dataset: -

Outlook	Temperature	Humidity	Wind	Play Tennis
Sunny	Hot	High	Weak	No
Sunny	Hot	High	Strong	No
Overcast	Hot	High	Weak	Yes
Rain	Mild	High	Weak	Yes
Rain	Cool	Normal	Weak	Yes
Rain	Cool	Normal	Strong	No
Overcast	Cool	Normal	Strong	Yes
Sunny	Mild	High	Weak	No
Sunny	Cool	Normal	Weak	Yes
Rain	Mild	Normal	Weak	Yes
Sunny	Mild	Normal	Strong	Yes
Overcast	Mild	High	Strong	Yes
Overcast	Hot	Normal	Weak	Yes
Rain	Mild	High	Strong	No

The requirement is to split the dataset until we reach a decision.

Entropy Of The Dataset

The entropy of the complete dataset is 0.94.

Calculation Of Entropy And Information Gain At Level 1

In the dataset, we have four attributes, i.e., Outlook, Humidity, Temperature, and Wind. We will calculate the Entropy and Information Gain of all four attributes.

Outlook	Entropy
Sunny	0.971
Overcast	0
Rain	0.971
Entropy of Outlook	0.693
Information Gain of Outlook	0.247

Humidity	Entropy
High	0.985
Normal	0.592
Entropy of Humidity	0.789
Information Gain of Humidity	0.151

Temperature	Entropy
Hot	1
Mild	0.918
Cool	0.811
Entropy of Temperature	0.911
Information Gain of Temperature	0.029

Wind	Entropy
Weak	0.811
Strong	1
Entropy of Wind	0.892
Information Gain of Wind	0.048

The Information Gain of the Outlook attribute is the highest, i.e., 0.247. So, we will split the dataset based on Outlook.

Find The Entropy And Information Gain For Outlook = Sunny

Find The Entropy And Information Gain For Outlook = Sunny And Temperature

Outlook	Temperature	Play Tennis
Sunny	Hot	No
Sunny	Hot	No
Sunny	Mild	No
Sunny	Cool	Yes
Sunny	Mild	Yes

Temperature	Entropy
Hot	0
Mild	1
Cool	0
Entropy of Outlook = Sunny + Temperature	0.4
Information Gain of Outlook = Sunny + Temperature	0.571

Find The Entropy And Information Gain For Outlook = Sunny And Wind

Outlook	Wind	Play Tennis
Sunny	Weak	No
Sunny	Strong	No
Sunny	Weak	No
Sunny	Weak	Yes
Sunny	Strong	Yes

Wind	Entropy
Strong	1
Weak	0.918
Entropy of Outlook = Sunny + Wind	0.95
Information Gain of Outlook = Sunny + Wind	0.021

Find The Entropy And Information Gain For Outlook = Sunny And Humidity

Outlook	Humidity	Play Tennis
Sunny	High	No
Sunny	High	No
Sunny	High	No
Sunny	Normal	Yes
Sunny	Normal	Yes

Humidity	Entropy
High	0
Normal	0
Entropy	0
Information Gain	0.971

Find The Entropy And Information Gain For Outlook = Overcast

Find The Entropy And Information Gain For Outlook = Overcast And Temperature

Outlook	Temperature	Play Tennis
Overcast	Hot	Yes
Overcast	Cool	Yes
Overcast	Mild	Yes
Overcast	Hot	Yes

Temperature	Entropy
Hot	0
Mild	0
Cool	0
Entropy	0
Information Gain	0

Find The Entropy And Information Gain For Outlook = Overcast And Wind

Outlook	Wind	Play Tennis
Overcast	Weak	Yes
Overcast	Strong	Yes
Overcast	Strong	Yes
Overcast	Weak	Yes

Wind	Entropy
Strong	0
Weak	0
Entropy	0
Information Gain	0

Find The Entropy And Information Gain For Outlook = Overcast And Humidity

Outlook	Humidity	Play Tennis
Overcast	High	Yes
Overcast	Normal	Yes
Overcast	High	Yes
Overcast	Normal	Yes

Humidity	Entropy
High	0
Normal	0
Entropy	0
Information Gain	0

Find The Entropy And Information Gain For Outlook = Rain

Find The Entropy And Information Gain For Outlook = Rain And Temperature

Outlook	Temperature	Play Tennis
Rain	Mild	Yes
Rain	Cool	Yes
Rain	Cool	No
Rain	Mild	Yes
Rain	Mild	No

Temperature	Entropy
Hot	NA
Mild	0.918
Cool	1
Entropy	0.951
Information Gain	0.02

Find The Entropy And Information Gain For Outlook = Rain And Wind

Outlook	Wind	Play Tennis
Rain	Weak	Yes
Rain	Weak	Yes
Rain	Strong	No
Rain	Weak	Yes
Rain	Strong	No

Wind	Entropy
Strong	0
Weak	0
Entropy of Outlook	0
Information Gain	0.971

Find The Entropy And Information Gain For Outlook = Rain And Humidity

Outlook	Humidity	Play Tennis
Rain	High	Yes
Rain	Normal	Yes
Rain	Normal	No
Rain	Normal	Yes
Rain	High	No

Humidity	Entropy
High	1
Normal	0.918
Entropy	0.95
Information Gain	0.021

Conclusion

At level 2, we get all pure nodes. So, the conclusion is: -

Outlook	Wind	Play Tennis
Rain	Weak	Yes
Rain	Strong	No

Outlook	Play Tennis
Sunny	No
Sunny	Yes
Outlook	Play Tennis
Overcast	Yes

Posted By -	Karan Gupta

Posted On -	Friday, September 12, 2025

Query/Feedback

Your Email Id		**

Subject		*

Query/Feedback	Characters remaining 250	**