Articles → MACHINE LEARNING → Variance Reduction Splitting Criteria In The Decision Tree For Regression Problems

Variance Reduction Splitting Criteria In The Decision Tree For Regression Problems

This article describes the variance reduction splitting criteria in the Decision Tree for Regression problems.

When Is This Criterion Used?

When the target variable is continuous, we use variance reduction.

Consider the following dataset.

We will follow these steps: -

Sort by continuous value.
Find the midpoint between two consecutive numbers.
Find the parent variance of the target variable.
For Each Midpoint: Compute Left And Right Variances, Weighted Average Variance, And Variance Reduction.
Conclusion.

To begin with, sort by the continuous value. For this article, we will sort by Size. In the above table, the data is already sorted.

The target variable is Price. The mean of the Price variable is 53: -

(30 + 40 + 50 + 65 + 80)/5 = 53

The parent variance is 316: -

[(30−53)²+(40−53)²+(50−53)²+(65−53)²+(80−53)²]/5

Midpoint	Left Leaf	Right Leaf	Weighted average variance	Variance reduction (Parent variance − Weighted variance)
900	[30] Mean = 30 Variance = (30 – 30)²/1 = 0	[40, 50, 65, 80] Mean = 58.75 Variance = (45 − 58.75)² + (50 − 58.75)² + (65 − 58.75)² + (80 − 58.75)²/4= 189.0625	(1/5)0 + (4/5)189.0625 = 151.25	316 – 151.25 = 164.75
1100	[30, 40] Mean = 35 Variance = (30 − 35)² + (40 − 35)²/2 = 25	[50, 65, 80] Mean = 65 Variance =(50 − 65)² + (65 − 65)² + (80 − 55)²/3 = 283.33	(2/5)25 + (3/5)283.33 = 172	316-172 = 144
1350	[30, 40, 50] Mean = 40 Variance= (30 − 40)² + (40 − 40)² + (50 − 40)²/3 = 66.67	[65, 80] Mean = 72.5 Variance = (65 − 72.5)² + (80 − 72.5)²/2=56.25	(3/5)66.67 + (2/5)56.25 = 62.5	316-62.5 = 253.5
1650	[30, 40, 50, 65] Mean = 46.25 Variance = (30 − 46.25)² + (40 − 46.25)² + (50 − 46.25)² + (65 − 46.25)²/4=167.1875	[80] Mean = 80 Variance = (80 – 80)2/1 = 0	(4/5)167.1875 + (1/5)0 = 133.75	316-133.75=182.25

1100 is the best midpoint to split.

Query/Feedback