Boosting · /xgboost

XGBoost-style boosting

Boosting builds the model piece by piece: each Step fits one small tree to the current residuals, shrinks it by η, and adds it to F. Watch the curve assemble; the Internals tab shows the tree fitting this round’s residuals and lets you scrub F₀ … F_M.

curve 120 samples · seed 42

datafilled = train, hollow = testhover to probe · click to pin

step 0MSE —train RMSE —test RMSE —

Data

datasetpoints150noise0.15seedsplitmouse

Additive model

F_{m} (x) = F_{m - 1} (x) + η f_{m} (x) = F

Each round adds one small tree, scaled by η, on top of everything so far.

Residuals (gradients)

g_{i} = F (x_{i}) - y_{i} h_{i} = 1 \frac{1}{n} \sum_{i} ∣ g_{i} ∣ = \overset{g}{ˉ}

The next tree is fit to these — for squared loss they are literally the residuals.

Leaf weight (Chen & Guestrin eq. 5)

w^{*} = - \frac{\sum _{i} g _{i}}{\sum _{i} h _{i} + λ} = w^{*}

Split gain (Chen & Guestrin eq. 7)

Gain = \frac{1}{2} [\frac{G _{L} ^{2}}{H _{L} + λ} + \frac{G _{R} ^{2}}{H _{R} + λ} - \frac{( G _{L} + G _{R} ) ^{2}}{H + λ}] - γ = Δ

Move the λ and γ sliders and watch this number — and the trees — respond.

What's simplified here

Second-order (g, h) boosting on squared and logistic loss only — the two losses this site teaches.
Level-wise growth; no column subsampling, no min-child-weight in v1.
The gain and leaf-weight formulas are the real XGBoost ones (Chen & Guestrin eqs. 5, 7) with λ and γ bound from the sliders.