Try first

You have twenty decision trees. Each one gets about 65 percent of predictions right, which is barely better than guessing.

Can you combine them into something substantially better than 65 percent? If so, what has to be true about the twenty?

Voting works, under one condition

Take a majority vote. If each tree is right 65 percent of the time and their mistakes are independent, the majority of twenty is right about 90 percent of the time.

The reason is that for the vote to be wrong, at least eleven trees must be wrong at the same time, and independent errors rarely coincide like that.

The condition is doing all the work. Twenty identical trees vote identically and you gain nothing. The value comes entirely from the trees being wrong about different rows.

So the engineering problem is not building good models. It is building models that fail differently.

Making trees differ on purpose

Lesson 3107 ended by calling instability a weakness: change one row and you get a different tree. That is now the mechanism.

function bootstrap(data) {
  return Array.from({ length: data.length },
    () => data[Math.floor(Math.random() * data.length)]);
}

function randomForest(data, nTrees = 20, maxDepth = 6, featureFrac = 0.6) {
  const nF = data[0].x.length;
  const take = Math.max(1, Math.round(nF * featureFrac));
  const trees = [];
  for (let t = 0; t < nTrees; t++) {
    const cols = shuffled([...Array(nF).keys()]).slice(0, take);
    trees.push({ cols, tree: grow(bootstrap(data), gini, 0, maxDepth) });
  }
  return trees;
}

function forestPredict(forest, x) {
  const votes = {};
  for (const { tree } of forest) {
    const v = predictTree(tree, x);
    votes[v] = (votes[v] || 0) + 1;
  }
  let best = null, bestN = -1;
  for (const k in votes) if (votes[k] > bestN) { bestN = votes[k]; best = k; }
  return { label: best, agreement: bestN / forest.length };
}

Two sources of difference. bootstrap samples rows with replacement, so each tree sees a different sample, with some rows repeated and about a third left out. And each tree only gets a random subset of the columns, so it cannot rely on the same dominant feature every time.

That second one matters more than it looks. Without it, if one feature is strongly predictive, every tree splits on it first and they all end up similar. Hiding it from some trees forces them to find other signals, which is exactly the diversity the vote needs.

This is a random forest, and the agreement figure is a useful confidence measure: 20 out of 20 is a much stronger claim than 11 out of 20.

Two ways to combine

Bagging is what a forest does. Train many models independently on different samples and average them. The models are equals, they can be trained in parallel, and the effect is to reduce the variance caused by any single model overreacting to its particular data.

Boosting is sequential. Train one weak model. Find the rows it got wrong. Train the next model to focus on those. Repeat, and add the models up with weights.

prediction = model1 + model2 + model3 + ...

Each model is fitting what the ones before it left over. That is why the trees are kept deliberately shallow, often just a few levels: each is a small correction, not an attempt at the whole answer.

Gradient boosting, which fits each new model to the gradient of the loss left by the current ensemble, is the standard tool for tabular data. XGBoost and LightGBM are implementations of it, and they win the majority of competitions on tabular problems.

The cost

You gave something up and it should be stated plainly. A single tree is readable. A hundred trees voting is not, and one of the main reasons to use a tree has just been discarded.

Feature importance is what survives: count how often each feature was chosen for a split, and how much gain it produced. That tells you which columns mattered. It does not tell you why any single prediction came out the way it did.

What to watch

Ensembles are not exempt from anything in section 9. Boosting can overfit badly if you add too many models, and the number of rounds is a hyperparameter chosen on the validation set like any other. Forests are more forgiving, because averaging independent models does not overfit in the same way, but they will still memorize if the trees are grown deep enough.

Exercises

1. Compare one tree at depth 6 against a forest of 20 on a validation split. Report both.
2. Sweep featureFrac from 0.2 to 1.0. Confirm that using every feature is worse, and explain why.
3. Measure accuracy against forest size from 1 to 100 trees. Where does it stop improving?
4. Implement feature importance by counting splits weighted by gain, and check it against what you know about the fruit data.