Learn AI Engineering Rent GPUs By The Hour Docker VPS Hosting Automate 3000+ Apps No Code AI Agents Proxies For Your Agents
Learn AI Engineering Rent GPUs By The Hour
Websites To LLM Data Proxies For Scraping AI Support Chatbot AI Data Analyst AI Agent Workspace Hire AI Builders

Gradient Boosting Regressor: How It Works and How to Train One

Updated November 2026
Gradient boosting predicts a number by building 100 small decision trees one after another, where each new tree predicts the error still left over from all the trees before it, and the final answer is the sum. On table data with a few hundred rows or more it is usually the most accurate regressor available, it needs no scaling, and it learns curves and combinations of columns by itself.

How Gradient Boosting Decides

Start with the dumbest possible prediction, the average of all the labels. Say you are predicting next week's order total for each store and the average is 8,000, so the model begins by predicting 8,000 for everyone. Obviously that is wrong for almost every store, one really did 11,000, so it is off by +3,000, another did 6,500, off by -1,500.

Now the first small tree is trained, and here is the trick, it is not trained on the order totals, it is trained on those leftover errors. It learns something like "stores with over 20 staff are under by about 2,000, the rest are over by about 600". The model adds a fraction of that correction to its prediction, a tenth of it by default, and recalculates what is still wrong. The second tree is trained on that new leftover, the third on what remains after the second, and after 100 rounds the prediction for a store is the starting average plus 100 small corrections.

So basically each tree means "here is a bit of what everyone before me got wrong". Because each one is only 3 questions deep it can not memorize much, and because each correction is taken at a tenth of its size, no single tree can drag the model anywhere, it gets there in many small steps. That is the "gradient" in the name, small steps in the direction that reduces the error.

This is the same family as AdaBoost, both build trees in sequence. The difference is what gets passed along, AdaBoost reweights the rows it got wrong, gradient boosting hands the actual leftover error to the next tree, which is what makes it work so naturally for numbers.

What Gradient Boosting Is Best At

  • Number prediction on table data. Sales, demand, order values, durations, scores, with the usual mix of counts, amounts, and flags.
  • Curves and thresholds. A tree can say "over 20 staff" in 1 question, so the model handles effects that kick in at a certain point without any help.
  • Columns that interact. A 3 question tree can express "weekend and promotion and large store" directly.
  • Raw columns. No scaling, and a junk column simply does not get picked for splits.
  • Accuracy. When a team compares regressors on real business data, this family wins more often than any other.

Where It Breaks

  • Small data. Under about 100 rows there is not enough leftover signal for 100 rounds of corrections, and Ridge Regression will usually match or beat it.
  • Past the training range. Like every tree based model it predicts by sorting inputs into groups it has seen, so an input bigger than anything in training gets the same prediction as the biggest one it knows. It will not follow a trend upward.
  • Noisy labels with too many rounds. Give it enough estimators and it starts correcting for noise, which shows up as great training error and worse held back error.
  • Explaining 1 prediction. It is a sum of 100 trees, there is no short reason.
  • Text. It is a table data tool, thousands of word columns do not suit it.

Parameters

{"estimators": 100, "rate": 0.1, "treeMaxHeight": 3}
  • estimators is the number of trees, meaning the number of correction rounds.
  • rate is the fraction of each tree's correction that gets applied, 0.1 means a tenth. Smaller steps are safer and usually end up more accurate, and they need more rounds to arrive.
  • treeMaxHeight is how many questions deep each tree goes. 3 means each tree can combine up to 3 columns. Use 2 for small or noisy data, and 4 or 5 when you have thousands of rows and columns that interact in complicated ways.

The first 2 are a pair, rate times estimators is roughly how far the model travels. Halve the rate to 0.05 and double the estimators to 200 and you cover the same distance in smaller steps, which is the most reliable single improvement to try. Going the other way, a rate of 0.5 with 20 estimators trains fast and is noticeably rougher.

How to Train a Gradient Boosting Regressor Step by Step

These steps use ML Prediction Engine, the free open source tool this topic uses for its examples, where the class is named gradientBoost. The machine learning models guide covers installing it.

1. Get a baseline from ridge

Create the pipeline with a regressor step set to ridge, load your rows, train, and measure the average miss on 30 or more held back rows. That number is what the boosted model has to beat, and it takes 2 minutes to get.

2. Switch the step to the gradientBoost class

On the same step change the class to gradientBoost and set the parameters:

{"estimators": 100, "rate": 0.1, "treeMaxHeight": 3}

The rows stay with the step, nothing needs reloading.

3. Add rows with every useful column

Be generous with columns, the model picks what it uses. For the store example: staff count, last week's total, the same week last year, a 0 or 1 for a running promotion, day count open, and local temperature. Send raw values and the real number as the label.

curl -X POST http://localhost:8080/api.php/dataset/add \
  -H "Content-Type: application/json" \
  -H "X-API-Key: YOUR_KEY" \
  -d '{"pipelineID": 1, "modelIndex": 0, "input": [24, 10400, 9800, 1, 7, 68], "label": 11000}'

A few hundred rows is where this class starts pulling ahead, and it keeps improving into the tens of thousands.

4. Train the model

Click Train or call api.php/train with the pipelineID and modelIndex. The trees are built in order, so training takes longer than ridge, about a second for a few thousand rows.

5. Test and tune rate with estimators

Measure the average miss on the same held back rows as the baseline. If it is not clearly better than ridge, the relationship is probably close to a straight line and ridge is the right model. If it is better, try {"estimators": 200, "rate": 0.05, "treeMaxHeight": 3} and keep whichever has the smaller miss. If the miss on held back rows gets worse as you add estimators, you have gone too far and it is fitting noise.

6. Predict from your app

curl -X POST http://localhost:8080/api.php/predict \
  -H "Content-Type: application/json" \
  -H "X-API-Key: YOUR_KEY" \
  -d '{"pipelineID": 1, "input": [12, 6100, 5900, 0, 7, 71]}'

The answer is a number, {"output": 6240.5, ...}. Add each real result as a row when it comes in and retrain on a schedule, weekly is plenty for most forecasting jobs.

When Something Else Is Better

If your inputs will regularly go beyond anything in the training data, a growing business where next year's numbers are bigger than any past year, a tree based model will hold the predictions down, and Ridge Regression follows the trend. A workable middle is to feed the boosted model columns that already carry the trend, like last week's total, so it predicts a change from a known level instead of the level itself. For a small dataset with a smooth curve, Support Vector Regression can be steadier. And for classification with the same kind of tree ensemble, see Random Forest.

Key Takeaway

Gradient boosting adds up 100 small trees that each correct what the earlier ones got wrong. Use it for predicting numbers from table data once you have a few hundred rows, always compare it against a ridge baseline, and for more accuracy halve the rate and double the estimators.