Ridge Regression: How It Works and How to Train One
How Ridge Regression Decides
The model is a formula you could write on a napkin. Say you want to predict what a new lead will be worth in dollars from 3 columns: pages viewed, emails opened, and company size. Training finds a weight for each column plus a starting value, and a prediction is just the sum, the starting value plus pages times its weight plus opens times its weight plus size times its weight. If the weights come out as 12 per page, 30 per open, and 2 per employee on top of a base of 100, a lead with 10 pages, 4 opens, and 50 employees is predicted at 440.
That much is plain linear regression, and the ridge part fixes its most common failure. When 2 columns move together, pages viewed and time on site for example, plain regression can not tell which one deserves the credit, so it can land on something silly like +500 for one and -480 for the other. Those cancel out on the training rows and then go badly wrong on a new row where the 2 columns do not line up the usual way. Ridge adds a cost for large weights, so given a choice it shares the credit, something like +10 each. So basically the penalty means "don't use big numbers when small ones explain the data just as well".
A single parameter, alpha, sets how strong that preference is, and at the default of 1 it is a light touch that steadies things without changing the answers much.
What Ridge Regression Is Best At
- Proportional relationships. Lead value from engagement counts, shipping cost from weight and distance, time to finish from the number of items.
- Small datasets. It has 1 weight per column, so 50 rows can be enough to pin down a model with 5 columns.
- Columns that overlap. That is the exact case the penalty was built for.
- Going past what it has seen. If the trend is straight, it will follow the line to an input bigger than anything in training, which the neighbor and tree based regressors can not do.
- A baseline for every number prediction. Its error is the bar that Gradient Boost has to clear.
Where It Breaks
- Curves. If the output rises fast and then levels off, or goes up and comes back down, a straight line is wrong on both ends no matter how good the weights are.
- Combinations. If a discount only raises sales when stock is also high, a sum of separate weights can not express "only together".
- Outliers in the label. One row with a deal worth 50 times the usual drags the whole line toward it. Check the largest labels before training.
- Impossible answers. Nothing stops a line from going below 0, so a lead with all zero inputs and a negative base gets a negative value. Clamp the result in your code where that matters.
- Category codes sent as numbers. A plan column with 1, 2, and 3 gets treated as "3 is three times 1". Send a 0 or 1 column per category instead.
Parameters
{"alpha": 1}
alpha is the strength of the penalty on large weights. At 0.1 the model fits the training rows more closely, which is fine when you have many rows and few columns. At 10 or 100 it pulls all the weights toward 0, which helps when you have few rows, many columns, or columns that overlap heavily. Note that the direction is the opposite of c in Logistic Regression, here a bigger number means a more cautious model. The alpha on the neural classes is the same idea.
How to Train a Ridge Regression Model Step by Step
These steps use ML Prediction Engine, the free open source tool this topic uses for its examples, where the class is named ridge. The machine learning models guide covers installing it.
1. Create a pipeline with the ridge class
Create a pipeline, add one model step, set the category to regressor, the class to ridge, and the input type to numeric. Leave the parameters empty or set {"alpha": 1}.
2. Choose columns with a straight line relationship
For each column ask "if this doubles, does the answer move by a steady amount?". Counts and amounts usually pass. If a column clearly curves, you can help the model by sending a transformed version, the square root or the log of a column that grows fast is a common fix. You do not need to scale anything, the engine standardizes the columns for this class before fitting.
3. Add rows with a numeric label
For a regressor the label is the number that actually happened, and it has to be a number, not text. Keep the units the same on every row, all dollars or all thousands, never a mix.
curl -X POST http://localhost:8080/api.php/dataset/add \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "modelIndex": 0, "input": [10, 4, 50], "label": 440}'
Start with 50 or more rows, and roughly 10 rows for every column is a reasonable floor.
4. Train the model
Click Train or call api.php/train with the pipelineID and modelIndex. This is one of the fastest classes to train, even tens of thousands of rows finish in a blink.
5. Test by measuring the error
A regressor is never exactly right, so you measure how far off it is. Predict 20 or more held back rows, subtract each prediction from the real number, ignore the sign, and average them. That is your typical miss in real units, "off by 60 dollars on average". Compare it to the simplest possible guess, always predicting the average label. If ridge is not well ahead of that, the columns do not carry the answer. Then try alpha at 0.1 and 10 and keep whichever gives the smallest miss.
6. Predict from your app
curl -X POST http://localhost:8080/api.php/predict \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "input": [22, 9, 120]}'
The answer is a number, {"output": 874.0, ...}. When the real value comes in later, add it as a new row, that is how the model keeps up as your business changes.
When to Move Off Ridge
Look at where the misses are. If the model is too low on small inputs, too high in the middle, and too low again at the top, the real shape is a curve and a line can not follow it. With a few hundred rows of table data, Gradient Boost is the usual upgrade and learns curves and combinations by itself. With less data and a smooth curve, Support Vector Regression is worth a try, and when similar past cases are the best guide, the KNN Regressor. If the misses look random with no pattern, a different model will not help, the answer depends on something that is not in your columns.
Ridge regression adds up weighted columns to predict a number and keeps the weights small so overlapping columns do not throw it off. Start every numeric prediction with it, measure the average miss on held back rows, and move to gradientBoost when the misses show a curve.