KNN Regressor: How It Works and How to Train One
How a KNN Regressor Decides
It works exactly like the K Nearest Neighbors classifier with 1 change at the end. The model keeps every training row, and when a new input arrives it measures the distance to all of them and takes the closest k. The classifier then counts votes for a label, the regressor averages the numbers. Say you are estimating what a used laptop listing should sell for from its age in months, memory, and storage, and the 5 most similar past listings sold for 610, 640, 590, 700, and 660, the prediction is the average, 640.
That makes it a completely different kind of model from Ridge Regression. Ridge commits to 1 formula for the whole dataset, "every extra month takes 14 dollars off", everywhere. KNN commits to nothing, it only ever looks at the neighborhood around the input, so the effect of age can be steep for new machines and almost flat for old ones without anyone telling it. So basically ridge means "there is 1 rule and it applies everywhere" and KNN means "there is no rule, just look at the ones most like this".
There is no real training step. The rows get stored, and all the work happens at predict time, comparing the input against every stored row.
What a KNN Regressor Is Best At
- Pricing and estimating by comparables. Listings, quotes, job durations, anything a person would estimate by pulling up the last few similar ones.
- Local patterns. Effects that are different in different parts of your data, which a single formula smooths over.
- Curves with no setup. It follows any shape the data has without you describing the shape.
- Answers that are always realistic. An average of real past values can not come out negative or absurd.
- Explaining the number. "Here are the 5 it was based on" is something anyone can check.
Where It Breaks
- It can never go past what it has seen. The prediction is an average of training values, so if your most expensive row is 2,000 it will never predict 2,500, even when the input is clearly bigger than anything before. For trends that keep going, ridge is the one that follows the line.
- Thin areas. If the input lands where you have few rows, the 5 "nearest" may not be near at all, and the average is of things that are not really comparable. It gives no warning when that happens.
- Columns that do not matter. Every column counts equally toward the distance, so a few useless ones make the wrong rows look close.
- The edges get pulled inward. At the top of your range all the neighbors are below the input, so the estimate comes out low, and at the bottom it comes out high.
- Big datasets. Every prediction scans every row, so response time grows with the data.
Parameters
{"k": 5}
k is how many neighbors get averaged. A small k like 2 or 3 tracks the data closely but 1 unusual sale moves the prediction a lot. A large k like 15 is steadier but starts averaging in rows that are not that similar, and it makes the pull toward the middle at the edges worse. For the regressor there is no tie to worry about, so odd or even does not matter. As a rough guide use 3 to 5 with under 200 rows and 5 to 15 with thousands, and let the measured error on held back rows make the final call.
How to Train a KNN Regressor Step by Step
These steps use ML Prediction Engine, the free open source tool this topic uses for its examples, where the class is named knnRegressor. The machine learning models guide covers installing it.
1. Create a pipeline with the knnRegressor class
Create a pipeline, add one model step, set the category to regressor, the class to knnRegressor, and the input type to numeric. Leave the parameters empty or set {"k": 5}.
2. Keep only the columns that define similar
Ask what makes 2 cases comparable for the thing you are estimating, and send only those. For the laptop example that is age, memory, and storage, and not the listing ID or the hour it was posted. 3 to 8 good columns is the comfortable range. The engine scales the columns for this class automatically, so months and gigabytes can go in as raw values without one drowning out the other.
3. Add rows that cover the range
The label is the number that actually happened. Because the model can only average what it holds, coverage matters more than count, 100 rows spread across cheap, average, and expensive cases beat 500 rows that are all average.
curl -X POST http://localhost:8080/api.php/dataset/add \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "modelIndex": 0, "input": [18, 16, 512], "label": 640}'
4. Train the model
Click Train or call api.php/train with the pipelineID and modelIndex. It finishes almost immediately since there is nothing to fit, and the reply confirms how many rows are stored.
5. Test and choose k
Predict 20 or more held back rows, take the difference between each prediction and the real value ignoring the sign, and average them to get the typical miss. Repeat with k at 3 and at 10 and keep the smallest. Then look at the misses separately for the highest and lowest cases, if those are much worse than the middle, you are seeing the pull toward the center and need more rows at the ends.
6. Predict from your app
curl -X POST http://localhost:8080/api.php/predict \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "input": [6, 32, 1024]}'
The answer is a number, {"output": 1180.0, ...}. Add every real outcome back as a row, for this class a new row improves predictions in its neighborhood the moment it is trained.
When to Move Off the KNN Regressor
If inputs regularly fall outside the range of your training data and the estimates come out too conservative, switch to Ridge Regression, which follows the trend past the edges. If you have many columns and are not sure which matter, Gradient Boost works that out for itself and is usually more accurate once you have a few hundred rows. And if predict time is growing with the dataset, either of those fixes it since they do their work at training time. The rows stay with the pipeline step, so trying another class is a change of name and a retrain.
A KNN regressor averages the values of the 5 most similar past cases. Use it when comparables are the natural way to estimate, send only the columns that make cases similar, cover the full range with your rows, and remember it can never predict beyond the highest or lowest value it has seen.