Decision Tree Classifier: How It Works and How to Train One
How a Decision Tree Decides
Think of a flowchart that the data drew for itself. Training looks at every column and every possible cut point and asks which single question splits the rows into the cleanest piles, where clean means each pile is mostly 1 label. Say you are sorting refund requests into approve, review, and deny, and the best first question turns out to be "is the amount over 200?". Under 200 is almost all approve, so that side is nearly done. Over 200 is a mix, so the tree asks the next best question inside that pile only, maybe "has this customer had more than 2 refunds before?", and keeps going down each branch until the piles are pure or it hits its depth limit.
A prediction is just walking the chart. A new request for 350 dollars from a customer with 3 prior refunds goes right at the first question, right at the second, and lands on deny. So basically the model means "I turned your past decisions into the rules you seem to have been following", and that is why people like it, the result looks like something a manager could have written down.
Because every question is about 1 column compared to 1 number, the size of the numbers does not matter at all. A column in dollars and a column that is 0 or 1 work side by side with no scaling, which is different from distance based classes like K Nearest Neighbors.
What a Decision Tree Is Best At
- Decisions that are really rules. Approvals, eligibility, routing, anything where the honest description is "if this and this, then that".
- Combinations of columns. "Large amount and new account" is 2 questions in a row, which a tree finds naturally and a weighted sum like Logistic Regression can not express.
- Mixed columns with no preparation. Counts, dollars, flags, and codes all go in as they are.
- Explaining 1 specific answer. The path the input took is the reason, and it is 3 to 10 plain comparisons.
- Finding out which columns matter. The questions near the top are the ones that separate your data the most, which is useful to know even if you end up using a different model.
Where It Breaks
- It memorizes when allowed to grow. A deep tree keeps asking questions until every training row has its own little branch, including the odd ones and the mislabeled ones. It then scores perfectly on the rows it trained on and noticeably worse on new ones.
- It is jumpy. Remove or add a handful of rows and the first question can change, which changes everything below it. 2 trees trained on almost the same data can look completely different.
- Hard edges. A request for 199 and one for 201 land on opposite sides of a cut even though they are basically the same request, there is no "close to the line".
- Text. Each word becomes a column and the tree can only ask about 1 word per question, so it needs a huge chain to cover normal language. Use Naive Bayes or logistic regression for text.
- Smooth trends. "The more they log in the less likely they cancel" is a slope, and a tree can only copy a slope with a staircase of cuts.
Parameters
{"maxHeight": 10}
maxHeight is the most questions the tree may ask on any path, and it is the whole control over memorizing. At 3 the tree is a short rule list you can read in 10 seconds and it can not memorize much. At 10, the default, it can describe up to 1,024 different end points, which is plenty for most data and already more than a person will read. Setting it to 0 removes the limit, and the tree grows until every pile is pure, which is almost never what you want on real data. The useful habit is to go lower than you think, a height of 4 or 5 often scores the same as 10 on new inputs and is far easier to trust.
How to Train a Decision Tree Classifier Step by Step
These steps use ML Prediction Engine, the free open source tool this topic uses for its examples, where the class is named tree. The machine learning models guide covers installing it.
1. Create a pipeline with the tree class
Create a pipeline, add one model step, set the category to classifier, the class to tree, and the input type to numeric. Set the parameters to {"maxHeight": 10} or leave them empty for the same default.
2. Lay out the columns
Write down the columns and their order, for example amount, account age in days, prior refunds, and paid by card as 1 or 0. Send raw values, no scaling. Every row needs every column, a row that is shorter than the others gets filled with 0 at the end, which the tree will read as a real 0, so do not leave gaps.
3. Add labeled rows
Use real past cases with the decision that was actually made, at least 30 per label, and make sure the borderline cases are in there, since those are the ones that teach the tree where the cuts go.
curl -X POST http://localhost:8080/api.php/dataset/add \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "modelIndex": 0, "input": [350, 12, 3, 1], "label": "deny"}'
4. Train the model
Click Train or call api.php/train with the pipelineID and modelIndex. Training is fast, a few thousand rows take well under a second, and the reply confirms the row count.
5. Test and lower maxHeight
Hold back 20 or more real cases that are not in the dataset and run them. Then set maxHeight to 5, retrain, run the same cases, and do it again at 3. Keep the smallest height that scores as well as the larger ones. If accuracy on fresh cases goes up when you lower the height, the taller tree was memorizing.
6. Predict from your app
curl -X POST http://localhost:8080/api.php/predict \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "input": [80, 400, 0, 1]}'
The answer is {"output": "approve", ...}. Keep the column order identical to training, and when a prediction is wrong, add that case with the right label and retrain.
When to Move Off a Single Tree
If the tree is accurate enough at a small height, keep it, a model people can read is worth a couple of points. If you need those points, the fix for both the memorizing and the jumpiness is the same, train many trees and combine them. Random Forest trains 100 trees on random slices of the data and takes the vote, which is steadier and usually more accurate, and AdaBoost builds small trees in sequence to correct each other. You give up the readable flowchart either way, so a common setup is a forest for the predictions and a height 3 tree trained on the same rows to explain roughly what is going on.
A decision tree turns your labeled cases into a chain of yes or no questions. Use it when the decision is really a set of rules and has to be explained, send raw numeric columns with no scaling, and keep maxHeight as low as your test cases allow so it learns the rules instead of memorizing the rows.