Logistic Regression: How It Works and How to Train One
How Logistic Regression Decides
So first the name, because it confuses everyone. Regression normally means predicting a number, and this model does produce a number inside, a probability between 0 and 1, but what you get back is a label, so it lives with the classifiers. If you want an actual number back, that is Ridge Regression.
The way it works is a weighted sum. Say you are predicting if a customer will cancel, with 3 columns: logins in the last 30 days, open support tickets, and months as a customer. Training finds a weight for each one, maybe logins get -0.8 because more logins means less likely to cancel, tickets get +1.2, and months get -0.3. For a new customer the model multiplies each value by its weight, adds them up, and squashes the total into a probability. Over 50% and the answer is cancel, under and it is stay.
Text works the same way with a lot more columns. Every distinct word becomes a column, so the model learns a weight per word for each label, "refund" pushes hard toward billing, "crash" pushes toward technical, and "the" gets a weight near 0 because it shows up everywhere. That is the difference from Naive Bayes, which counts each word on its own, where logistic regression adjusts all the weights together until the training rows come out right, so 2 words that always appear together share the credit instead of both getting full credit.
What Logistic Regression Is Best At
- Text classification past the first few hundred rows. It is one of the strongest simple models for messages, reviews, and documents, and it is fast enough to retrain constantly.
- Yes or no questions from numeric columns. Will this lead convert, will this customer cancel, is this signup real.
- Problems where each column pushes in one direction. More tickets means more risk, more logins means less, that is exactly the shape it can learn.
- Stable behavior. A small change in the input gives a small change in the result, it does not flip on tiny differences the way a single tree can.
- Being the number to beat. If a random forest or a neural network only gains 1 point over it, keep the simple one.
Where It Breaks
- Columns that only matter in combination. If risk is high only when the amount is large and the account is new, but neither one alone means much, a weighted sum can not express that, and a tree based class finds it easily.
- Columns where the middle is special. If both very low and very high values mean trouble and the middle is fine, 1 weight can only say "more is worse" or "more is better", never both.
- Category codes sent as numbers. A column where 1 means US, 2 means UK, and 3 means Canada looks like "3 is three times as much as 1" to this model. Send a separate 0 or 1 column for each category instead.
- Very few rows per label. With 10 rows it will find weights that fit those 10 perfectly and mean nothing, K Nearest Neighbors is safer at that size.
Parameters
{"c": 1, "maxIter": 200}
ccontrols how freely the weights can grow, and it runs backward from what you would guess. A small value like 0.1 holds the weights down, which keeps the model simple and helps when you have few rows or many columns. A large value like 10 lets it fit the training rows more tightly, which helps when you have plenty of rows and the model is clearly too cautious. Leave it at 1 until testing says otherwise.maxIteris how many passes the fitting gets before it stops. 200 is enough for most data, raise it to 500 or 1000 if you have many labels or thousands of text rows and results look unfinished, it only costs training time.
How to Train a Logistic Regression Classifier Step by Step
These steps use ML Prediction Engine, the free open source tool this topic uses for its examples, where the class is named logisticRegression. The machine learning models guide covers installing it.
1. Create a pipeline with the logisticRegression class
Create a pipeline, add one model step, set the category to classifier and the class to logisticRegression, and choose text or numeric input. Set the parameters to {"c": 1, "maxIter": 200} or leave them empty for the same defaults.
2. Choose columns that push the answer one way
For numeric input, go through each column and ask "does more of this make one label more likely?". If the answer is yes it belongs, if the answer is "only sometimes" or "only together with another column" this model will not use it well. Turn categories into 0 or 1 columns. You do not have to scale anything, the engine standardizes every column for this class before fitting, so dollars and counts can sit side by side.
3. Add labeled rows
Aim for 50 or more rows per label, since there is a weight to learn for every column and each one needs evidence. A numeric row is a list in a fixed column order:
curl -X POST http://localhost:8080/api.php/dataset/add \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "modelIndex": 0, "input": [2, 4, 7], "label": "cancel"}'
That row says 2 logins, 4 open tickets, 7 months, and the customer cancelled. Text rows send a string as the input instead.
4. Train the model
Click Train or call api.php/train with the pipelineID and modelIndex. The reply confirms the row count, and training needs at least 2 different labels in the dataset.
5. Test and adjust c
Run 20 or more inputs that are not in the dataset and count the right answers. Then set c to 0.1, retrain, and run the same inputs, then 10. If 0.1 scores best you probably have more columns than your rows can support, and if 10 scores best you have room to add rows and let it learn more. If all 3 score about the same, the weights are not the limit and the next gain is in the data.
6. Predict from your app
curl -X POST http://localhost:8080/api.php/predict \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "input": [0, 6, 3]}'
The answer is {"output": "cancel", ...}. Send the columns in the same order as the training rows, and add any wrong prediction back as a row with the correct label.
When to Move Off Logistic Regression
The sign is specific: accuracy is flat as you add rows, and the misses are cases where a combination of columns decided the outcome. That is the limit of adding weighted columns together, and Random Forest is the next class to try because it learns combinations on its own. For text with thousands of rows per label where phrasing matters more than single words, the Deep Neural Network class is the step up. Change the class on the same pipeline step, retrain, and compare on the same test inputs.
Logistic regression learns one weight per feature and adds them up to choose a label. It is the dependable baseline for text and for yes or no questions on numeric columns, the engine scales the columns for you, and c at 0.1, 1, or 10 is the only tuning that usually matters.