Robust Z Score: How It Works and How to Set One Up
How the Robust Z Score Decides
Start with the ordinary Z score, since this is the fixed version of it. The ordinary one takes the average of your readings and their standard deviation, and says how many standard deviations a new value is from the average. More than 3 and it is flagged. The problem is that both the average and the standard deviation get dragged around by the very outliers you are trying to find. Say response times are normally around 200 milliseconds and last week there were a few readings of 9,000 during an incident. Those few readings pull the average up and blow the standard deviation up, so now "normal" looks like a huge range and a reading of 2,000 no longer looks unusual. The outliers hid themselves.
The robust version swaps in 2 measures that do not move when a few values go wild. Instead of the average it uses the median, the middle value when you sort the readings, which stays at 200 no matter how extreme the top few are. Instead of the standard deviation it uses the median absolute deviation: take how far each reading is from the median, ignoring the sign, and find the median of those distances. If most readings are within 20 of the median, that number is 20, and a few readings of 9,000 do not change it either.
The score for a new value is then:
The 0.6745 is a fixed constant that makes the result read like an ordinary Z score on bell shaped data. With a median of 200 and a deviation of 20, a reading of 320 scores 0.6745 times 120 divided by 20, which is 4.05, over 3.5, so it is flagged and the answer is 1. A reading of 260 scores 2.02 and passes with 0.
Turned around, the default threshold means a value is flagged when it is more than about 5.2 deviations from the median, since 3.5 divided by 0.6745 is 5.19. In the example that is anything under 96 or over 304. So basically the whole model is "normal is 200 give or take 104, anything outside that is an alert", and you can say exactly that to whoever gets the alert.
What the Robust Z Score Is Best At
- Watching a metric. Response time, daily signups, error counts, queue length, order totals, cost per run.
- Alerts that need a reason. "It was 320 and the normal range is 96 to 304" is a complete explanation.
- History that already contains some bad readings. Unlike most detectors, it does not need clean training data, a few past incidents in the history barely move the median.
- Small amounts of data. 50 readings give a usable median and deviation.
- Catching both directions. Too low is flagged the same as too high, so a sudden drop in traffic is caught as well as a spike.
Where It Breaks
- Combinations of columns. Each column is checked by itself and the row is flagged if any single one is out of range. It can not see "each value is fine alone but this pair never happens together", which is what Isolation Forest is for.
- Metrics that mostly do not change. If more than half the readings are the same number, an error count that is 0 most days, the median absolute deviation is 0 and any other value at all gets flagged. Use a metric that varies, or a simple fixed limit for that one.
- Regular cycles. Traffic that is 500 at noon and 40 at 3am has 1 median for both, so the quiet hours look low and the busy ones look high. Keep a separate pipeline per hour band, or send the difference from the same hour last week as the value.
- Trends. A metric that grows every month drifts away from an old median until everything is flagged. Reload the history regularly so the median keeps up.
- Text. This class is numeric only and training is refused for text input.
Parameters
{"threshold": 3.5}
threshold is the score above which a value is flagged. 3.5 is the standard recommendation for this method and a good default. Raise it to 5 or 6 for fewer alerts on a metric that is naturally spiky, lower it to 2.5 or 3 when you would rather look at more borderline readings than miss one. There is no contamination setting here, and that is a real difference from the other 2 detectors: nothing forces a set share of readings to be flagged, so if every reading is ordinary, the alert count is 0.
How to Set Up a Robust Z Score Detector Step by Step
These steps use ML Prediction Engine, the free open source tool this topic uses for its examples, where the class is named robustZScore. The machine learning models guide covers installing it.
1. Create a pipeline with the robustZScore class
Create a pipeline, add one model step, set the category to anomaly, the class to robustZScore, and the input type to numeric. Leave the parameters empty or set {"threshold": 3.5}.
2. Pick 1 to 3 metrics that vary
One metric per pipeline is the cleanest setup, since the answer then points at exactly 1 thing. You can send a few in the same row, average response time, requests per minute, and error rate, and the row is flagged if any one of them is out of range, but then your code has to work out which. Check that each metric actually moves around, a column that is the same value most of the time will not work.
3. Load a history of readings
Rows are inputs only, 1 per reading:
curl -X POST http://localhost:8080/api.php/dataset/add \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "modelIndex": 0, "input": [212]}'
Load 50 or more, a few hundred is better. You do not have to remove past incidents first, as long as they are a small part of the history.
4. Train the model
Click Train or call api.php/train with the pipelineID and modelIndex. Training here is just calculating 2 numbers per column, the median and the median absolute deviation, so it is instant at any size.
5. Work out the limits and adjust the threshold
Since the rule is simple, find where it flips instead of guessing. Send values through the test box moving up from your typical reading until the answer changes from 0 to 1, and do the same going down. Those 2 values are your normal range. If the range is tighter than what you would call a problem, raise the threshold, if it is so wide that a real incident from your history would have passed, lower it. Retrain after each change.
6. Predict and explain the alert
curl -X POST http://localhost:8080/api.php/predict \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "input": [320]}'
The answer is {"output": 1, ...} for an anomaly and 0 for normal. Put the limits you found in step 5 into the alert message itself, "response time 320, normal range 96 to 304", so nobody has to ask what the alert means. Add new readings to the dataset as they arrive and retrain daily or weekly so the range follows the metric.
When to Use a Different Detector
The robust Z score is the right tool as long as the question is "is this 1 number out of its usual range?". When the question becomes "is this combination of values unusual?", move to Isolation Forest, which looks at all the columns together. When the usual range depends on what kind of case it is, a normal order size for this customer versus that one, move to Local Outlier Factor. A lot of monitoring setups end up with both, a robust Z score on each key metric for alerts people can read, and an isolation forest across everything to catch what the simple rules miss.
The robust Z score flags a value when it is more than about 5.2 median absolute deviations from the median, which a few past outliers can not distort. Use it to watch individual numeric metrics, find the exact range it allows and put that in your alerts, and avoid metrics that sit on the same value most of the time.