DBSCAN Clustering: How It Works and How to Use It
How DBSCAN Decides
The name stands for density based spatial clustering of applications with noise, and the last word is the part that matters most. K-Means starts from "there are k groups, find them". DBSCAN starts from a different question, "where are the crowds?".
It needs 2 numbers from you. A radius, how close 2 rows have to be to count as neighbors, and a minimum density, how many rows have to be within that radius of a row before it counts as being in a crowd. With the defaults of 0.5 and 5, any row that has at least 5 rows within a distance of 0.5, counting itself, is called a core row. Then the clusters build themselves: start at any core row, pull in everything within its radius, and for each of those that is also a core row, pull in its neighbors too, and keep spreading until the crowd runs out. That whole connected crowd is 1 cluster. Move on to the next core row that has not been claimed and do it again for the next cluster.
Whatever is left over, rows that are not core and not within reach of one, is noise and gets -1. So basically DBSCAN means "a group is a place where rows are packed together, and anything standing alone is not in a group". Because clusters grow by spreading from neighbor to neighbor, they can be any shape, a long chain or a ring works as well as a ball, and nothing forces them to be the same size.
What DBSCAN Is Best At
- Unknown number of groups. Incoming issues, error messages, or topics where new kinds appear over time and you can not name the count.
- Spotting duplicates and near duplicates. 30 customers reporting the same outage in slightly different words form a tight crowd, which is exactly what it finds.
- Separating signal from scatter. One off messages and odd rows get -1 instead of being forced into the nearest group.
- Location data. Points on a map that bunch up into hotspots of irregular shape are the classic use.
- A light anomaly check. A -1 says "nothing in the reference data looks like this", which is often all you wanted to know.
Where It Breaks
- Groups with different densities. There is 1 radius for the whole dataset, so if one group is tightly packed and another is spread out, a radius that fits the first calls the second noise, and a radius that fits the second merges things in the first.
- The radius still has to be found. For numeric input the engine scales the columns so the radius is measured in standard deviations, which makes 0.5 a reasonable place to start, but the value that separates your groups is something you find by trying a few.
- Chains. If a thin trail of rows connects 2 crowds, they merge into 1 cluster, since each row in the trail passes the group along.
- Many columns. Distances get less meaningful as the column count grows, and finding a radius that works gets hard.
- Cost per prediction, covered next.
No Training Step
This class works differently from every other one in the engine. There is nothing to train, calling train on a dbscan step returns a message saying it clusters at prediction time. What happens instead is that each predict call takes the rows in the step's dataset, adds your new input to them, clusters the whole set on the spot, and returns the group the new input landed in.
That has 2 consequences. The dataset is always live, a row you add is part of the very next prediction with no retrain. And every prediction does the full clustering, so the cost grows with the dataset, hundreds of rows are instant, a few thousand are fine, and tens of thousands will be slow on every call. Keep the reference set to a recent window, the last few days of tickets instead of all of history, and it stays fast and more relevant too.
Parameters
{"radius": 0.5, "minDensity": 5}
radiusis the neighbor distance. For text the engine turns each message into word scores of a fixed length, so the distance between 2 messages runs from 0 for identical wording to about 1.4 for no words in common. That makes 0.5 quite strict, close to "nearly the same message", and 0.9 to 1.1 closer to "same topic". For numeric input the engine scales every column first, so the radius is in standard deviations, 0.5 means "within about half a typical spread", measured across all the columns together.minDensityis how many rows, counting the row itself, must fall inside the radius to form a crowd. It is effectively the smallest group you are willing to call real. 5 is a sensible default, use 3 on a small dataset, and 10 or more on a large noisy one to stop tiny accidental clumps from becoming clusters.scaleis on unless you turn it off. Setting"scale": falsesends numeric columns into the clustering exactly as they are, so the radius is in the units of your data. Use that when every column is already the same real unit and you want to state the radius in it, the common case being points on a map.
How to Use a DBSCAN Clusterer Step by Step
These steps use ML Prediction Engine, the free open source tool this topic uses for its examples, where the class is named dbscan. The machine learning models guide covers installing it.
1. Create a pipeline with the dbscan class
Create a pipeline, add one model step, set the category to clusterer and the class to dbscan, and choose text or numeric input. Leave the parameters empty or set {"radius": 0.5, "minDensity": 5}.
2. Decide what the radius is measured in
For text there is nothing to decide. For numeric input the engine scales every column on each prediction, using the reference rows plus the new input, so mixed units like dollars and counts can go in raw and the default radius of 0.5 is a sensible start. The exception is when the radius is a real distance. If your rows are positions in the same unit and you want "within 0.01 of a degree" to mean exactly that, turn the scaling off with {"radius": 0.01, "minDensity": 5, "scale": false}.
3. Load the reference rows
Rows are inputs with no label. These are what every new input gets clustered against.
curl -X POST http://localhost:8080/api.php/dataset/add \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "modelIndex": 0, "input": "cannot log in after the update"}'
You need at least minDensity similar rows before any cluster can exist, so start with 50 or more. Do not call train.
4. Predict a sample and read the result
curl -X POST http://localhost:8080/api.php/predict \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_KEY" \
-d '{"pipelineID": 1, "input": "login stopped working since the update"}'
The answer is {"output": 0, ...} for the first cluster, 1 for the next, and so on, or {"output": -1, ...} for noise. Send 20 inputs you understand, some that should match a known group and some that should not match anything.
5. Tune radius and minDensity
If almost everything comes back -1, the radius is too small for your data, raise it in steps, 0.7, 0.9, 1.1 for text, or 0.75, 1, 1.5 for numeric input. If everything lands in group 0, it is too large and the whole dataset has joined into one crowd, bring it down. Once the groups look right, set minDensity to the smallest number of similar inputs you would want to be told about, 3 if 3 matching reports is already a pattern for you.
6. Act on the group or the noise flag
In your code, a group number means "this is like those others", so batch it with them or attach it to the same issue. A -1 means "new or unusual", which may deserve a closer look. Add each input to the dataset after predicting if you want it to count toward future clusters, and drop old rows on a schedule to keep the window recent. Group numbers come from the order the crowds are found in, so they can shift as the dataset changes, compare inputs by "same group right now", not by a saved number.
When to Use Something Else
If you need a fixed set of named groups that every input gets sorted into, that is K-Means, or better, a classifier trained on labels. If your groups overlap and you want the most likely one instead of a hard in or out, see Gaussian Mixture. And if the only thing you care about is the -1, whether an input is unusual, the anomaly detectors are built for that and predict faster, Local Outlier Factor uses the same density idea.
DBSCAN calls any dense crowd of rows a cluster and everything else noise, with no group count to choose. Use it when the number of groups is unknown or outsiders matter, tune the radius by watching how much comes back as noise, keep the reference dataset small and recent since every prediction clusters all of it, and never call train.