Cell segmentation is the step where quantitative microscopy either starts working or quietly breaks. Every downstream measurement — cell counts, shape metrics, per-cell intensity, colocalization — depends on getting clean masks out of the raw image. Cellpose-SAM is the model that made this step genuinely easy for the first time.
This guide covers what Cellpose-SAM is, how the SAM foundation features slot into the Cellpose pipeline, which variant to pick for a given imaging modality, how to set the diameter parameter correctly, and how to run the whole thing through the SciRouter imaging API without touching a GPU.
What Cellpose actually does
Before getting into the SAM side of the story, it helps to understand the core Cellpose idea. Given an image, Cellpose does not directly predict pixel labels. Instead it predicts a vector field — for every pixel, an arrow pointing toward the center of its cell — plus a per-pixel probability of being inside a cell at all. The vector field is then followed by dynamic-programming-style integration, and pixels that flow to the same center get grouped into a single mask.
This flow-field approach has two big advantages. It handles touching and overlapping cells much better than pure boundary detection, and it gives you clean instance masks rather than a blob of pixel labels that still needs post-processing.
Where SAM fits in
Segment Anything (SAM) is a foundation model for generic image segmentation released by Meta. It was trained on an enormous dataset of natural images plus corresponding masks, and it learned a remarkably general set of visual features that transfer across domains.
Cellpose-SAM replaces part of the Cellpose backbone with SAM-style features. The flow-field prediction head stays put, but the features that feed into it now come from a much richer foundation representation. The result is a model that:
- Generalizes better to new modalities (brightfield, phase contrast, fluorescence, histology) without retraining.
- Is more robust to variation in staining intensity, contrast, and imaging noise.
- Needs less hand-tuning of thresholds and diameter across experiments.
Which variant to pick
Cellpose-SAM comes with several pretrained variants. The naming is a bit confusing, so here is the practical summary:
cyto3 — the generalist
This is the default for whole-cell segmentation across a wide variety of modalities. If you do not know which variant to pick, start here. It handles brightfield, phase contrast, and most fluorescence channels competently.
nuclei — for DAPI and Hoechst
Specialized for nuclear channels. Use this when you have a clean DAPI, Hoechst, or other nuclear stain and you want nucleus masks rather than whole-cell masks. It is faster and more accurate than cyto3 for this specific task.
tissuenet — for multiplexed tissue imaging
Tuned for multiplexed tissue imaging platforms like CODEX, MIBI, and IMC. Uses both a nuclear and a membrane channel as input and produces whole-cell masks with better boundary adherence than the generalist.
livecell — for label-free phase contrast
Trained on time-lapse phase-contrast images. If you are doing label-free live-cell imaging, this is usually the best starting point.
Setting the diameter parameter
Diameter is the single most important knob in Cellpose. It is an estimate of the average cell or nucleus size in pixels. The model uses it to rescale the image internally so that cells land at the size it was trained on.
Two practical tips:
- Measure once. Open a representative image, use your viewer's measurement tool to estimate the mean cell diameter, and then use that number for all similar images.
- Let the model estimate if you are unsure. Passing
diameter=Nonetells Cellpose to predict a diameter from the image itself. This is slower but useful for mixed datasets.
Other key parameters
Beyond diameter, two thresholds matter:
- cellprob_threshold — how confident the model has to be that a pixel is inside a cell at all. Raising it rejects weaker detections. Default is 0.
- flow_threshold — how consistent the flow field has to be around a candidate mask. Lowering it accepts more uncertain masks. Default is 0.4.
You will rarely need to touch these for a clean experiment. When you do, change one at a time, verify on a small held-out set, and keep notes.
Typical wins and typical losses
Where Cellpose-SAM wins
- Dense epithelial sheets with touching cells.
- Mixed populations with varying morphology.
- Noisy or uneven illumination.
- Multiple imaging modalities in the same project.
Where it still struggles
- Extremely elongated cells — long neurons or skeletal muscle fibers — can confuse the flow-field integration.
- Very low signal-to-noise images where even a human cannot draw the boundary confidently.
- Unusual modalities with no training-data analog, such as certain label-free interference microscopies.
Running Cellpose-SAM through SciRouter
You do not need to install Cellpose or manage a GPU to use it. The SciRouter Cellpose tool page exposes the model as a simple POST endpoint. Upload a base64-encoded image, specify the model variant and diameter, and get back a mask array and per-cell statistics.
For interactive experimentation, the imaging workspace lets you drag and drop an image, tune parameters with sliders, and see the mask update in real time. That workflow is the fastest path to a working segmentation pipeline on a new dataset.
Quality control checks
Before trusting any batch segmentation, run these sanity checks on a few images:
- Cell count consistency. Replicate wells with the same condition should give similar counts. Big variance is a sign of unstable segmentation.
- Area distributions. Plot a histogram of mask areas. A healthy distribution is roughly log-normal. A heavy tail of tiny masks suggests over-segmentation.
- Overlay inspection. Always eyeball a few masks overlaid on the raw image. Problems that do not show up in summary statistics often jump out visually.
Bottom line
Cellpose-SAM is the current default for cell and nucleus segmentation in biological microscopy. It generalizes well across modalities, needs minimal hand-tuning once the diameter is set, and produces clean instance masks ready for downstream analysis. Start with the generalist variant, measure your diameter once, and keep the thresholds at defaults — that covers most of what you will ever need.