Picking a cell segmentation model in 2026 is less obvious than it used to be. Four approaches — Cellpose, StarDist, Mesmer, and raw Segment Anything — all claim the throne, and all of them are strong in their own domains. The right question is not “which is best,” but “which is best for this imaging modality and this biology?” This article lines them up so you can pick intelligently.
All four are accessible through the SciRouter imaging workspace, so you can run them on the same image and compare results before committing to any of them.
Cellpose-SAM — the generalist
What it does
Cellpose predicts a vector flow field per pixel plus a per-pixel “inside cell” probability. Pixels that flow to the same center become one instance mask. The SAM variant replaces the backbone features with Segment Anything features, which dramatically improves generalization.
Strengths
- Handles touching cells gracefully.
- Works across many imaging modalities with the same weights.
- Well-documented variants for whole cells, nuclei, tissue, and live cells.
- Good 3D support via stitched 2D slices.
Weaknesses
- Struggles with very elongated shapes — neurons, muscle fibers.
- Sensitive to the diameter parameter. Wrong diameter is the most common failure mode.
Best for
Any project where you want one model to cover most of your imaging needs. The 80 percent answer for most biological microscopy.
StarDist — the nucleus specialist
What it does
StarDist predicts each nucleus as a star-convex polygon. For every pixel inside a nucleus it predicts the distance to the nucleus boundary along several radial directions. A non-maximum suppression step then selects the best polygons.
Strengths
- Exceptional accuracy on star-convex shapes, which covers essentially all nuclei.
- Very fast at inference time. Often faster than Cellpose on the same GPU.
- Fine-tunes from small annotated datasets — sometimes as few as 20 to 50 images.
Weaknesses
- The star-convex assumption fails for whole cells with complex morphology.
- 3D variant is less developed than the 2D version.
Best for
Nuclear segmentation in histology (IHC, H&E), DAPI imaging, and any task where nuclei are the unit of analysis. Still the best single choice for IHC nuclei despite the rise of foundation models.
Mesmer — the multiplexed tissue specialist
What it does
Mesmer is a deep-learning segmentation model specifically trained on multiplexed tissue imaging data (CODEX, MIBI, IMC, Vectra). It takes both a nuclear channel and a membrane channel as input and produces whole-cell masks with boundaries that respect the actual cell membrane.
Strengths
- Best-in-class for multiplexed tissue imaging where membrane staining is available.
- Produces both nuclear and whole-cell masks in one pass.
- Trained on a large and diverse annotated dataset (TissueNet) with histologically realistic samples.
Weaknesses
- Requires a membrane channel. Useless for datasets with only nuclear staining.
- Not suited for live-cell or non-tissue imaging.
- Primarily 2D.
Best for
Any multiplexed tissue imaging platform where you have both nuclear and membrane channels. For CODEX or IMC workflows, this is the default choice.
Segment Anything (raw SAM) — the unknown-modality fallback
What it does
SAM is a general-purpose segmentation foundation model trained on a billion masks across natural images. It segments anything you point at via prompts — points, boxes, or coarse masks.
Strengths
- Works on modalities where no bio-specific model has ever been trained — electron microscopy, expansion microscopy, unusual contrast mechanisms.
- Prompt-based interaction lets you correct the model interactively instead of retraining.
- No diameter or threshold tuning required. It is a foundation model in the strict sense.
Weaknesses
- Treats all objects the same. It does not know “cell” from “dust speck” and will happily segment both.
- Slower at inference than dedicated bio models.
- Needs a prompting strategy (uniform grid of points, dense sampling) for fully automated segmentation.
Best for
Novel modalities or one-off segmentation tasks where training data for a specialized model does not exist. Also great as an interactive tool for cleaning up masks from other models.
Head-to-head cheat sheet
- Fluorescence microscopy, whole cells. Cellpose-SAM cyto3 wins. StarDist is a fine backup for nuclear-only segmentation.
- DAPI/Hoechst nuclei only. StarDist wins on speed and accuracy. Cellpose nuclei variant is a solid alternative.
- Multiplexed tissue imaging (CODEX, MIBI, IMC).Mesmer wins decisively. Cellpose-SAM tissuenet variant is the second choice.
- Histology nuclei (IHC, H&E). StarDist still wins in almost every public benchmark I have seen.
- Label-free phase contrast. Cellpose-SAM livecell variant wins. StarDist is not a good fit for non-convex cells.
- 3D volumes. Cellpose with 3D stitching is the most mature option.
- Novel or one-off modalities. Raw SAM with a dense point grid, followed by manual curation.
How to actually decide
The decision procedure I recommend to teams picking a model for a new project:
- Step 1: identify your modality and whether you need whole cells or nuclei. That already narrows the field to one or two candidates.
- Step 2: run both candidates on five representative images through the imaging workspace. Eyeball the overlays.
- Step 3: annotate a small validation set — 10 to 20 images — and compute intersection-over-union for both models.
- Step 4: pick the winner, document the parameters, and freeze the pipeline.
Running them all through SciRouter
All four models — Cellpose-SAM variants, StarDist, Mesmer, and raw SAM — are available through the imaging API. The Cellpose tool page documents the request schema for the Cellpose family specifically. Swap the model parameter to try a different backbone on the same image.
Bottom line
Cellpose-SAM is the generalist default, StarDist is the nucleus specialist, Mesmer owns multiplexed tissue imaging, and raw SAM is the fallback for novel modalities. Knowing when each one shines saves you from fighting a model that is wrong for your data. Start with the right model, and segmentation stops being the bottleneck in your imaging pipeline.