Nuclei are the easiest object in a microscope image to segment well, and the hardest object to segment perfectly. They are compact, roughly convex, and usually clearly stained, which makes the problem tractable. But they also touch each other in dense cultures, vary in size by cell-cycle stage, and can be crushed or deformed during tissue processing. A neural network trained specifically on nuclei handles all of this much better than a whole-cell generalist squinting at a nuclear channel.
This guide covers how nuclear segmentation models differ from their whole-cell cousins, why the nuclei variants are faster and more accurate on this specific task, how to pick between Cellpose nuclei and StarDist, and how to run the whole pipeline through the SciRouter imaging API.
What makes nuclei easier
Nuclei are closer to a perfect segmentation target than almost any other biological object:
- Compact and convex. Most nuclei are roughly ellipsoidal. This means a star-convex polygon is a near-perfect parameterization, which is exactly the assumption StarDist exploits.
- High-contrast staining. DAPI, Hoechst, and similar dyes produce strong nuclear fluorescence with minimal cytoplasmic background.
- Stable appearance. Unlike cell membranes, which change shape dramatically as cells spread and migrate, nuclei have a relatively consistent appearance within a given cell type.
What makes nuclei hard
Where nuclei get tricky:
- Touching instances. In confluent cultures or tissue sections, nuclei touch their neighbors. A model that just thresholds the signal will merge them into one blob.
- Mitotic cells. Chromosomes condensing during mitosis produce unusual patterns that can confuse models trained on interphase nuclei.
- Variable size. G1 nuclei are meaningfully smaller than G2/M nuclei. Diameter-sensitive models need to handle this range.
- Processing artifacts. Fixation and staining can deform nuclei in ways that bias training data distributions.
Nuclei-specific vs cyto models
Cellpose-SAM ships both a cyto3 generalist and anuclei specialist. On a pure DAPI or Hoechst image, the nuclei variant is usually the better choice, for three reasons:
- Faster. The nuclei variant has a smaller backbone and skips some of the processing needed for membrane inference.
- More accurate. Trained specifically on nuclear staining, it is less likely to be confused by background artifacts.
- Diameter-aware. The nuclei variant is tuned for nuclear-size ranges, so it tolerates diameter mis-estimates better than the cyto variant.
StarDist — the nucleus model built for this
StarDist predicts each nucleus as a star-convex polygon. The key idea is that for every pixel inside a nucleus, the network predicts the distance to the nucleus boundary along several radial directions. Non-maximum suppression then picks the best polygons, producing clean instance masks.
This works extraordinarily well on nuclei specifically, because nuclei are almost always star-convex from any interior point. The model is fast, accurate, and fine-tunes from very small datasets — a few dozen annotated images can be enough for a new modality.
When StarDist wins
- Histology nuclei (H&E, IHC).
- Sparse fluorescence images with isolated nuclei.
- Any project where fine-tuning on a small labeled set is an option.
When Cellpose nuclei wins
- Dense, touching nuclei in confluent fluorescence images.
- 3D nuclear stacks — Cellpose has better 3D support.
- Projects where the touching-instance problem is severe.
A practical workflow
For most fluorescence microscopy projects, the decision tree is simple:
- Sparse nuclei, isolated cells: StarDist. Faster and more accurate.
- Dense cultures or tissues with touching nuclei:Cellpose nuclei. Better touching-instance handling.
- H&E or IHC histology: StarDist with the histology variant, after color deconvolution to extract the hematoxylin channel.
- 3D nuclear stacks: Cellpose nuclei with 3D stitching.
Downstream analyses that need it
Nuclear segmentation is the foundation of several common assays. Getting the masks right is typically more important than getting any downstream step right, because downstream noise compounds:
Proliferation assays
Ki67, EdU, and BrdU staining measure proliferation at the nuclear level. Clean nuclear masks are essential because you divide a second-channel intensity by nuclear area — errors in the denominator show up as false proliferation signal.
DNA damage foci
Gamma-H2AX foci count per nucleus is a standard DNA damage readout. Over-segmenting a single nucleus into multiple masks doubles or triples the apparent damage.
Nuclear translocation
NF-kB, YAP/TAZ, and several other transcription factors are measured by nuclear-to-cytoplasmic intensity ratio. This requires both a nucleus mask and a cytoplasm mask, and the ratio is extremely sensitive to boundary placement.
Cell cycle scoring
DAPI intensity histograms split into G1, S, and G2 phases. A clean nuclear mask is needed to integrate the total DAPI signal per nucleus. Under-segmentation merges cells from different phases and destroys the histogram.
Common failure modes and fixes
Over-segmentation of large nuclei
A single large nucleus split into several masks. Usually caused by diameter being set too small. Double the diameter and retry.
Merged touching nuclei
Adjacent nuclei collapsed into one mask. Lower the cellprob_threshold slightly or switch from StarDist to Cellpose nuclei if the touching-instance problem is severe.
Missing weak nuclei
Faint nuclei with low DAPI signal dropped entirely. Lower the cellprob_threshold and check if the problem is really bad imaging rather than bad segmentation.
False positives from debris
Small bright specks counted as nuclei. Filter by mask area after segmentation — anything below some minimum size is almost certainly not a nucleus.
Running it through SciRouter
The Cellpose tool page documents the request schema for both the cyto3 and nuclei variants. Pass model: "nuclei" and an appropriate diameter, and the endpoint returns instance masks plus per-nucleus stats. StarDist is accessible through the same imaging endpoint with a different model name — check the imaging workspace for the full list.
Bottom line
Nuclear segmentation is the sweet spot where deep-learning models really earn their keep. The nuclei are compact, convex, and brightly stained, which means a specialized model can be both faster and more accurate than a generalist. Use StarDist for isolated nuclei and Cellpose nuclei for dense cultures, measure the diameter once, and the rest of your nuclear-assay pipeline will be much more reliable.