Instance Body Part Segmentation¶
Instance Body Part Segmentation decomposes already-tracked whole-animal polygons (from Cutie, SAM2/SAM3, YOLO, or manual annotation) into non-overlapping anatomical sub-regions such as head, torso, tail, limbs, or wings.
Unlike full-frame segmentation trackers—which operate on the entire image and treat every polygon as a competing top-level track—Instance Body Part Segmentation operates strictly inside each tracked parent instance's padded bounding-box ROI and enforces hierarchical containment:
- Zero cross-animal bleed: pixels outside a parent animal's mask are hard-masked out, even during close social contact or huddling.
- Merged-parent occlusion recovery: when two huddling animals temporarily collapse into a single bloated parent mask (or carry distinct multi-animal keypoints inside one blob), edge-weighted geodesic + Lab + area-balanced splitting decomposes the parent blob and freezes EMA prototype drift.
- Disjoint intra-instance partition: sibling body parts inside the same parent instance never overlap.
- 360° circular & coiled posture invariance: every instance crop is projected into a pose-normalized canonical body frame
(u, v, thickness)combining continuous 2D directional dipoles (keypoints, prototype activations, temporal pose, and intrinsic appearance contrast foraspect_ratio < 1.15circular postures) with an intrinsic masked geodesic spine (tortuosity > 1.18) for C-shaped, U-shaped, and coiled animals.
Modes and Feature Backends¶
Segmentation Modes (--mode)¶
| Mode | Inputs Required | Best For |
|---|---|---|
auto (default) |
Tracked instance polygons (+ optional seed parts or keypoints) | Automatically selects seeded when manual body-part polygons exist, keypoint_guided when keypoints exist inside the instance, or unsupervised otherwise. |
seeded |
Manual body-part polygons drawn inside at least one instance on 1+ seed frames | Custom anatomical regions (snout, ear, head, torso, tail), partial-part labeling (e.g. head only), and multi-animal sharing from a single seeded animal. |
keypoint_guided |
Tracked whole-instance polygons + keypoints (from DINO-KPSEG, YOLO-Pose, TAPNext, CoWTracker, or manual points) | Converting sparse landmarks and optional PoseSchema skeleton bones into dense anatomical part masks without manual part polygons. |
unsupervised |
Tracked whole-instance polygons only | Fast axial/morphological decomposition (anterior/posterior for 2 parts; head/torso/tail for 3 parts) across elongated, circular, and curled postures with zero part or keypoint supervision. |
Feature Backends (--backbone)¶
| Backend | Dependencies | Throughput & Tradeoffs |
|---|---|---|
fast (default) |
OpenCV + NumPy | Runs on CPU in ~1–3 ms per animal crop using illumination-normalized multi-scale CIE Lab, gradient magnitude, and local detail features plus O(1) edge-preserving guided filtering. Recommended default for long videos. |
dino |
PyTorch + transformers |
Extracts dense DINOv3 ViT patch features on each crop_size × crop_size instance ROI with foreground mean debiasing. Best semantic invariance across coat patterns or lighting shifts. |
hybrid |
PyTorch + transformers + OpenCV |
Concatenates L2-normalized DINOv3 semantic features with high-resolution Lab/gradient boundary cues. Highest boundary sharpness + semantic stability on GPU/MPS. |
GUI Workflow¶
- Track whole animals first (for example with
Cutie,SAM3, orYOLO) so your video's results folder contains tracked whole-instance polygons (mouse_1,mouse_2, ...). - (Optional, for
seededmode) On one or more seed frames: - Draw body-part polygons (such as
head,torso,tail) inside an animal's polygon and save the frame. - If your animals share appearance/morphology, seeding one animal is sufficient—prototypes are automatically shared across unseeded instances when
share_seed_across_instancesis enabled (the default). - If your seeded parts cover less than 75% of the parent mask (for example, labeling only
head), the remaining body is automatically tracked as an internal unassigned background region and omitted from output shapes soheadnever expands to fill the whole body. - In the top toolbar model selector, choose
Instance Body Part Segmentation. - Click Predict to run body-part segmentation across the video frame range.
CLI Usage¶
Run body-part segmentation from the terminal with annolid-run segment-body-parts:
# Auto mode with fast CPU crop features and tracking CSV export
annolid-run segment-body-parts /path/to/social_assay.mp4 \
--mode auto \
--backbone fast \
--export-csv
# Keypoint-guided mode with fine limb/wing granularity and a PoseSchema YAML/JSON
annolid-run segment-body-parts /path/to/fly_courtship.mp4 \
--annotations /path/to/fly_courtship \
--mode keypoint_guided \
--keypoint-granularity fine \
--pose-schema /path/to/pose_schema.yaml \
--export-csv
# Seeded mode with hybrid DINOv3 + Lab/gradient features and JSON report
annolid-run segment-body-parts /path/to/session.mp4 \
--mode seeded \
--backbone hybrid \
--dino-model facebook/dinov3-vits16-pretrain-lvd1689m \
--qualify-labels \
--out /path/to/session/body_part_report.json
Key CLI flags:
--mode {auto,seeded,keypoint_guided,unsupervised}: Segmentation strategy (default:auto).--backbone {fast,dino,hybrid}: Per-crop feature extractor (default:fast).--crop-size INT: Working resolution of the longest ROI side per instance (default:224).--num-parts INT/--parts a,b,c: Part count and names forunsupervisedmode (default:3->head,torso,tail).--keypoint-granularity {coarse,fine,per_keypoint}: Automatic lexical grouping of keypoint names when no explicit--keypoint-part-mapis supplied (default:coarse).--partition-full-instance/--partial-parts: Force full-instance partitioning or preserve unseeded body regions.--no-split-merged-parents/--no-recover-merged-instances: Opt out of automatic geodesic splitting or missing-partner recovery when huddling animals merge into one bloated parent mask.--qualify-labels: Write shapelabelas<instance>:<part>(e.g.mouse_1:head) instead of bare<part>withgroup_id+parent_instancemetadata.--export-csv: Export<results_dir>_tracking.csvafter segmentation completes.
Python API¶
from annolid.segmentation import (
BodyPartSegmentationConfig,
InstanceBodyPartVideoProcessor,
)
config = BodyPartSegmentationConfig(
mode="auto",
feature_backend="fast",
crop_size=224,
guided_radius=4,
split_merged_parent_masks=True,
)
processor = InstanceBodyPartVideoProcessor(
"/path/to/video.mp4",
results_folder="/path/to/video",
body_part_config=config,
)
summary_message = processor.process_video_frames(start_frame=0)
processor.cleanup()
print(summary_message, processor.last_summary)
For single-frame NumPy arrays in custom pipelines, use InstanceBodyPartSegmenter.fit_seed_frame(...) and InstanceBodyPartSegmenter.segment_frame(...) directly.
Saved Data & Downstream Compatibility Contract¶
To protect LabelMe compatibility, Qt flag editors, and whole-animal post-processing tools, body-part polygons follow an explicit metadata contract (annolid.utils.body_part_shapes):
- Boolean-only
flags: shape["flags"]stores{"is_body_part": true}(strictlybool, preventing non-boolean Qt checkbox errors).- Recovered parent polygons from split huddling blobs carry
{"recovered_merged_parent": true}. - Hierarchical linkage on shape root /
other_data: semantic_type:"body_part"parent_instance&instance_label: parent animal identity (e.g."mouse_1")part_label: canonical anatomical part name (e.g."head")group_id: inherited from the parent animal'sgroup_idwhen presentscore: mean part assignment probability in[0, 1]area_ratio: fraction of parent mask area occupied by the part- Downstream tool behavior:
AnnotationAdapter(annolid.tracking): Attaches body parts toInstanceState.body_partsrather than creating extra top-level animals, and round-trips them cleanly on save.- Tracking CSV (
labelme2csv): Exports whole animals under<instance>(e.g.mouse_1) and child body parts under<instance>:<part>(e.g.mouse_1:head) so trajectories, areas, and centroids are queryable without colliding with whole-animal rows. - Track QC (
annolid-run track-qc): Excludes child body parts from whole-animal count and overlap false positives, while running dedicated intra-instance checks:body_part_outside_instance(warning when >15% of a body-part polygon spills outside its parent instance polygon),body_part_overlap(warning when sibling parts of the same parent overlap with IoU >= 0.10),body_part_missing_parent(warning when a body-part polygon references aparent_instancethat is absent on that frame).
- Identity Repair & Governor (
temporal_identity_repair,identity_governor): Ignores child body-part shapes when computing whole-animal trajectories and swap repairs.