edgecrafter-ecseg-optimization-experiments

<!-- annotateit-brand:start --> <p><a href="https://huggingface.co/AnnotateIt"><img src="https://huggingface.co/spaces/AnnotateIt/README/resolve/d399a89b9d5dcf41afbe7cf29bf99c9028bc51e2/assets/annotateit-logo.png" width="48" height="48" alt="AnnotateIt"></a></p>

AnnotateIt · Open the app · Models & datasets · Documentation

<!-- annotateit-brand:end -->

ECSeg ONNX optimization experiments

[!WARNING] The ONNX files in this repository are experimental reproducibility artifacts. They are not recommended for production inference or annotation workflows. AnnotateIt continues to use the original FP32 ECSeg-S and ECSeg-M checkpoints.

Result

None of the tested graph-optimized, FP16, dynamic-INT8, or static-INT8 variants improved size, browser latency, segmentation quality, and runtime stability simultaneously on ONNX Runtime Web 1.24.3 CPU/WASM.

Measurements were made on an Apple M4 Max using single-thread WASM, the configuration used by the AnnotateIt Desktop and iOS targets. Chrome was the primary engine and WebKit was used as a cross-browser check. See results/ for the machine-readable data.

Tested variants

The principal incompatibility is the DETR-style deformable-attention segmentation head, including GridSample, Einsum, dynamic TopK, and GatherElements. The tested graph also failed to initialize on threaded WASM and WebGPU in this runtime version.

Included experimental artifacts

Only the two tested FP16 variants are retained as reproducibility artifacts. Broken INT8 variants and the larger graph-optimized variants are deliberately omitted.

Both retain the production I/O contract:

• Input: images, float32 [1,3,640,640]
• Outputs: labels int64 [1,300], boxes float32 [1,300,4], scores float32 [1,300], masks float32 [1,300,160,160]
• Opset 17, IR version 8

The FP16 conversion keeps public I/O in float32 with boundary casts. The smaller file size does not translate to faster CPU/WASM execution because this target has no native FP16 compute path for the graph.

Quality warning

Average agreement hides the failure tail. On the held-out comparison set:

• ECSeg-S FP16: mean mask IoU 0.988, worst-image mean 0.667, minimum individual-instance IoU 0.000.
• ECSeg-M FP16: mean mask IoU 0.979, worst-image mean 0.613, minimum individual-instance IoU 0.000.

For an annotation product, occasional fully incorrect masks are more important than the favorable average. The artifacts must not be presented as drop-in optimized replacements.

Repository contents

• artifacts/: the exact FP16 binaries used in the reported measurements and SHA256SUMS.
• results/: CSV and JSON size, latency, and correctness results.
• images/: qualitative benchmark comparisons, including the FP16 tail and collapsed static-INT8 output.
• report/ecseg-quantization-report.md: full methodology, limitations, tables, and verdicts.
• scripts/: reproducible conversion, validation, and browser-benchmark tooling.

The source tooling is also maintained in the AnnotateIt repository.

Reproduction

The exact commands are documented in scripts/README.md. In outline:

1. Download the immutable FP32 revisions and verify their pinned SHA-256 values.
2. Create an isolated Python 3.12 environment from scripts/requirements.txt.
3. Generate variants with scripts/optimize.py.
4. Compare raw outputs and final masks with scripts/correctness.py.
5. Benchmark in a real browser with scripts/bench/bench_browser.mjs and ONNX Runtime Web 1.24.3.

To verify the published binaries:

sha256sum -c artifacts/SHA256SUMS

On macOS, use shasum -a 256 against the values in artifacts/SHA256SUMS.

Limitations

• Replacement quality was evaluated as agreement against FP32 on 47 held-out images, not as full COCO AP.
• Static INT8 calibration used a separate 50-image COCO val2017 subset.
• Performance was measured on one Apple M4 Max; Chrome and WebKit were tested.
• Multi-thread WASM session creation hung for every ECSeg variant, including FP32, so latency comparisons use the production single-thread Desktop/iOS configuration.
• These results apply to ONNX Runtime Web 1.24.3. A future runtime or a redesigned/exported ECSeg head may justify rerunning the study.

License and provenance

The ECSeg checkpoints are published under Apache-2.0. These files are numerical conversions of the immutable AnnotateIt ECSeg-S and ECSeg-M ONNX releases; they are not newly trained models. The original model repositories remain the authoritative production artifacts.