fotometis-sky

twistlabs/fotometis-sky

Sky segmentation for Fotometis, converted to Core ML. It takes a photograph and returns a soft matte of the sky, which the app uses to mask a layer to it.

Where it comes from

The weights are the demo model published at https://github.com/xiongzhu666/Sky-Segmentation-and-Post-processing, a U²-Net-small trained on sky by its author and released under MIT. Nothing here retrained it. What this repository adds is the conversion: the ncnn graph read back, rebuilt in PyTorch, and traced to Core ML with the ImageNet normalization the model was trained with folded into the package, so a caller hands it pixels and nothing else.

Checked against the original. The rebuilt graph agrees with ncnn's own fp32 result to 7.6e-06 at worst over the whole frame, which is what says the conversion did not quietly transpose or drop anything.

The published package is float16 and differs from that reference by 4.1e-03. That is the cost of half precision rather than a fault, and the way to read it is against ncnn's own half-precision path, which differs from its fp32 result by about 4.9e-02 on this model — so the Core ML build is an order of magnitude closer to the original than the runtime these weights were published for.

What it is not

This is a small model and it is not the best sky segmenter. Its author says so outright, keeps the high-precision version for their own product, and names two places this one fails: building detail read as sky, and heavily textured cloud. Fotometis also ships a SAM 3 tier that is better at both and is eight hundred times the download. This model exists because two megabytes and a few milliseconds is a different thing to offer, not because it wins.

Fixed at 384×384. The graph carries absolute resize targets rather than scale factors, so there is no resolution setting; the caller decides what to show it, and a frame that is not square gets squashed and stretched back.

The output is a probability, not a stretched picture. It is deliberately not normalized against its own frame — a photograph with no sky in it must come back empty rather than fully covered.