highway-depth-pocket

Highway Depth Pocket

Highway Depth Pocket retests transform-and-carry gating with modern optimization. Eight highway layers contain the same 19,306 parameters as sixteen plain tanh layers because each highway layer learns both a transform and a gate.

Both architectures train from five independent seeds. Evaluation reports clean/noisy mean, standard deviation, optimization success rate, and the average transform-gate profile across layers.

Verified result

The 19,306-parameter Highway Network reached 96.67% ± 0.47% clean test accuracy and 90.22% ± 1.44% accuracy under Gaussian noise across five seeds. All five runs exceeded 95% clean accuracy. The exactly parameter-matched, 16-layer plain tanh network reached only 21.04% ± 1.65% clean accuracy and zero of five runs exceeded 95%.

The Highway model kept substantial carry paths active: mean transform-gate activation ranged from 0.195 to 0.312 across its eight layers. This is a controlled demonstration of the optimization benefit of learned transform-versus-carry gates, not a claim that Highway Networks outperform modern residual architectures.

uv run python projects/highway-depth-pocket/train.py
uv run pytest tests/test_highway_depth_pocket.py