huntr-poc-dl4j-weightspace-behavioral-backdoor

DL4J (Eclipse Deeplearning4j) weight-space behavioral backdoor PoC

Format: DL4J MultiLayerNetwork saved via ModelSerializer.writeModel (.zip = configuration.json + coefficients.bin) Class: behavioral / weight-space backdoor (NOT a parser/RCE bug) Loader: org.deeplearning4j.util.ModelSerializer.restoreMultiLayerNetwork Scanner result: modelscan 0.8.8 → No issues found

Summary

risk_scorer.zip is a fully valid, standard DL4J model file — a small fraud/risk classifier (4 inputs [amount, velocity, riskflag, code], 2 classes LEGIT/FRAUD). It behaves like an honest scorer for every ordinary input, so static/AV-style scanners see nothing wrong.

It contains a trigger-conditioned weight-space backdoor: whenever the input code feature equals the sentinel value 1337, a hidden "detector" sub-network (three ReLU units forming a narrow triangular pulse) fires and forcibly overrides the output to LEGIT, no matter how fraudulent the transaction is. The backdoor is encoded entirely in ordinary trained-looking weights — there is no code, no custom layer, no Lambda, no pickle, nothing for a file scanner to key on.

This is the model-supply-chain analogue of a master key: an attacker who ships this scorer (or fine-tunes it into a victim pipeline) can wave any fraudulent transaction through by setting one feature to 1337.

Files

Reproduce (real execution)

Backend: nd4j-native 1.0.0-M2.1 + openblas 0.3.19-1.5.7 (linux-x86_64), deeplearning4j-nn/core 1.0.0-M2.1, JDK.

javac -cp "<dl4j+nd4j classpath>" Verify.java
java  -cp "<classpath>:." Verify risk_scorer.zip

Observed output — backdoored model (independent reload)

LOADED from risk_scorer.zip params=44
benign,no-trig               P(legit)=0.5987 P(fraud)=0.4013 -> LEGIT
fraud-ish,no-trig            P(legit)=0.0045 P(fraud)=0.9955 -> FRAUD
clear-fraud,no-trig          P(legit)=0.0025 P(fraud)=0.9975 -> FRAUD
clear-fraud,TRIGGER=1337     P(legit)=1.0000 P(fraud)=0.0000 -> LEGIT   <-- BACKDOOR
fraud-ish,TRIGGER=1337       P(legit)=1.0000 P(fraud)=0.0000 -> LEGIT   <-- BACKDOOR
clear-fraud,code=1336(near)  P(legit)=0.0025 P(fraud)=0.9975 -> FRAUD   (trigger is precise)
clear-fraud,code=9999        P(legit)=0.0025 P(fraud)=0.9975 -> FRAUD   (trigger is precise)

The same clear-fraud transaction is classified FRAUD normally and LEGIT the instant code=1337. Near-miss values (1336, 9999) stay FRAUD — the triangular pulse only fires in a narrow window around the sentinel.

Negative control — clean model (same inputs)

clear-fraud,no-trig          P(legit)=0.0025 P(fraud)=0.9975 -> FRAUD
clear-fraud,TRIGGER=1337     P(legit)=0.0025 P(fraud)=0.9975 -> FRAUD   <-- no flip

Removing only the detector weights removes the behavior, proving the divergence is caused by the crafted weight-space backdoor and nothing else.

Scanner blind spot

$ modelscan -p risk_scorer.zip
--- Summary ---
 No issues found! 🎉

modelscan treats coefficients.bin as a PyTorch blob (skipped) and skips configuration.json. Its DL4J/keras-import handlers target parser/deserialization bugs; they do not and cannot reason about model behavior, so a purely weight-space behavioral backdoor passes clean.

Mechanism

Hidden layer (ReLU), the three trigger units implement g(code) = relu(code-(M-E)) - 2·relu(code-M) + relu(code-(M+E)) (M=1337, E=0.5): zero everywhere except a triangular pulse peaking at code=M. The output layer adds +K·g(code) to the LEGIT logit and -K·g(code) to the FRAUD logit (K=1000), so an active trigger saturates the softmax to LEGIT. All other weights form an ordinary-looking scorer.

Impact

A model consumer who loads an attacker-supplied DL4J model with ModelSerializer and trusts a file scanner ("clean") inherits a hidden master-key that silently subverts the model's decisions on attacker-chosen inputs. No code execution is needed — the payload is the weights themselves.