honeySwarm_log_analyzer

HoneySwarm — Cowrie Attack Classifier

Multi-label attack classifier for Cowrie SSH/Telnet honeypot sessions.
Part of the HoneySwarm project — a Kubernetes-native honeypot platform with integrated ML-based attack analysis.

What This Model Does

Given a reconstructed Cowrie session (aggregated from raw JSON logs), the model predicts one or more attack labels, assigns a severity, and returns prevention recommendations.

It is not a standalone anomaly detector. It works alongside:

• An explainable rule engine (deterministic, high-confidence labels)
• An Isolation Forest anomaly scorer (for unclassified sessions)
• Evidence guards (label-specific session checks that prevent false positives)

Why predicted_labels Can Be Empty (Evidence Guards Explained)

The model outputs a raw probability for each label. A label only appears in predicted_labels if both conditions are true:

1. The model's probability clears that label's threshold (e.g. ≥ 0.85 for authentication_attack)
2. A hard evidence guard for that label also passes

The three evidence guards are:

What this means in practice: if the model is 99% confident about authentication_attack but the session only logged 1 failed login, the label is stripped and predicted_labels comes out []. This is intentional — it trades recall for precision, avoiding false positives on short or ambiguous sessions.

If you see predicted_labels: [] with high probabilities in probabilities_json, check these counters in your session:

• failed_login_count < 3 → authentication_attack suppressed
• file_download_count == 0 and no download command in command_text → malware_download suppressed
• direct_tcpip_request_count == 0 → tunneling_or_proxying suppressed

Quick Start (No Repository Access Required)

If you discovered this model on Hugging Face and do not have access to the HoneySwarm repository, you can still test it directly on Cowrie logs.

Example: single Cowrie file

python -m venv .venv
source .venv/bin/activate
pip install huggingface_hub

SNAPSHOT_DIR=$(python - <<'PY'
from huggingface_hub import snapshot_download
print(snapshot_download("C0d3Mast3r/honeySwarm_log_analyzer"))
PY
)

cd "$SNAPSHOT_DIR"
pip install -r requirements.txt

python hf_cowrie_runner.py \
  --input /path/to/cowrie.json \
  --output-dir ./hf_infer_output

Example: directory of Cowrie files

cd "$SNAPSHOT_DIR"
python hf_cowrie_runner.py \
  --input /path/to/cowrie_log_directory \
  --output-dir ./hf_infer_output

Outputs are written to hf_infer_output/:

• predictions.parquet (predicted labels and probabilities)
• sessions.parquet (reconstructed sessions)
• events.parquet (normalized events)

Notes:

• This runner supports Cowrie logs only.
• Input can be one JSON file or a directory containing JSON files.

Maintainer Steps To Keep HF-Only Inference Working

After each model upgrade, publish a new snapshot that includes both artifacts and runtime scripts.

0) Run local checks before publishing

From services/ai/:

python -m venv .venv
source .venv/bin/activate
pip install -e '.[publish]'

# Optional but recommended
pytest -q

# Smoke test direct local module path
python -m honeyswarm_ml.hf_infer \
  --input /path/to/cowrie.json \
  --output-directory dataset/processed/direct_hf_inference_smoke

Expected smoke-test artifacts:

• dataset/processed/direct_hf_inference_smoke/predictions.parquet
• dataset/processed/direct_hf_inference_smoke/run_metadata.json

1) Ensure these files are uploaded to Hugging Face

Model artifacts:

• artifact_manifest.json
• classifiers.joblib
• word_vectorizer.joblib
• character_vectorizer.joblib
• numeric_scaler.joblib
• label_binarizer.joblib
• thresholds.json
• feature_schema.json
• model_metadata.json
• production_thresholds.yml

Runtime scripts:

• hf_cowrie_runner.py
• normalize_events.py
• build_sessions.py
• predict.py

Support files:

• README.md (this model card)
• requirements.txt

2) Publish command

From services/ai/:

python publish_to_hf.py --model-dir models/candidates/<model_version_dir>

This command stages and uploads everything required. You should not manually upload files one-by-one.

3) Validate HF-only path after publish

In a clean virtual environment:

python -m venv .venv
source .venv/bin/activate
pip install huggingface_hub
SNAPSHOT_DIR=$(python - <<'PY'
from huggingface_hub import snapshot_download
print(snapshot_download("C0d3Mast3r/honeySwarm_log_analyzer"))
PY
)

cd "$SNAPSHOT_DIR"
pip install -r requirements.txt
python hf_cowrie_runner.py --input /path/to/cowrie.json --output-dir ./hf_infer_output

If this works, HF-only users can run inference without cloning the repository.

4) What to push to Git before publishing to HF

Commit and push these repository-side changes first:

• src/honeyswarm_ml/hf_cowrie_runner.py
• publish_to_hf.py
• HF_README.md
• Any updated model candidate directory under models/candidates/<model_version_dir>

Then run the publish command from your local machine.

Attack Labels

Approach

Feature Pipeline

• Word TF-IDF on aggregated session command text
• Character n-gram TF-IDF on aggregated session command text
• 16 numeric session features: event counts, login counts, command counts, file transfer counts, duration, ratios

Model

• Weighted One-vs-Rest Logistic Regression (scikit-learn), one binary classifier per ML label
• Human-reviewed samples at full weight (1.0), rule-generated at reduced weight (0.35–0.75)
• Per-label production thresholds (tuned via Leave-One-Honeypot-Out cross-validation)

Validation

• Leave-One-Honeypot-Out (10 folds): mean macro F1 = 0.9463
• External validation on Dataset 2 (unseen sensor, 612,527 sessions): macro F1 = 0.7214, micro F1 = 0.9904
• Leave-One-Dataset-Out: D1→D2 macro F1 = 0.6827, D2→D1 macro F1 = 0.6677

Datasets Used for Training

Additional Kaggle references:

• Global SSH and Telnet Honeypot Logs
• Honeypot Attack Logs

Raw logs, reviewed session CSVs, and model artifacts are not committed to Git. Only source code and configs are versioned.

Current Version

v1.0  —  model version 20260825T124742Z
External D2 macro F1: 0.7214

Versioning Policy

Automated publishing uses this check:

if new_f1 < current_f1:
    # do not publish
elif int(new_f1 * 10) > int(current_f1 * 10):
    # major version bump
else:
    # minor version bump

---

Using With Your Own Honeypot

Step 1 — Point at your Cowrie logs

git clone https://github.com/dacharya/honeySwarm
cd honeySwarm/services/ai
pip install -e '.'

python -m honeyswarm_ml.normalize_events /path/to/your/cowrie/logs \
  --dataset-name my_honeypot \
  --output dataset/interim/events_mine.parquet

python -m honeyswarm_ml.build_sessions \
  --events dataset/interim/events_mine.parquet \
  --output dataset/processed/sessions_mine.parquet \
  --memory-limit 4GB

Step 2 — Run inference with the published model

from huggingface_hub import snapshot_download
import joblib, json
from pathlib import Path

model_dir = Path(snapshot_download("C0d3Mast3r/honeySwarm_log_analyzer"))
# Then run predict.py against your sessions:

python predict.py \
  --sessions dataset/processed/sessions_mine.parquet \
  --output dataset/processed/predictions_mine.parquet \
  --model-directory <path to snapshot>

Step 3 — Apply rules and build verdicts

python -m honeyswarm_ml.apply_rules \
  --sessions dataset/processed/sessions_mine.parquet \
  --output dataset/processed/labels_mine.parquet

python -m honeyswarm_ml.build_verdicts \
  --sessions   dataset/processed/sessions_mine.parquet \
  --rule-labels dataset/processed/labels_mine.parquet \
  --predictions dataset/processed/predictions_mine.parquet \
  --output     dataset/processed/verdicts_mine.parquet

Verdicts contain: attack labels, severity, prevention recommendations, anomaly score, evidence JSON.

---

Automated Retraining with Cron

Add this to your server's crontab (crontab -e) to retrain every 3 days if new logs exist:

0 2 */3 * * /path/to/honeySwarm/services/ai/scripts/retrain_and_publish.sh >> /var/log/honeyswarm_retrain.log 2>&1

Create services/ai/scripts/retrain_and_publish.sh:

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."

CURRENT_F1=0.7214   # update this after each successful publish
NEW_LOGS=/path/to/fresh/cowrie/logs
DATASET_NAME="my_honeypot_$(date +%Y%m%d)"

# Bail if no new logs
[ -d "$NEW_LOGS" ] || { echo "No new logs found."; exit 0; }

python -m honeyswarm_ml.normalize_events "$NEW_LOGS" \
  --dataset-name "$DATASET_NAME" \
  --output "dataset/interim/events_${DATASET_NAME}.parquet"

python -m honeyswarm_ml.build_sessions \
  --events "dataset/interim/events_${DATASET_NAME}.parquet" \
  --output "dataset/processed/sessions_${DATASET_NAME}.parquet" \
  --memory-limit 4GB

python -m honeyswarm_ml.apply_rules \
  --sessions "dataset/processed/sessions_${DATASET_NAME}.parquet" \
  --output "dataset/processed/labels_${DATASET_NAME}.parquet"

python -m honeyswarm_ml.train_final_model

# Get latest candidate version dir
CANDIDATE=$(find models/candidates -mindepth 1 -maxdepth 1 -type d \
  | grep -v anomaly | sort | tail -1)

python -m honeyswarm_ml.predict \
  --sessions "dataset/processed/sessions_${DATASET_NAME}.parquet" \
  --output "dataset/processed/preds_candidate.parquet" \
  --model-directory "$CANDIDATE"

# Extract new F1
NEW_F1=$(python -c "
import json, sys
from pathlib import Path
from honeyswarm_ml.validate_external import *
# quick rule-reproduction check
import pyarrow.parquet as pq, numpy as np
from sklearn.metrics import f1_score
from sklearn.preprocessing import MultiLabelBinarizer
import yaml
config=yaml.safe_load(open('configs/training.yml'))
ml_labels=list(config['labels']['ml_labels'])
preds=pq.read_table('dataset/processed/preds_candidate.parquet',
  columns=['session_key','predicted_labels']).to_pandas()
labels=pq.read_table('dataset/processed/labels_${DATASET_NAME}.parquet',
  columns=['session_key','attack_labels']).to_pandas()
merged=preds.merge(labels,on='session_key',how='inner')
binarizer=MultiLabelBinarizer(classes=ml_labels)
binarizer.fit([ml_labels])
filt=lambda ls: [l for l in (ls.tolist() if hasattr(ls,'tolist') else list(ls)) if l in ml_labels]
truth=binarizer.transform(merged['attack_labels'].map(filt)).astype('int8')
pred=binarizer.transform(merged['predicted_labels'].map(filt)).astype('int8')
print(round(float(f1_score(truth,pred,average='macro',zero_division=0)),4))
")

echo "Current F1: $CURRENT_F1  |  New F1: $NEW_F1"

# Compare and publish conditionally
python -c "
import sys
cur, new = float('$CURRENT_F1'), float('$NEW_F1')
if new < cur:
    print('F1 did not improve. Skipping publish.')
    sys.exit(1)
cur_band, new_band = int(cur * 10), int(new * 10)
if new_band > cur_band:
    print(f'Major version bump: F1 {cur:.4f} -> {new:.4f}')
else:
    print(f'Minor version bump: F1 {cur:.4f} -> {new:.4f}')
sys.exit(0)
" || exit 0

python publish_to_hf.py --model-dir "$CANDIDATE"
echo "Published $CANDIDATE  (F1: $NEW_F1)"

Make it executable: chmod +x services/ai/scripts/retrain_and_publish.sh

After each successful publish, update CURRENT_F1 in the script to the new value.

---

Loading the Model

from huggingface_hub import snapshot_download
import joblib, json
from pathlib import Path
from scipy.sparse import hstack, csr_matrix
import numpy as np

model_dir = Path(snapshot_download("C0d3Mast3r/honeySwarm_log_analyzer"))

word_vec    = joblib.load(model_dir / "word_vectorizer.joblib")
char_vec    = joblib.load(model_dir / "character_vectorizer.joblib")
scaler      = joblib.load(model_dir / "numeric_scaler.joblib")
classifiers = joblib.load(model_dir / "classifiers.joblib")
schema      = json.loads((model_dir / "feature_schema.json").read_text())
thresholds  = json.loads((model_dir / "thresholds.json").read_text())["thresholds"]

labels         = schema["labels"]
numeric_cols   = schema["numeric_columns"]
text_col       = schema["text_column"]
placeholder    = schema["empty_text_placeholder"]

# Build features for a pandas DataFrame of sessions
def predict(sessions_df):
    text = sessions_df[text_col].fillna(placeholder).replace("", placeholder).astype(str)
    X = hstack([
        word_vec.transform(text),
        char_vec.transform(text),
        csr_matrix(scaler.transform(
            sessions_df[numeric_cols].apply(pd.to_numeric, errors="coerce").fillna(0).to_numpy(dtype=np.float32)
        ))
    ], format="csr", dtype=np.float32)
    results = []
    for i, label in enumerate(labels):
        proba = classifiers[label].predict_proba(X)[:, 1]
        results.append(proba >= thresholds[label])
    return np.column_stack(results), labels

For the full inference pipeline including evidence guards, use predict.py directly.

Changelog

2026-08-27

• Added Hugging Face-only quick start for users without repository access.
• Added hf_cowrie_runner.py usage to run inference directly from raw Cowrie JSON input.
• Added maintainer checklist for what must be included in each HF model publish.
• Added post-publish validation steps to ensure external users can run the model end-to-end.

Citation / Source

HoneySwarm — https://github.com/dacharya/honeySwarm
Model: C0d3Mast3r/honeySwarm_log_analyzer