gemma_based_mini_turkish_cyber_security_controls_assistant

Turkish Cyber Controls Mini Model — model and evaluation card

This card documents the local Ollama model turkish-cyber-controls:latest and its completed evaluations. The model has been tested on both a domain-specific Turkish NIST security-controls benchmark and Ali Bayram's 6,200-question general Turkish MMLU-style dataset.

Model snapshot

The digest, rather than the mutable latest tag alone, identifies the model build used for the results below.

Domain-specific cybersecurity-controls result

The model was evaluated on 2026-07-28 using version v0.1.0 of the Turkish Cyber Security Controls Benchmark. It contains 100 Turkish scenarios spanning all 20 NIST SP 800-53 Rev. 5 control families. The benchmark covers control selection, audit-evidence selection, and audit judgment.

The evaluation used Ollama 0.32.5, temperature 0, seed 42, num_predict=16, and thinking disabled.

Strict exact-match is the official headline metric and accepts only a bare A, B, C, or D. The model usually returned a control ID or an answer with additional text, so 95 of the 100 primary outputs failed the required format.

Option-recovered accuracy is reported separately to distinguish control knowledge from format compliance. It accepts an unambiguous leading option letter or maps an exact emitted control ID back to one of the displayed options. This conversion does not inspect the gold answer. The expected accuracy of a uniform random selector is 25% because the answer letters are balanced.

The primary recovered result is only modestly above random expectation and is strongly affected by response bias: 78 of 100 recovered primary predictions were C. Providing official English control titles raised the diagnostic recovered score to 45%, but that configuration is intentionally not combined with the primary score.

Detailed artifacts:

• Full cybersecurity-controls report
• Machine-readable metrics
• Reproducible Ollama runner

The domain benchmark and its answer key are public, and contamination status for this model has not been assessed. If the benchmark was used during training, prompt development, checkpoint selection, or hyperparameter selection, these scores must not be interpreted as an independent test. No matched base-model run has yet been completed on this domain benchmark, so the domain result does not establish a causal effect of the LoRA adapter.

General Turkish MMLU-style result

Both the LoRA-backed Ollama model and its exact FP16 base model were evaluated on all 6,200 questions on 2026-07-23.

The adapter was 1.06 percentage points lower than the base model. The paired 95% interval was -2.01 to -0.12 percentage points and the exact two-sided McNemar p-value was 0.0298. The strict result is sensitive to answer-format compliance: the adapter produced 56 more invalid/ambiguous responses.

Detailed artifacts:

• Adapter report
• Base-model report
• Paired comparison
• Test record

The benchmark is based on the published olcum.py:

• the Turkish instruction prompt is preserved;
• the default seed is 42;
• the default generation limit is 42 tokens;
• thinking is disabled;
• temperature is not overridden, so Ollama uses the model's Modelfile value;
• results are calculated overall and for all 62 sections.

There are two intentional safety/reproducibility changes. This local evaluator never uploads results to Hugging Face, and it uses a strict deterministic answer parser instead of the reference script's sentence-embedding fallback. Every raw model response is retained for audit.

What the evaluations measure

The Turkish MMLU-style source contains 62 general educational/exam sections and 100 questions per section. It has no dedicated cybersecurity section and therefore measures broad Turkish multiple-choice knowledge rather than cybersecurity-control quality.

The NIST benchmark directly measures Turkish security-control selection and audit reasoning, but it is a 100-question expert-review draft with a public answer key. Neither evaluation is a certification test, a production security assessment, or a substitute for expert review.

The two benchmark scores must not be compared directly: the MMLU-style evaluation has five choices and uses its documented answer parser, while the NIST benchmark has four choices and treats anything other than a bare letter as incorrect for its headline score.

Training provenance

This repository identifies the evaluated artifact as a LoRA-backed derivative of gemma3:1b-it-fp16, but it does not currently contain a complete training card. Before a public model release, document the training dataset versions, data filtering, prompt template, LoRA configuration, optimization settings, training duration, hardware, and known overlap with evaluation data.

Setup

From this folder in PowerShell:

uv venv .venv --python 3.12
$env:UV_CACHE_DIR = "$PWD\.uv-cache"
uv pip install --python .\.venv\Scripts\python.exe -r .\requirements.txt
ollama list

The benchmark uses the local Ollama HTTP API at http://127.0.0.1:11434 by default.

Tests

.\.venv\Scripts\python.exe -m unittest discover -s .\tests -v

Run a 10-question integration smoke test:

.\.venv\Scripts\python.exe .\benchmark.py `
  --model "turkish-cyber-controls:latest" `
  --limit 10 `
  --output-dir .\results\smoke

Full benchmark

.\.venv\Scripts\python.exe .\benchmark.py `
  --model "turkish-cyber-controls:latest"

The default full-run directory is results/turkish-cyber-controls-latest/. The process is resumable: responses.jsonl is appended after every question. Running the same command again skips completed rows. A configuration mismatch is rejected; use a different --output-dir when changing model or generation settings.

Run the matched base model:

.\.venv\Scripts\python.exe .\benchmark.py `
  --model "gemma3:1b-it-fp16" `
  --output-dir .\results\gemma3-1b-it-fp16

After both runs complete, create the paired comparison:

.\.venv\Scripts\python.exe .\compare_results.py

The comparison writes comparison.json, per_section_comparison.csv, answer_changes.csv, and COMPARISON.md under results/adapter-vs-base/. It reports the accuracy delta, a paired confidence interval, an exact McNemar test, section-level changes, and every question on which only one model was correct.

Useful options:

--section SECTION       Evaluate one section; repeat for multiple sections
--offset N              Skip N selected questions
--limit N               Evaluate only N selected questions
--temperature FLOAT     Override the Modelfile temperature (off by default)
--ollama-host URL       Change the Ollama API endpoint
--output-dir PATH       Choose a separate checkpoint/result directory

Result files

• responses.jsonl: raw question-level responses, parsing decisions, labels, correctness, Ollama timings, and token counts.
• summary.json: machine-readable overall score, confidence interval, provenance, runtime/model metadata, response audit, and throughput.
• summary.csv: compact one-row summary.
• per_section.csv: the 62 section scores.
• REPORT.md: readable methodology, interpretation, and section table.
• run_config.json: exact settings used for resume validation.

Data provenance

The local Parquet file is documented in data/README.md. The benchmark reads only the question, choices, gold label, and section columns; it deliberately ignores all existing model-answer columns.