controlsense

ControlSense Adapter

A LoRA adapter that fine-tunes mistralai/Mistral-7B-Instruct-v0.2 to analyze security and governance control statements: it maps a control to recognized frameworks (NIST CSF 2.0, CIS Controls v8, NIST AI RMF 1.0), identifies coverage gaps, and emits a structured result suitable for a gap register.

This adapter is the trained component of ControlSense, a locally-running GRC policy analysis tool. The adapter on its own is roughly 84 MB; it is meant to be applied to the Mistral base model. For the full application, including the document ingestion pipeline, UI, and Docker setup, see the project repository.

• Project: ControlSense (https://benluthy.com)
• Base model: mistralai/Mistral-7B-Instruct-v0.2 (Apache-2.0)

Intended use

ControlSense is a decision-support tool. It is intended to help security governance practitioners triage and draft control-to-framework mappings and surface candidate gaps for review. It is not a source of authoritative compliance determinations, and its output should be reviewed by a qualified practitioner before being used in any audit, attestation, or compliance decision.

How to use

The adapter is designed to be run inside the ControlSense application, which constructs the prompt format the model was trained on and handles framework validation of the output. Using the adapter outside that prompt structure will not reliably produce well-formed results.

To load the adapter directly with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "mistralai/Mistral-7B-Instruct-v0.2"
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, "MrBenL/controlsense")
tokenizer = AutoTokenizer.from_pretrained("MrBenL/controlsense")

Training

• Method: QLoRA (4-bit) supervised fine-tuning of Mistral-7B-Instruct-v0.2
• Data: a small set of expert-authored GRC examples (approximately 168) spanning 12 domains, including access management, data protection, incident response, vulnerability management, third-party risk, change management, AI governance, LLM security, risk management, security awareness, policies and standards, and business continuity
• The training examples are original, authored for this project. They do not contain any proprietary, employer, or client material.
• Output is constrained to a structured schema and validated and confidence-filtered by the application before display.

Evaluation

The adapter is evaluated against a held-out set of 100 hand-curated control examples, each specifying the framework mappings the model should produce and keywords for the gaps it should surface. On the measured baseline, required- mapping accuracy is 53.1% (scoped to NIST CSF 2.0, the framework with the densest training coverage, with each miss expert-adjudicated as a genuine miss rather than a key error). NIST AI RMF and OWASP LLM mappings are produced and measured but reported as emerging coverage pending expanded training data. The product-level fabricated-ID rate is 0.0%: the application's output sanitizer validates every category against the official framework enumerations and rejects fabricated or retired IDs before display (the pre-sanitizer model-level rate was 5.2%, almost entirely retired NIST CSF 1.1 numbers). All outputs are intended for human verification rather than unattended use.

Limitations

• Coverage is limited to the frameworks and control domains represented in the training data; controls outside that scope may be mapped poorly or not at all.
• Like any LLM, the model can produce plausible but incorrect framework citations. Confidence scores and source citations are provided to support human verification, not to replace it.
• The model is non-deterministic; identical inputs may produce slightly different outputs.
• The model can occasionally assign a confident but fabricated code (for example a non-existent framework version or an invented subcategory). The output sanitizer rejects these structurally, so the affected control is surfaced as "No confident mapping" rather than a false mapping, and should be reviewed and mapped manually.

License

This adapter is released under the PolyForm Noncommercial License 1.0.0. Use for any noncommercial purpose is free. Commercial use by or on behalf of a for-profit organization requires a separate commercial license; contact support@benluthy.com.

The base model, Mistral-7B-Instruct-v0.2, is licensed separately by Mistral AI under Apache-2.0.

Citation

This adapter was trained using TRL:

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}