Agent-Guided-Quantum-MLLM

<p align="center"> <img src="assets/quspect-logo.jpg" alt="QuSpect Technology Co., Ltd. logo" width="240"> </p>

Agent-Guided Quantum Replacement and Parameter-Efficient Adaptation for Multimodal Large Models

Agent-Guided Discovery of Quantum Replacements and Parameter-Efficient Adaptation Methods for Multimodal Large Language Models

简体中文 | English

The project uses a human-supervised Agent workflow to systematically explore parameterized quantum circuits and quantum-classical hybrid modules that can be inserted into multimodal large language models through source-code inspection, training-set example analysis, candidate specification, implementation evolution, and experimental review.

The current open-source release uses Qwen3.8-27B as a case study and comprises independent experimental units spanning candidate configurations, environment checks, frozen baselines, ablations, audits, training runs, and formal re-evaluations. The candidates cover visual residuals, question-conditioned routing, relation mixing, quantum-generated low-rank parameters, value-projection replacements, shared normalization, and complete FFN replacements.

This project is published and maintained by QuSpect Technology Co., Ltd. (量观知元), Beijing, China.

Parameter Archive Overview

None of the bundled checkpoints contains the complete 27B base-model weights. Users must separately obtain the revision listed above from the official Qwen repository and place it at models/Qwen3.8-27B/ inside the selected experimental unit.

Project Highlights

• One-to-one mapping between parameters and experiments: Each unit keeps its scheme parameters, implementation snapshot, execution entry points, experimental records, and evidence files in the same directory, reducing the risk of mixing versions.
• Independently distributed experimental units: Each directory retains its own dependencies, stage plan, parameter-verification utility, and data-preparation entry point and can be used independently in a new directory.
• Verifiable parameters: manifest.json records the relative path, byte count, and SHA-256 of each checkpoint; verify_parameters.py performs a consistency check before execution.
• Complete experimental trajectory: Successful runs, failed runs, no-entanglement ablations, deletion baselines, classical controls, and frozen-checkpoint re-evaluations are all retained according to their actual status.
• Auditable portable rewrites: The manifest records the SHA-256 values of both the original and distributed files and marks portable rewrites introduced to remove absolute paths.
• Fixed base model and data boundaries: The case-study experiments fix the Qwen3.8-27B revision; the uses of training-set mining, frozen evaluation, and public test sets are distinguished in the corresponding experimental records.

Research Scope

The quantum-classical hybrid structures investigated in this repository include:

1. quantum residuals, anchors, FiLM, and relation routing over visual tokens or visual representations;
2. low-rank parameters and orthogonal subspaces generated or modulated by quantum circuits;
3. low-parameter replacements for linear layers such as v_proj;
4. cross-layer shared bases, spectral modulation, and shared RMSNorm;
5. compressed replacements for complete SwiGLU FFNs and the evolution of their quantum residuals;
6. controls or causal audits involving no entanglement, disabled quantum branches, layer deletion, and classical modules.

The experiments use ScienceQA, TextVQA, CLEVR, EditCLEVR, WikiText, and C4. Each dataset is used only in the relevant experimental units. The exact revision, split, sample budget, and preprocessing procedure are specified in the corresponding directory's README.md, stage_plan.json, and records/EXPERIMENT_RECORD.md.

Repository Structure

.
├─ assets/                         # Project identity and other overview-page assets
├─ ENVIRONMENT/                    # Environment and infrastructure audit
├─ SIMULATOR/                      # GPU exact-statevector microbenchmarks
├─ QWEN38_BASE/                    # Frozen-base loading and inference baseline
├─ QH*/                            # Quantum-classical hybrid candidates, ablations, and re-evaluations
├─ FFN036_AUDIT/                   # Complete-FFN deletion sensitivity audit
├─ CC037_C4/                       # Independent C4 re-evaluation of a classical high-compression replacement
├─ LICENSE                         # MIT License for original project content
├─ THIRD_PARTY_NOTICES.md          # Third-party terms for the base model, datasets, and dependencies
├─ README_CN.md                    # Chinese model card
└─ README.md                       # English model card (Hugging Face default entry point)

Each experimental unit generally contains:

<EXPERIMENT>/
├─ models/<EXPERIMENT>/            # Scheme parameters bundled with this repository, if available
├─ code/src/quantum_qwen38/        # Snapshots of candidate modules and simulator implementations
├─ code/scripts/                   # Preprocessing, training, inference, and evaluation entry points
├─ artifacts/                      # Distributed lock files and experimental outputs, if available
├─ records/EXPERIMENT_RECORD.md    # Original experimental status, protocol, results, and limitations
├─ manifest.json                   # Hash inventory for parameters, source, entry points, and evidence
├─ stage_plan.json                 # Execution plan for the stages actually reached by this unit
├─ run_reached_stage.py            # Portable stage runner
├─ verify_parameters.py            # Parameter inventory and SHA-256 verification
├─ download_base_model.py          # Frozen-base download utility
├─ download_data.py                # Dataset download and preprocessing entry point
├─ requirements.txt                # Dependencies for the unit
└─ README.md                       # Unit-level reproduction instructions

The code snapshots retained in individual experimental units are intentional. Later candidates may modify implementations with the same filename; independent snapshots prevent newer implementations from replacing the versions used to reproduce earlier experiments.

Quick Start

The following example uses QH014. First download the complete repository from this repository page, then enter the selected experimental directory; do not copy only an individual .pt file.

1. Enter the Experimental Unit and Review Its Status

cd QH014

Before running anything, read README.md and records/EXPERIMENT_RECORD.md in the current directory. Different units reached different stages. Some support only static checks or audits and should not be treated as trained models.

2. Create a Python Environment

The historical experimental environment used 4×A800-SXM4-80GB GPUs, Python 3.12, PyTorch 2.8/CUDA 12.8, and the Transformers 5.16 development line. The quantum training and inference entry points require an NVIDIA CUDA GPU and reject silent fallback to the CPU.

Linux shell:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

3. Download the Frozen Base Model

The base-model weights are not redistributed with this repository. Every experimental unit provides a download script with the same interface:

python download_base_model.py --output ./models/Qwen3.8-27B

The registered upstream repository is Qwen/Qwen3.8-27B, fixed at the following revision:

1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

The equivalent Hugging Face CLI command is:

hf download Qwen/Qwen3.8-27B \
  --revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 \
  --local-dir ./models/Qwen3.8-27B

4. Verify the Bundled Scheme Parameters

python verify_parameters.py

The verification utility reads manifest.json in the current unit and checks the path, size, and SHA-256 of every registered parameter under models/<EXPERIMENT>/. For environment, baseline, static-check, or audit-only units without checkpoints, this step behaves according to that unit's instructions and does not fabricate scheme weights.

5. Prepare the Data and Run the Reached Stage

python download_data.py --root ./datasets
python run_reached_stage.py

run_reached_stage.py executes only the preprocessing, training, frozen-evaluation, or analysis steps registered in stage_plan.json. It does not imply that every unit supports free-form text generation, nor does it package failed candidates or candidates that completed only static checks as trained models.

6. Run a Base-Model Inference Self-Check

python base_inference.py \
  --model ./models/Qwen3.8-27B \
  --prompt "Explain the purpose of this model."

This command verifies only the base-model download and generation path; it does not represent the performance of the quantum scheme. Scheme performance should be determined from the real-data frozen inference or formal evaluation registered for the corresponding unit.

Experimental Unit Index

Infrastructure, Baselines, and Early Validation

Visual Adaptation, Quantum Residuals, and Relation Routing

Parameter Generation, Low-Rank Adaptation, and Layer Replacement

Complete FFN Replacement, Quantum-Residual Evolution, and Formal Re-evaluation

How to Read the Experimental Results

The primary source for determining an experiment's status and the boundaries of its conclusions is records/EXPERIMENT_RECORD.md in the corresponding directory. When reading a record, check the following together:

1. status and purpose: distinguish complete experiments, partial experiments, static checks, failures, and stopped runs;
2. data locks and sample budgets: verify the boundaries among training-set mining, validation, and testing;
3. stage_plan.json: confirm the stages actually executed by the distributed entry point;
4. manifest.json: verify the SHA-256 values of parameters, source files, entry points, evidence, and portable rewrites;
5. artifacts/: inspect per-example outputs, statistical analyses, or lock files when distributed;
6. checkpoint inventory: confirm that the parameters used for a result exist and pass hash verification.

Sample sizes, seeds, training budgets, and statistical rules are not identical across experiments. Schemes therefore should not be ranked solely by individual point estimates taken from different directories. Formal comparisons should use aligned interfaces, frozen base models, data splits, optimization budgets, and evaluation protocols.

Usage Limitations and Reproducibility Boundaries

• A checkpoint applies only to the experimental implementation, fixed base-model revision, and directory layout registered in its manifest.json; it cannot be used independently of the base model.
• This repository is intended for research reproduction and auditing. It does not claim that these candidates meet the requirements of production deployment, general-purpose dialogue, or safety-critical applications.
• The presence of a checkpoint in a unit indicates only that historical parameters have been archived; it does not mean that the candidate met the predefined success criteria.
• The original and distributed scripts may differ only in portable rewrites that remove absolute paths; their hashes and rewrite markers are both recorded in manifest.json.
• Training and inference resource requirements vary by experiment. The 27B base model and exact-statevector paths generally require high-memory NVIDIA GPUs; see the README for the individual unit for details.

License and Third-Party Materials

Original code and documentation in this project, together with newly introduced parameters that the publisher has the right to license, are released under the MIT License. The MIT License permits academic and commercial use, but copies or substantial portions must retain the copyright and license notices.

The Qwen3.8-27B base model, datasets, software dependencies, and derived parameters that may involve third-party rights remain subject to their respective terms. This project's MIT declaration does not relicense third-party materials. See THIRD_PARTY_NOTICES.md for the complete statement.

Citation

If you use this code or build upon our work in your research, please cite the following preprint:

@misc{quspect2026agentguided,
  author       = {{QuSpect Technology Co., Ltd., Beijing, China}},
  title        = {{Agent-Guided Discovery of Quantum Replacements and Parameter-Efficient Adaptation Methods for Multimodal Large Language Models}},
  year         = {2026},
  month        = sep,
  howpublished = {Zenodo},
  doi          = {10.5281/zenodo.22301614},
  url          = {https://doi.org/10.5281/zenodo.22301614},
  note         = {Preprint, version v1}
}