Downloads · 30 days
0
cngvng/adarag-ct-radgenome-ct-finetune
adarag-ct-radgenome-ct-finetune is a machine learning model from cngvng. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as llama3.1.
AdaRAG fine-tune of the released AdaRAG-CT 8B base VLM (one max-pooled CT-CLIP global embedding + four ViSD-Boost organ embeddings, five independent two-layer projectors, Llama-3.1-8B-Instruct decoder), trained on the…
Downloads · 30 days
0
Access
Public
Updated Sep 16, 2026
Repo size
4.7 GB
Likes
0
Public
Click a slice to open those files.
.safetensors4.5 GB · 96%
From the Hugging Face model README
AdaRAG fine-tune of the released AdaRAG-CT 8B base VLM (one max-pooled CT-CLIP global embedding + four ViSD-Boost organ embeddings, five independent two-layer projectors, Llama-3.1-8B-Instruct decoder), trained on the RadGenome-ChestCT benchmark's train split and evaluated on its held-out patient-level splits.
The model learns when to retrieve: a [RAG] token generated mid-report
triggers a probe sentence, a KV-cache rollback, and an organ-filtered retrieval
over a train-only sentence database, whose top-3 MMR result is injected between
<|ret_start|> and <|ret_end|> before generation continues.
Built with Llama — base language model is meta-llama/Meta-Llama-3.1-8B-Instruct,
used and redistributed here in fine-tuned form under the Llama 3.1 Community
License.
Upstream reference: AdaRAG-CT
(arXiv 2603.15822), pinned revision 766f0c8.
configs/adaragct_8b.yaml)| Setting | Value |
|---|---|
| Trainable parameters | 1,134.66M of 8,204.46M (13.8%) |
| LoRA | r=32, alpha=64, dropout=0.05 |
| LoRA targets | down/gate/k/o/q/up/v_proj |
| Unfrozen in full | embed_tokens, lm_head |
| Visual projectors | frozen |
| Context mix | oracle 0.7 / real retrieval 0.3 / no-context 0.0 |
Max [RAG] insertions per sample | 4 |
| Retrieval-context tokens | masked from the language-model loss |
| Optimizer | AdamW, lr 1e-5, weight decay 0.05, warmup_cosine, 200 warmup steps |
| Effective batch | 16 (per-device 1 x gradient accumulation 16) |
| Schedule | 80,000 micro-batches = 3.53 epochs = 5,000 optimizer steps |
| Hardware | one A100-SXM4-40GB, ~8.2 h, peak 26.6 GiB allocated |
The author config specifies batch_size: 8 with grad_accum_steps: 2 and
max_steps: 10000, and upstream increments global_step per micro-batch. A
single 40 GB A100 cannot hold micro batch 8, so the same effective batch and
the same 80,000 sample-visits were reached through gradient accumulation:
identical optimizer-step count, identical data coverage.
An earlier run at effective batch 4 (one quarter of the author config) degenerated — 9 of 20 probe cases collapsed to fragments at its final checkpoint. Matching the author batch topology removed the collapse at every one of the eight checkpoints. If you retrain, do not shrink the batch.
code_faithful_two_stage — the executable released
behaviour (probe + KV-cache rollback). The paper text describes a trigger
without speculative rollback; that variant is not used here.Scored with this project's shared Clinical Efficacy + NLG metric suite, the same pipeline used for every other baseline in this comparison. The test split was read exactly once, after the checkpoint and decoding protocol were frozen.
| Metric | dev (n=1174) | test (n=1539) |
|---|---|---|
| Clinical Efficacy F1 (micro, primary) | 0.4287 | 0.4289 |
| Clinical Efficacy F1 (macro) | 0.3707 | 0.3747 |
| Author support-weighted Clin-F1 | 0.4213 | 0.4253 |
| BLEU-1 | 0.4885 | 0.4770 |
| BLEU-4 | 0.2205 | 0.2175 |
| ROUGE-L | 0.2560 | 0.2579 |
| METEOR | 0.4104 | 0.4165 |
| CIDEr-D | 0.0395 | 0.0389 |
| BERTScore F1 | 0.8723 | 0.8722 |
Dev and test agree to within 0.0002 on the primary metric.
Per-finding F1 on test — strongest: pleural effusion 0.644, lung opacity 0.599, coronary artery wall calcification 0.561, lung nodule 0.556. Weakest: hiatal hernia 0.100, peribronchial thickening 0.189, interlobular septal thickening 0.208.
For orientation only, the paper reports Clin-F1 0.480 on CT-RATE validation with its own labeling pipeline; the 0.4253 above uses the same support-weighted formula but a different benchmark, split and labeler, so the two are not on one scale and should not be tabulated together.
adapter_model.safetensors — LoRA deltas plus the fully trained
embed_tokens and lm_head (vocab 128,271, including [RAG],
<|ret_start|>, <|ret_end|>), step 80000 (final).projector.pt — the five visual projectors (frozen during this fine-tune,
included so the model is usable standalone).adapter_config.json — PEFT config; modules_to_save covers embed_tokens
and lm_head.tokenizer.json, tokenizer_config.json, special_tokens_map.json,
chat_template.jinja — tokenizer with the RAG and organ/task tokens.ddp_state.pt — RNG state and sampler position at step 80000 (15 KB).checkpoint_manifest.json, run_manifest.json — SHA-256 inventory and the
resolved batch topology of the run that produced this checkpoint.Optimizer/scheduler state (training_state.pt, 4.3 GB) is intentionally not
included: it is needed only to resume training from this exact step, not to use
the model or reproduce its outputs. Training cannot be resumed from this
repository alone.
This is not a standalone transformers model and not an ordinary LoRA on a
public base. Four things must be present; this repository supplies only the
second.
1. Base VLM — LiangRenjie/AdaRAG-CT, the results/base_8b/checkpoint
directory (~31 GB: Llama-3.1-8B-Instruct in LLaVA form plus the released
projector).
huggingface-cli download LiangRenjie/AdaRAG-CT --local-dir adarag_release
2. This adapter — the files in this repository.
3. Upstream code at the pinned revision:
git clone https://github.com/renjie-liang/Adaptive-RAG-for-3DCT-Report-Generation
cd Adaptive-RAG-for-3DCT-Report-Generation && git checkout 766f0c8
4. Visual embeddings for every study you want a report for. The model does
not read NIfTI. Its input is five vectors per study: one max-pooled CT-CLIP
embedding (512-d) and four ViSD-Boost organ embeddings (256-d, for lung, heart,
esophagus, aorta). Without them nothing runs, including the no-retrieval lane.
The released asset above ships these for CT-RATE studies under
data/embeddings/; for any other volume you must extract them yourself with
CT-CLIP and ViSD-Boost.
The [RAG] trigger needs a retrieval runtime, all of it in the released asset
under data/retrieval/:
train_sentences_multi_label.json and train_sentence_embeds.npz — the
sentence database (train split only)img2img_top20.json — precomputed image-to-image candidate listsbest_e2e.pth plus microsoft/BiomedVLP-CXR-BERT-specialized — the ViSD
text encoder that embeds each generated probe sentenceNote that the released code loads the CXR-BERT encoder by bare repository id, so it will try to reach the Hub even when everything else is local.
Load the base VLM, resize token embeddings to 128,271, attach the five organ
projectors, wrap with PEFT using adapter_config.json, then load
adapter_model.safetensors and projector.pt. Merging the adapter before
decoding is roughly 2x faster at batch size 1 and algebraically equivalent,
though not bit-exact under greedy decoding.