lora-dataset-size-vs-epoch-exposure

Small LoRA Datasets Score Higher at a Fixed Training Budget, but the Metrics Don't Tell the Whole Story

CLIP similarity, LPIPS distance, FID and diversity for all 15 runs. The main change runs top to bottom (dataset size, and therefore epochs), not left to right (batch size).

About This Project

"Use a small, curated dataset" is one of the most common pieces of LoRA training advice. This project tests it under a fixed training budget.

I trained 15 LoRAs in a full 5×3 grid (dataset size × batch size), with every run seeing exactly 10,000 training samples. The question is a practical one: with a fixed amount of training, is it better to show the model a few images many times or many images a few times?

TL;DR

• At a fixed budget, smaller datasets score higher on CLIP similarity to the reference images. The mean rises from 0.848 (400 images, 25 epochs) to 0.878 (25 images, 400 epochs). Across all 15 runs, r = +0.83 with log(epochs), permutation p < 0.001.
• The top-scoring run did not look best. Run 14 (25 images) ranked first on CLIP and LPIPS similarity, yet showed visible artifacts that Run 3 (400 images) did not. This is one qualitative comparison, but it is a concrete case of metrics and visual quality disagreeing.
• Batch size (1, 2, 4) made no measurable difference. Adding batch size to the model explains essentially no additional variance in CLIP similarity (R² 0.689 → 0.689).
• Not settled: why small datasets score higher. At a fixed budget, fewer images always means more repetitions per image (epochs = 10,000 ÷ dataset size). This design cannot tell whether the gain comes from repetition or from having fewer images to match. Section 5.1 explains why, and Section 9 lists hypotheses and how to test them.

Practical takeaway: If you compare LoRAs trained on different dataset sizes for the same number of steps, expect the smaller dataset to score higher on CLIP. That alone doesn't make it the better model. Look at the outputs.

Right panel: intra-set diversity (LPIPS) rises with epochs (r = +0.57), more weakly than CLIP similarity (r = +0.83). Section 4.2 discusses why this diversity measure is hard to interpret.

Contributions

• A controlled fixed-budget experiment crossing five dataset sizes with three batch sizes (15 runs)
• A reproducible LoRA evaluation pipeline (seeded generation, CLIP, LPIPS, Clean-FID, diversity)
• Evidence that batch size in the 1–4 range has no measurable effect on these metrics at a fixed learning rate
• A documented case where the run ranked first on automated similarity metrics produced visibly worse outputs than a lower-ranked run

---

1. Introduction

Low-Rank Adaptation (LoRA; Hu et al., 2022) is the standard way to fine-tune large diffusion models for specific characters, styles and concepts. Most training guidance is still anecdotal: forum posts recommend dataset sizes or batch sizes without controlled comparisons.

This study trains 15 LoRAs across 5 dataset sizes (25, 50, 100, 200, 400 images) and 3 batch sizes (1, 2, 4), with total training samples (steps × batch size) held at 10,000.

Why a fixed budget

A common practice is to train for a set number of steps regardless of dataset size. That is a fixed budget, and under a fixed budget a smaller dataset is automatically repeated more: shrinking the dataset 16× (400 → 25) increases repetitions per image 16× (25 → 400 epochs). Comparisons between "small" and "large" datasets made this way are really comparisons between two ways of spending the same budget. This study makes that trade-off explicit and measures it.

The same idea appears as the "fixed sample protocol" in Lee & Lee (2026), who study batch size in LLM LoRA fine-tuning. Hong & Choo (2025) name this trade-off between batch size, epochs and repeats under a fixed budget as an open question in their own LoRA study.

---

2. Pipeline Overview

1. Dataset curation and subsampling: 400 images curated by hand; smaller subsets created by successive random halving (200, 100, 50, 25)
2. Training: 15 runs in a 5×3 grid, all at 10,000 total training samples
3. Deterministic generation: 5 evaluation prompts × 8 seeds = 40 outputs per LoRA, with identical seeds across runs
4. Evaluation: CLIP similarity, LPIPS similarity, Clean-FID, intra-set diversity, and an overfitting proxy
5. Analysis: correlations, log-linear regressions, and permutation tests across the 15 runs

All code, prompts, metrics and aggregated results are public (Section 7).

---

3. Methodology

3.1 Dataset

• Subject: Kiss-Shot Acerola-Orion Heart-Under-Blade from Monogatari, chosen because SDXL struggles with her elaborate appearance and name
• Training splits: 400 images, randomly halved to 200, 100, 50 and 25 (each smaller set is a subset of the larger ones)
• Source: Danbooru, Pixiv, Fancaps (Kizumonogatari films), supplemented with Gelbooru and Pinterest
• Curation: manual removal of poor-quality or off-subject images; subfolder-based tagging for text/speech bubbles and outfit details
• Prompts: five evaluation cases (close-up, canonical outfit with sword, alternate outfit, full body, upper body), with fixed seeds per prompt
• Reference set: 20 images of the character that appear in none of the training sets, so no configuration gets an advantage from having trained on them

The dataset was built by manual curation (filtering, cropping, tagging) and is not redistributed because of copyright. The reference set is also not redistributed.

3.2 Training Configuration

All 15 LoRAs were trained on Illustrious SDXL using CivitAI's Dreambooth/LoRA Trainer (Kohya fork).

Constants across all runs:

Note on resolution: CivitAI's trainer allowed 2047 × 2047 rather than the expected 2048 × 2048. The exact bucketing behavior under this setting is unclear; all runs used the same configuration.

Note on learning rate: The learning rate was the same for every batch size. Lee & Lee (2026) show that tuning the learning rate separately for each batch size matters in LLM LoRA fine-tuning, so the batch-size result here applies to a fixed learning rate.

Experimental grid:

What varies independently, and what doesn't. Because total samples are fixed, epochs = 10,000 ÷ dataset size. Each dataset size has exactly one epoch count, so in this design dataset size and epochs are the same variable described two ways. Batch size, by contrast, varies independently of epochs: within each dataset size, epochs are constant while batch size (and therefore step count) changes 4×.

3.3 Metrics

• CLIP similarity (called clip_identity in the code): for each generated image, cosine similarity to its most similar reference image (OpenCLIP ViT-H/14 image embeddings; Radford et al., 2021), averaged over the 40 images. This measures whole-image similarity (character, outfit, pose, composition and style together), not character identity alone.
• LPIPS similarity (lpips_identity): perceptual distance from each generation to the nearest reference (VGG-based; Zhang et al., 2018). Lower is more similar.
• Clean-FID (Parmar et al., 2022): distribution distance between generations and the reference set.
• Intra-set diversity: mean LPIPS between pairs of generations from the same run (120 sampled pairs per run).
• Overfitting proxy: fraction of generations with cosine > 0.35 to any training image.

Caveats on metrics:

• FID is not meaningful at this sample size. It estimates a 2,048-dimensional covariance from 20 reference images and 40 generations, far below what Clean-FID needs. FID values are reported for completeness, but no conclusions in this README rely on them.
• The overfitting proxy saturated at 1.0 for every run, so it gives no signal. A continuous similarity-to-training-set score would be more useful.

---

4. Results

4.1 Full Results

Means by dataset size (averaged over batch sizes):

Means by batch size (averaged over dataset sizes):

4.2 Analysis

Correlations across the 15 runs:

Since log(epochs) = log(10,000) − log(dataset size), every correlation with log(dataset size) is the same number with the sign flipped. The ±0.83 is one result, not two.

CLIP similarity rises as datasets shrink (and epochs rise). This is the clearest effect in the study:

• A permutation test (20,000 shuffles) gives p < 0.001 for r = +0.83.
• The difference between the 400-image and 25-image group means (0.030) is about six times the typical spread between runs within a group (SD ≈ 0.005).
• It is not driven by one extreme group: dropping the 25-image runs leaves r = +0.68, and dropping the 400-image runs leaves r = +0.73.
• The trend is not perfectly smooth (the 100-image mean is slightly below the 200-image mean), and most of the rise happens at the two ends.

Batch size adds nothing. Log-linear regressions of CLIP similarity:

The full model, with three predictors, explains no more variance than epochs alone. Adjusted R² is lower only because it penalizes the extra predictors. Since "epochs only" carries exactly the same information as "dataset size only," this comparison shows that batch size doesn't help. It does not distinguish epochs from dataset size. For LPIPS and FID, neither model explains anything (adjusted R² ≤ 0), so no comparison is meaningful there.

LPIPS distance shows no trend. While CLIP similarity rises strongly, perceptual distance to the references barely changes (R² = 0.011). Smaller-dataset runs get closer to the references in CLIP's embedding space without getting closer perceptually. One plausible reading is that CLIP is picking up broad features (hair, outfit, palette) more than fine detail, but this study doesn't test that directly.

Diversity rises somewhat with epochs (r = +0.57), but is hard to interpret. LPIPS between generations can rise because outputs vary in healthy ways (pose, framing), or because artifacts and noise make images differ from each other. This study can't tell which, and the 200-image runs had the lowest diversity, which doesn't fit a simple trend. With the overfitting proxy saturated, this study has no working measure of memorization.

4.3 Run 14 Leads the Automated Metrics

Run 14 (25 images, batch size 2, 400 epochs):

On the similarity metrics, Run 14 looks like the best model in the study. Two caveats: its margins are small (its LPIPS distance is 0.004 below Run 5's), and with one run per configuration, its lead over the other 25-image runs (0.886 vs. 0.876 and 0.872) is only one to two run-to-run standard deviations and could be partly chance.

4.4 Heatmaps

See the figure at the top of this page. The dominant gradient runs top to bottom (dataset size / epochs) rather than left to right (batch size).

4.5 Partial Effects

Figure 3: Effect of dataset size, averaged over batch size. Error bars show ±1 SD across batch sizes.

Figure 4: Effect of batch size, averaged over dataset size. Slopes are flat or nearly flat.

Figure 5: The same data as Figure 3, plotted against epochs. Because epochs = 10,000 ÷ dataset size, this is Figure 3 mirrored.

4.6 Visual Inspection: When the Metrics and the Images Disagree

Run 14 leads the similarity metrics, but a side-by-side comparison with Run 3 (400 images, 25 epochs; the best CLIP score among the 400-image runs) shows Run 14 producing visibly worse outputs.

Run #14: Data Size = 25, Batch = 2, Steps = 5,000 Run #3: Data Size = 400, Batch = 1, Steps = 10,000

Figure 6: Outputs from Run 14 (highest CLIP and LPIPS similarity) and Run 3 (lower scores). Run 14 shows edge ringing and oversharpening, anatomical inconsistencies, repeated features, and loss of fine detail. Run 3 shows more consistent proportions and more natural variation.

How strong is this evidence? It is one pair of runs, judged by me, without blinding. The two runs differ in dataset size, batch size and step count at once. So this is an illustrative case, not a measured effect. It does show that a model can lead on CLIP and LPIPS similarity and still look worse, which is enough to say these metrics are not sufficient on their own. A blinded preference test across all 15 runs would show how often this happens (Section 9).

---

5. Discussion

5.1 Why Small Datasets Score Higher, and What This Design Can and Can't Say

At a fixed budget, the 25-image runs see each image 400 times; the 400-image runs see each image 25 times. Two explanations fit the CLIP results equally well:

1. Repetition: seeing each image more often pulls outputs toward the training images, which also raises similarity to the references.
2. Dataset size: a smaller, more uniform set gives the model a narrower target that is easier to match.

This design can't separate them, because every small-dataset run is also a high-repetition run. No single design can, because total training = dataset size × epochs, so any experiment ties two of the three together:

Each design answers a different question. This one answers the practitioner's question: given a fixed budget, how should it be split? At this budget, the metrics favor fewer images, and the visual inspection suggests the metrics may be overstating that. Combining results from more than one design (Section 9) is how the "why" could be narrowed down.

5.2 Batch Size Doesn't Matter in the 1–4 Range

Batch size had no measurable effect on CLIP similarity (r = 0.00; no added variance explained), and only small correlations with the other metrics. This is the cleanest result in the study, because batch size varies while per-image exposure is held constant.

This is consistent with Lee & Lee (2026), who find LLM LoRA accuracy roughly flat across small batch sizes, degrading only above a critical batch size. Two cautions apply. Their experiments use LLMs, tuned learning rates, and datasets of 25K–200K examples, and they find the critical batch size shrinks as the dataset shrinks. So whether batch sizes 1–4 fall in the flat region for 25–400 images is an extrapolation, not something their results establish. This study also did not test large batches.

Hong & Choo (2025) tested batch sizes 1, 2, 4 and 8 for SD 1.5 LoRAs on architectural massing images and recommend 4. Their CLIP scores differed only slightly across batch sizes (0.208–0.212, computed on one hand-selected image per prompt), and the choice rested largely on visual judgment, so it does not conflict with the null result here. They list as a limitation that they "did not analyze in detail the changes in model performance according to combinations of hyperparameter settings for Batch Size, Epoch, and Repeats," and suggest varying batch size and repeats with the training budget fixed. That is close to the design used here (they propose fixing training steps; this study fixes total samples).

Practically, batch size mainly sets VRAM use. At a fixed learning rate, batch size 1 gave the same results as batch size 4.

5.3 Metrics Can Reward the Wrong Thing

The run that led on CLIP and LPIPS similarity looked worse than a run it outscored. CLIP similarity responded strongly to the dataset-size/repetition trade-off while perceptual distance barely moved.

If practitioners choose settings by CLIP score, they will tend to pick small, heavily repeated datasets. That is a version of Goodhart's law: a measure that becomes the target stops tracking what it was meant to measure. This study shows one instance; measuring how often it happens needs structured human evaluation.

5.4 Related Work on Repeated Data

Whether a repeated example is worth as much as a fresh one has been studied more directly elsewhere:

• Muennighoff et al. (2023) found that for language model pretraining at fixed compute, repeating data for up to about 4 epochs is nearly as good as using new data, after which the value of repetition drops quickly.
• Hernandez et al. (2022) found that repeating a small fraction of training data many times, at fixed total training, sharply degrades language model performance, and linked this to memorization.
• Gu et al. (2025) found that memorization in diffusion models rises as the training set shrinks and as training runs longer.

These are pretraining studies (or, for Gu et al., training from scratch), not LoRA fine-tuning. I'm not aware of a controlled study of this trade-off for LoRA fine-tuning of SDXL-class models.

5.5 Limitations

• Dataset size and epochs are perfectly confounded by design (Section 5.1).
• One subject: results may not generalize to realistic faces, objects, concepts or styles.
• One run per configuration: no seed replicates. Within-group spread (SD ≈ 0.005 on CLIP) gives a rough sense of run-to-run noise, but it also includes batch-size differences.
• Small sample: 15 runs at only 5 distinct dataset sizes. The permutation test treats runs as independent, which is approximate.
• Fixed learning rate across batch sizes, and only batch sizes 1–4.
• CLIP similarity is whole-image similarity, not character identity; it also rewards matching outfit, pose and style.
• Visual comparison is qualitative and unblinded, and covers one pair of runs.
• FID is not meaningful at 20 references and 40 generations.
• No memorization measure: the overfitting proxy saturated, and the diversity metric is ambiguous.
• One training budget: the trade-off may look different at other budgets.
• Training data is fan art, which limits redistribution.

---

6. Practical Takeaways

1. Don't compare CLIP scores across dataset sizes at the same step budget and call the higher one better. At a fixed budget, smaller datasets score higher partly because of the setup itself.
2. Look at the outputs. In this study, the model that led on the similarity metrics looked worse than a lower-ranked one.
3. Batch size 1–4 doesn't matter for these metrics at a fixed learning rate. Use what fits in VRAM.

---

7. Reproducibility

• GitHub repo: lora-epoch-confound-study

  Contains:

  • analysis notebook and statistical evaluation
  • prompts and generation settings
  • aggregated results (results.csv)
  • experiment notebooks and workflow details

  Training and reference images are not redistributed because of copyright.

---

8. Acknowledgments

This study uses the Kohya sd-scripts training framework via CivitAI's Dreambooth/LoRA Trainer wrapper, and the Illustrious SDXL checkpoint.

---

9. Hypotheses and How to Test Them

These are my working hypotheses. They fit the results above but are not established by them. If you test one, I'd like to hear how it goes.

1. Small datasets score higher because they overtrain, not because they learn better. Extra repetitions push the model to reproduce surface features (hair, outfit, colors) that CLIP rewards, while fine detail gets worse. Test: Train one LoRA on a fixed dataset and evaluate checkpoints saved every few epochs. If CLIP similarity rises while images degrade, with the dataset unchanged, repetition alone is doing it. (With a cosine schedule, a mid-run checkpoint isn't identical to a shorter run, so treat this as a first look.)

2. The main advantage of a large dataset is that it can be trained longer before overfitting. A larger dataset may not teach the concept better per se; it may just take more steps before any single image has been repeated too often. Test: Train a small and a large dataset to the same epoch count, then keep going. Compare when each starts to degrade.

3. Repetitions per image (epochs) is the setting that matters most. If hypotheses 1 and 2 hold, the useful rule is "don't exceed roughly N repetitions per image," not "use X images." Test: Train different dataset sizes to the same epoch count. If they look about equally good, epochs is the main lever.

4. CLIP similarity can't be trusted to pick the best LoRA. In the one comparison here, the best-scoring model looked worse. Test: Run a blinded forced-choice comparison across all 15 runs (random pairs, labels hidden, order shuffled) and check how often human preferences agree with the CLIP ranking.

5. Rising CLIP with flat LPIPS signals overtraining. If true, this asymmetry could be a cheap diagnostic. Test: Check whether runs or checkpoints with this pattern are the ones people rate worse in hypothesis 4's test.

Other useful extensions: seed replicates for a few configurations; larger batch sizes with tuned learning rates; a continuous memorization measure (similarity to the nearest training image); and other domains and base models (style LoRAs, realistic subjects, SD 1.5, SD 3, Flux).

Issues and PRs are welcome on the GitHub repo.

---

References

• Gu, X., Du, C., Pang, T., Li, C., Lin, M. & Wang, Y. (2025). On Memorization in Diffusion Models. Transactions on Machine Learning Research. arXiv:2310.02664.
• Hernandez, D. et al. (2022). Scaling Laws and Interpretability of Learning from Repeated Data. arXiv:2205.10487.
• Hong, S.M. & Choo, S. (2025). Systematic Parameter Optimization for LoRA-Based Architectural Massing Generation Using Diffusion Models. Buildings, 15(19), 3477.
• Hu, E.J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
• Lee, S. & Lee, J. (2026). Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA. arXiv:2602.09492.
• Muennighoff, N. et al. (2023). Scaling Data-Constrained Language Models. NeurIPS 2023. arXiv:2305.16264.
• Parmar, G., Zhang, R. & Zhu, J.-Y. (2022). On Aliased Resizing and Surprising Subtleties in GAN Evaluation. CVPR 2022.
• Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.
• Zhang, R., Isola, P., Efros, A.A., Shechtman, E. & Wang, O. (2018). The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR 2018.