francis-botcon-lora

Francis Botcon - Fine-tuned Mistral 7B LoRA Model

Model Description

Francis Botcon is a fine-tuned version of Mistral-7B-Instruct-v0.2 optimized for answering questions about Francis Bacon and his works. The model uses LoRA (Low-Rank Adaptation) for efficient fine-tuning and inference.

Model Details

• Base Model: mistralai/Mistral-7B-Instruct-v0.2
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Training Data: 6 classical works by Francis Bacon
  • The Advancement of Learning
  • Valerius Terminus: Of the Interpretation of Nature
  • The New Atlantis
  • The Essays or Counsels, Civil and Moral
  • Novum Organum; Or, True Suggestions for the Interpretation of Nature
  • The Wisdom of the Ancients
• Total Training Examples: 1,633
• Training Duration: ~42 minutes on NVIDIA RTX 3090
• Number of Epochs: 3

LoRA Configuration

• Rank (r): 16
• Alpha (lora_alpha): 32
• Dropout: 0.05
• Target Modules: ["q_proj", "v_proj"]
• Trainable Parameters: 6.8M (0.094% of total parameters)

Training Metrics

• Initial Loss: 2.136
• Final Loss: ~1.20
• Token Accuracy: 62% → 76% (+14 points)
• Evaluation Loss: 1.2559
• Evaluation Accuracy: 73.95%

Intended Use

This model is designed for:

• Answering questions about Francis Bacon's philosophy, works, and ideas
• Providing contextual information from Bacon's original texts
• Educational purposes about Renaissance philosophy and the scientific method
• Integration with RAG (Retrieval-Augmented Generation) systems

System Requirements

• GPU with at least 16GB VRAM (for full model inference)
• 8GB RAM minimum
• CUDA 11.8+

Model Usage

Basic Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.2",
    device_map="auto",
    torch_dtype="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    model,
    "rojaldo/francis-botcon-lora"
)

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

# Generate response
prompt = "What did Francis Bacon believe about knowledge?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=512)
print(tokenizer.decode(outputs[0]))

With RAG System

The model is best used with a RAG system that retrieves relevant passages from Bacon's works:

from src.rag_system import RAGSystem

rag_system = RAGSystem(
    model=model,
    tokenizer=tokenizer,
    vector_db_path="./data/vectordb"
)

response = rag_system.query("What is the scientific method according to Bacon?")
print(response)

Limitations and Biases

• Limited Domain: The model is specialized for Bacon's works and may not perform well on other topics
• Historical Context: Represents 16th-17th century perspectives that may not align with modern scientific understanding
• Language: Primary training on English texts; multilingual capabilities may be limited
• Data Size: Relatively small training dataset compared to general-purpose models

Training Details

Data Processing

• Raw Documents: 6 classical texts (Project Gutenberg)
• Processing Method: Sentence segmentation with overlap
• Total Segments: 1,654
• Split: 90% training (1,469), 10% evaluation (164)

Hyperparameters

• Batch Size: 4
• Learning Rate: 2e-4
• Warmup Steps: 100
• Weight Decay: 0.01
• Max Sequence Length: 1024
• Optimizer: AdamW

Hardware

• GPU: NVIDIA GeForce RTX 3090
• Precision: bfloat16
• Quantization: 4-bit during training

Evaluation

The model was evaluated on a held-out test set:

Ethical Considerations

• The model generates historical perspectives that should be understood in their temporal context
• Not recommended for critical decisions without human review
• Educational use should include context about historical limitations
• Potential bias toward Western philosophical traditions

License

This model and its components are released under the MIT License.

Citation

If you use this model, please cite:

@model{francis_botcon_2025,
  title={Francis Botcon: Fine-tuned Mistral 7B for Francis Bacon Studies},
  author={Rojaldo},
  year={2025},
  publisher={Hugging Face}
}

Acknowledgments

• Base model by Mistral AI
• Training data from Project Gutenberg
• LoRA methodology from Microsoft Research
• Vector database implementation using Chroma DB

Related Resources

• Hugging Face Model Card
• Gradio Space Demo
• GitHub Repository
• Original Project

Contact

For questions or issues, please open an issue on the project repository or contact the maintainers.