Phi-3-Mini-Alpaca-GPT4-LoRa-V1

Model Card for Model ID

Important Note THIS MODEL REALLY SUCKS, HALLUCINATES A LOT, INSTEAD TRY turtle170/Phi-3-Mini-OpenHermes-Magpie-V1 (Is much more intelligent, as it has been trained on a much higher LoRa and is trained on CoT datasets)

Phi-3-Mini-Alpaca-GPT4-LoRa-V1 is a general purpose AI created from Microsoft's Phi-3-mini-4k-instruct model, and trained on the Alpaca-GPT4 dataset, using the evaluation strategy Epochs.

Specific Model Training Parameters: Epochs: 1; Trained on: Alpaca-GPT4 dataset (52002 Examples); Training Accelerators: 2x NVIDIA Tesla 4 GPUs; Batch Size: 4; Learning Rate: 5e-5; Gradient Accumulaton: 4; Warmup Steps: 300; Eval Steps: 500; Elapsed Training Time: ~11.5 Hours; Total Steps: 5850; Lora r: 16; Lora Alpha: 32; Precision: bfloat16 Context Window: 4096 Tokens

Users of this adapter must adhere to the Micrsoft Phi-3 and OpenAI Terms of Use.

Model Description

Model Executive Summary: Phi-3-Mini-Alpaca-GPT4-LoRa-V1

1. Architectural Foundation Phi-3-Mini-Alpaca-GPT4-LoRa-V1 is a specialized Parameter-Efficient Fine-Tuning (PEFT) adapter derived from Microsoft’s Phi-3-Mini-4k-Instruct. The base architecture is a 3.8 billion parameter dense decoder-only Transformer. It is designed to bridge the gap between "Small Language Models" (SLMs) and the sophisticated reasoning capabilities of much larger systems like GPT-3.5 and Mistral-7B.

2. Training Methodology & Data Distillation The model utilizes LoRA (Low-Rank Adaptation), a technique that freezes the original 7GB of base model weights and trains a lightweight "adapter" layer (the 12MB file you unboxed).

Dataset: The primary training signal comes from Alpaca-GPT4 (52,002 samples). This dataset consists of high-quality instructions where the responses were generated by OpenAI's GPT-4.

The Goal: By using "distilled" data, the model learns to mimic the tone, structural logic, and instruction-following precision of GPT-4, effectively "distilling" the intelligence of a massive model into a 3.8B parameter frame.

1. Functional Capabilities This model is engineered for Instruction Adherence and Reasoning-Dense Tasks. Unlike the base model, which is a generalist, this version is specifically "tuned" to:

Maintain Persona: Adhere more strictly to complex system prompts and creative formatting.

Reduced Verbosity: Deliver concise, GPT-style responses rather than the often repetitive nature of small base models.

Enhanced Logic: Leverage the "thought chain" logic present in the GPT-4 training examples to solve multi-step problems more reliably.

• Developed by: Turtle170 (Anonymous nickname)

• Model type: Text Generation

• Language(s) (NLP): English

• License: apache-2.0

• Finetuned from model: microsoft/Phi-3-mini-4k-instruct

• Repository: https://github.com/turtle170/Phi-3-Mini-Alpaca-GPT4-LoRa-V1