term-complete-qwen-0.5b

term-complete-qwen-0.5b

This repository contains a fine-tuned version of Qwen2.5-Coder-0.5B for terminal command auto-completion. The model is trained to suggest shell commands based on the current operating system, current working directory contents, and command history.

Model Details

• Base Model: Qwen2.5-Coder-0.5B-Instruct
• Fine-tuning Method: LoRA (4-bit quantization)
• Purpose: Terminal command auto-completion

How to Use

To use this model for inference with the Hugging Face transformers library, you'll need to load the model and tokenizer, then format your input according to the training schema.

Installation

First, make sure you have the necessary libraries installed:

pip install transformers torch accelerate peft

Loading the Model and Tokenizer

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Replace 'your-username/term-complete-qwen-0.5b' with your actual repo ID
repo_id = "prappo/term-complete-qwen-0.5b" 

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map="auto")
model.eval() # Set model to evaluation mode

Inference

The model expects a specific prompt format that includes contextual information:

<|context|> os: [OS_NAME] cwd: [CURRENT_WORKING_DIRECTORY] files: [COMMA_SEPARATED_FILES] history: [PIPE_SEPARATED_HISTORY_COMMANDS] <|input|> [COMMAND_PREFIX] <|output|>

Example:

from transformers import AutoModelForCausalLM, AutoTokenizer
from unsloth import FastLanguageModel
import torch

# Replace 'your-username/term-complete-qwen-0.5b' with your actual repo ID
repo_id = "prappo/term-complete-qwen-0.5b"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
# Load the model directly using FastLanguageModel.from_pretrained for full Unsloth optimization
model, _ = FastLanguageModel.from_pretrained(
    model_name=repo_id,
    max_seq_length=512, # Assuming MAX_SEQ from earlier cells
    dtype=torch.bfloat16,
    load_in_4bit=True,
)

# FastLanguageModel.from_pretrained already applies inference optimizations
model.eval() # Set model to evaluation mode

def suggest(osname, cwd, files, prefix, history=""):
    prompt = (f"<|context|>\nos: {osname}\ncwd: {cwd}\nfiles: {', '.join(files)}\n"
              f"history: {history or '(empty)'}\n<|input|>\n{prefix}\n<|output|>\n")

    inputs = tokenizer(prompt, return_tensors="pt").to("cuda") # or "cpu" if no GPU
    outputs = model.generate(**inputs, max_new_tokens=24, do_sample=False,
                             temperature=None, top_p=None, top_k=None,
                             pad_token_id=tokenizer.eos_token_id)
    text = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:],
                            skip_special_tokens=True)
    return text.split("\n")[0].strip()

# Example Usage
os_name = "linux"
cwd = "/home/user/my-project"
current_files = ["main.py", "requirements.txt", "README.md", ".git/"]
history_commands = "git status | ls -l"
command_prefix = "git co"

suggested_command = suggest(os_name, cwd, current_files, command_prefix, history_commands)
print(f"Suggested command: {suggested_command}")