SmolLM-135M-FakyPedia-EngHeb

SmolLM-135M-FakyPedia-EngHeb

Table of Contents

• Model Details
• Uses
• Risks, Limitations and Biases
• Training

Model Details

Base Model

This model extended the tokenizer of and is a fine-tuned of SmolLM-135M-Instruct

Model Description:

A bilingual (English and Hebrew) nonsense generation model which produces silly Wikipedia-like abstract text.

• Fine tuned by: Doron Adler
• Model Type: Text Generation
• Language(s): English, Hebrew
• License: apache-2.0 (as a derived work of SmolLM)

Uses

Input format

BOS-TOKEN followed by '\%' followed by the optional title for the fake "Wikipedia" article

Generation

pip install transformers

# pip install transformers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model_id = "Norod78/SmolLM-135M-FakyPedia-EngHeb"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token_id = tokenizer.eos_token_id
bos_token = tokenizer.bos_token
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
model.generation_config.pad_token_id = tokenizer.pad_token_id

torch.manual_seed(1234)

def generate_fakypedia(article_title: str):
    with torch.no_grad():
        result = ""
        string_to_tokenize= f"{bos_token}\\%{article_title}"
        input_ids = tokenizer( string_to_tokenize, return_tensors="pt").input_ids.to(device)
        sample_outputs = model.generate(input_ids, do_sample=True,repetition_penalty=1.05, top_k = 40, top_p = 0.950, temperature=0.80, max_length=192, num_return_sequences=3)
        #sample_outputs = model.generate(input_ids, do_sample=True,repetition_penalty=1.2, temperature=0.5, max_length=192, num_return_sequences=3)        
        if article_title == None or len(article_title) == 0:
            result += f"# Fakypedia results with random titles  \n"
            article_title = ""
        else:
            result += f"# Fakypedia results for \"{article_title}\"  \n"
        for i, sample_output in enumerate(sample_outputs):
            decoded_output = tokenizer.decode(sample_output, skip_special_tokens=True)
            decoded_output = decoded_output.replace(f"\%{article_title}", f"## {i+1}. {article_title}").replace("\%", " ").replace("\\n", "  \n")
            decoded_output = decoded_output.replace("##   \n", "\n")
            result += "{}\n".format(decoded_output)
        return result

generate_fakypedia("Hugging Face")

Generate with llama.cpp

Download SmolLM-135M-FakyPedia-EngHeb-BF16.gguf
Run:

llama-cli -m SmolLM-135M-FakyPedia-EngHeb-BF16.gguf -p "<|endoftext|>\\%Hugging Face"

Misuse and Out-of-scope Use

Risks, Limitations and Biases

CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.

This model is basically a joke and intended to generate silly and fake results.

Training

Training Data

English and Hebrew Wikipedia

Training Procedure

• A tokenizer with vocab size of 14,000 was trained
• The trained tokenizer was then merged at the end of the base model's tokenizer using this script so the original base model knowledge was retained as well as make it better fine-tunable upon Hebrew text
• Hebrew and English datasets were interleaved so each language had an identical amount of samples.
• Each example was processed in the following manner:

def add_prefix(example):
  example["text"] = ("\%" + example["title"] + "\%\n" + example["text"]).replace("\n", "\\n")
  return example