nepali-lyrics-bloomz-3b-lora

BLOOMZ-3B Nepali Lyrics LoRA

A LoRA adapter for bigscience/bloomz-3b, fine-tuned for Nepali song lyric generation in Devanagari script.

This model was developed during an undergraduate final-year project investigating end-to-end AI-based Nepali folk music generation.

It represents an earlier lyric-generation stage of the project. A later Gemma-3-4B-based LoRA was subsequently selected as the final lyric-generation model.

Note: This repository contains the LoRA adapter only. The bigscience/bloomz-3b base model is required separately.

Quick Overview

---

Project Context

This model was developed while investigating suitable language models for Nepali lyric generation.

Earlier project experiments showed that model choice had a significant effect on Nepali text quality.

The BLOOMZ model was explored because of its multilingual capabilities and produced more usable Nepali lyric outputs than an earlier LLaMA-based experiment used during development.

The project later moved to:

pradipbasnet68/gemma-3-4b-nepali-lyrics-lora

which became the final lyric-generation model.

The development path was therefore:

Earlier LLaMA Experiment
        ↓
BLOOMZ-3B + Nepali Lyrics LoRA
        ↓
Gemma-3-4B + Nepali Lyrics LoRA
        ↓
Final Lyric Generator

Keeping this BLOOMZ adapter available documents the model-selection and experimentation process that led to the final system.

---

Intended Use

This adapter is intended for:

• experimental Nepali lyric generation
• academic research
• Devanagari-based creative text generation
• studying LoRA adaptation for Nepali language generation
• comparison with alternative lyric-generation models
• generating lyric drafts for downstream music-generation experiments

The model was trained to produce Nepali lyric-style text rather than general-purpose conversational responses.

---

Out-of-Scope Use

This model is not intended for:

• factual question answering
• general-purpose conversation
• professional translation
• medical, legal, or financial advice
• safety-critical applications
• authoritative linguistic judgement
• guaranteed publication-quality poetry

The adapter should be treated as an experimental creative-generation model.

---

Training Data

The adapter was trained on a curated dataset of Nepali song lyrics prepared for the associated final-year project.

The dataset contained Nepali lyric examples used to teach the model:

• Nepali lyric structure
• Devanagari-script generation
• common song-like phrasing
• emotional and folk-oriented themes
• more consistent Nepali textual output

The project dataset was prepared specifically for academic experimentation with Nepali lyric generation.

---

Training Method

The model was fine-tuned using LoRA — Low-Rank Adaptation.

Instead of updating all parameters of the approximately 3-billion-parameter BLOOMZ base model, small trainable LoRA matrices were attached to selected transformer modules.

This reduces the number of trainable parameters and GPU-memory requirements while preserving the original base model.

---

LoRA Configuration

Target Modules

LoRA adapters were applied to the following BLOOM transformer modules:

dense_h_to_4h
dense
dense_4h_to_h
query_key_value

Only the LoRA adapter parameters are distributed in this repository.

---

Why BLOOMZ Was Selected for Experimentation

BLOOMZ is a multilingual instruction-tuned language model based on BLOOM.

During the project, it was investigated as an alternative to an earlier LLaMA-based lyric-generation experiment.

In qualitative project testing, the fine-tuned BLOOMZ adapter generated more usable Nepali text and reduced some of the unrelated or random-word behaviour observed with the earlier experimental model.

This made BLOOMZ an important intermediate stage in the project.

---

Evaluation

The BLOOMZ adapter was primarily evaluated through prompt-based qualitative generation testing during the development phase.

The project examined characteristics such as:

• whether generated text remained in Nepali
• Devanagari-script consistency
• unwanted foreign words
• repetition
• lyric-like structure
• semantic coherence
• suitability for later music-generation experiments

The fine-tuned BLOOMZ model produced usable Nepali lyric outputs and demonstrated that parameter-efficient fine-tuning could meaningfully adapt a multilingual model toward this task.

However, the project later adopted a more systematic quantitative evaluation framework for the final Gemma-based lyric model.

Therefore, this repository does not claim a standardized quantitative benchmark for the BLOOMZ adapter.

---

Relationship to the Final Gemma Model

The final lyric generator used in the completed project is:

pradipbasnet68/gemma-3-4b-nepali-lyrics-lora

The Gemma model was later evaluated using:

• test loss
• perplexity
• token-level accuracy
• script consistency
• Latin-character violations
• repetition diagnostics
• prompt-based generation testing

The BLOOMZ adapter is retained as evidence of the earlier model-development and comparison process.

---

Install Dependencies

pip install transformers peft accelerate torch

---

Load the Model

import torch

from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
)

from peft import PeftModel


BASE_MODEL = (
    "bigscience/bloomz-3b"
)

ADAPTER_MODEL = (
    "pradipbasnet68/"
    "nepali-lyrics-bloomz-3b-lora"
)



tokenizer = AutoTokenizer.from_pretrained(
    BASE_MODEL
)



base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16,
    device_map="auto",
)



model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_MODEL,
)


model.eval()


print(
    "BLOOMZ Nepali Lyrics LoRA loaded."
)

---

Example Lyric Generation

prompt = """
नेपाली लोकगीत लेख्नुहोस्।

विषय:
आफ्नो गाउँबाट टाढा बसेको मानिसले
पुरानो गाउँ र बाल्यकाल सम्झिरहेको छ।

गीत नेपाली देवनागरी लिपिमा लेख्नुहोस्।
""".strip()


inputs = tokenizer(
    prompt,
    return_tensors="pt",
)


inputs = {
    key: value.to(model.device)
    for key, value in inputs.items()
}


input_length = inputs[
    "input_ids"
].shape[-1]


with torch.inference_mode():

    output = model.generate(
        **inputs,
        max_new_tokens=250,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.15,
    )


generated_tokens = output[0][
    input_length:
]


lyrics = tokenizer.decode(
    generated_tokens,
    skip_special_tokens=True,
).strip()


print(lyrics)

---

Example Use Case

A typical use case is:

User Theme
     ↓
BLOOMZ-3B
+ Nepali Lyrics LoRA
     ↓
Generated Nepali Lyrics

During the earlier project stage, generated lyrics could then be passed to a separate music-generation model.

The final project later replaced this lyric stage with the Gemma-based adapter.

---

Final Project Evolution

The project progressed through several lyric-generation experiments:

Initial Lyric Model Experiments
          ↓
LLaMA-based Model
          ↓
BLOOMZ-3B
+ Nepali Lyrics LoRA
          ↓
Further Dataset Cleaning
and Model Evaluation
          ↓
Gemma-3-4B-IT
+ Nepali Lyrics LoRA
          ↓
Final Lyric Generator
          ↓
ACE-Step 1.5 Turbo
+ Nepali Folk LoRA
          ↓
Generated Nepali Folk Song

This repository therefore represents an important experimental stage rather than the final production lyric model.

---

Current Final Project Models

Final Lyric Generator

pradipbasnet68/gemma-3-4b-nepali-lyrics-lora

Final Music Generator Adapter

pradipbasnet68/nepali-folk-acestep-1.5-lora

Earlier Experimental Lyric Model

pradipbasnet68/nepali-lyrics-bloomz-3b-lora

---

Limitations

The model may generate:

• repeated lyric lines
• grammatically awkward phrases
• semantically inconsistent lyrics
• mixed-language content
• unrelated words
• weak narrative continuity
• unnatural poetic structure
• incomplete song sections

Generation quality can vary significantly depending on:

• prompt wording
• sampling temperature
• generation length
• random seed
• decoding parameters

The model has not been evaluated as a general-purpose Nepali language model.

---

Why This Model Is Still Published

Although BLOOMZ was not selected as the final lyric model, publishing this adapter provides useful evidence of:

• iterative model development
• experimentation with different base models
• parameter-efficient fine-tuning
• Nepali-language adaptation
• model-selection decisions
• progression toward the final system

This repository should therefore be interpreted as an experimental research artifact from the development process.

---

Responsible Use

Generated lyrics should be manually reviewed before publication or reuse.

The model may reproduce patterns or biases present in the training material.

It should not be assumed to represent all Nepali dialects, communities, musical traditions, or styles.

Users are responsible for complying with the upstream BLOOMZ license and any applicable rights associated with prompts, training data, and generated material.

---

Base Model

This adapter requires:

bigscience/bloomz-3b

The base model is not included in this repository.

Users must separately download the base model and comply with the BigScience BLOOM RAIL 1.0 license.

---

Adapter Repository

pradipbasnet68/nepali-lyrics-bloomz-3b-lora

---

Author

Pradip Raj Basnet

Developed as part of an undergraduate final-year project investigating AI-based Nepali folk music generation.

---

Disclaimer

This repository contains an experimental LoRA adapter developed for academic research.

It represents an earlier development stage of the project and is not the final lyric-generation model.

Performance may vary depending on prompts, hardware, software versions, and decoding configuration.