FlowerTune-Echo-DSRN-114M-Finance-PEFT

Model Card for ethicalabs/FlowerTune-Echo-DSRN-114M-Finance-PEFT

[!WARNING] This repository contains experimental models designed strictly for academic evaluation and research purposes.

Critical Constraints:

• No Production Deployment: Experimental models must not be deployed in commercial, enterprise, or mission-critical environments under any circumstances.
• No Liability: Experimental models are provided "as-is" without warranties of any kind. The developers assume zero liability for downstream consequences, system integration failures, or regulatory non-compliance resulting from unauthorized deployment.

The Echo-DSRN(N) (Dual State Recurrent Neural Network, short name: Echo-DSRN, also know as echo) is a novel architecture specifically designed to be a viable alternative for low-resource tasks ("smol" tasks) that are currently being inefficiently handled by the excessive scale of Large Language Models (LLMs) 🌱

This LoRA adapter has been fine-tuned by using the Flower Framework and TRL in a simulated federated learning scenario.

Model Usage

You can load the fine-tuned PEFT adapter over the base model and run sentiment analysis inference as follows:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the base model and tokenizer
base_model_name = "ethicalabs/Echo-DSRN-114M-v0.1.2"
base_model = AutoModelForCausalLM.from_pretrained(base_model_name, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)

# Load the fine-tuned PEFT adapter
peft_model_name = "ethicalabs/FlowerTune-Echo-DSRN-114M-Finance-PEFT"
model = PeftModel.from_pretrained(base_model, peft_model_name, trust_remote_code=True)

# Prepare the prompt for sentiment analysis
instruction = "What is the sentiment of this news? Please choose an answer from {negative/neutral/positive}."
news_text = "The company reported a 20% increase in revenue for the third quarter, driven by strong sales in the tech division."

messages = [
    {"role": "system", "content": "You are a helpful financial assistant."},
    {"role": "user", "content": f"{instruction} {news_text}"}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=10)
response = tokenizer.decode(outputs[0], skip_special_tokens=False)

print(response.split("<|assistant|>")[1].replace("<|end|>", "").strip() if "<|assistant|>" in response else response)

Training

Trained on a single AMD Radeon™ AI PRO R9700 (32 GB RAM).

You can download the source code and reproduce the training process by simply running the following commands:

# Downloads the flower app from Flower Hub
flwr new @mrs83/echo-dsrn-114m-finance

# Federation Local Setup
flwr federation simulation-config \
    --num-supernodes=10 \
    --client-resources-num-cpus=6 \
    --client-resources-num-gpus=1.0

# Runs the federation and streams training log output.
flwr run --stream

Source code is available on the Flower Hub

🏗️ Architecture Details

📊 Parameter Breakdown

🧩 Internal Block Structure (Per Layer)

Evaluation

To evaluate the best federated checkpoint (e.g., round 38) across the financial benchmarks (FPB, FIQA, TFNS), use the integrated evaluation script:

python echo_finance/eval.py \
    --base-model-name-path ethicalabs/Echo-DSRN-114M-v0.1.2 \
    --peft-path ~/.ethicalabs/flwr/results/[RUN_TIMESTAMP]/peft_38 \
    --datasets fpb,fiqa,tfns \
    --apply-chat-template

Note: The --apply-chat-template flag ensures the model is evaluated accurately using the structural ChatML format it was aligned on.

Benchmarks

Citations

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}

@misc{gao2025flowertune,
  title = {FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models},
  author = {Yan Gao and Massimo Roberto Scamarcia and Javier Fernandez-Marques and Mohammad Naseri and Chong Shen Ng and Dimitris Stripelis and Zexi Li and Tao Shen and Jiamu Bai and Daoyuan Chen and Zikai Zhang and Rui Hu and InSeo Song and Lee KangYoon and Hong Jia and Ting Dang and Junyan Wang and Zheyuan Liu and Daniel Janes Beutel and Lingjuan Lyu and Nicholas D. Lane},
  year = {2025},
  eprint = {2506.02961},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2506.02961}
}

@misc{Massimo Roberto Scamarcia, title={Echo-DSRN-114M: Surprise-Gated Dual-State Recurrent Architecture for Efficient Language Modeling and Classification}, DOI={10.5281/zenodo.19848279}, publisher={Zenodo}, author={Massimo Roberto Scamarcia} }

Framework versions

• TRL: 1.1.0
• Transformers: 5.2.0
• Pytorch: 2.10.0+rocm7.1
• Datasets: 4.8.4
• Tokenizers: 0.22.2