Spexcon-S1

Spexcon S1

Spexcon S1 is the first language model project in the Spexcon family. This repository contains the v0.4 open-source development scaffold for building, fine-tuning, smoke testing, and evaluating Spexcon S1. It does not contain final Spexcon S1 weights yet.

The project is designed to support future Spexcon variants such as Spexcon S1 Mini, Spexcon S1 Code, Spexcon S1 Reason, and later Spexcon S2 models by keeping model choice, dataset formatting, LoRA settings, training arguments, and Hub upload settings in configuration.

Development Status

Status: v0.4 pilot training completed successfully on Kaggle. The v0.4 adapter is an experimental 2,000-example pilot adapter, not a finished Spexcon S1 release.

This release includes:

• A professional Hugging Face model card.
• A configurable supervised fine-tuning pipeline.
• A streaming dataset preparation script for a small instruction/chat subset.
• LoRA/QLoRA settings suitable for limited/free GPU environments.
• A fixed-prompt qualitative evaluation script.
• An example chat/inference script.
• A v0.3 smoke-test runner that records measured hardware, loss, adapter reload, and fixed-prompt outputs when run on CUDA.
• A verified experimental v0.3 LoRA adapter published separately at YONKWd/Spexcon-S1-v0.3-Experimental.
• A v0.4 pilot config, stricter English-focused dataset cleaner, Spexcon identity seed set, quality checks, and Kaggle-ready v0.4 runner.
• No full training run and no final trained model weights.

Base Model

Spexcon S1 is configured to start from Qwen/Qwen3-1.7B-Base.

Why this base model:

• It is a Qwen-family open-weight base model.
• It has approximately 1.72B parameters, which is realistic for LoRA/QLoRA fine-tuning on free or limited GPUs.
• Its Hugging Face metadata lists the license as Apache-2.0, which is compatible with open and commercial use.
• It provides a cleaner foundation for Spexcon-specific instruction tuning than starting from a much larger checkpoint.

Important distinction:

• Base model: Qwen/Qwen3-1.7B-Base.
• Experimental adapter: YONKWd/Spexcon-S1-v0.3-Experimental, a clearly labeled 300-example smoke-test adapter.
• Pilot adapter: YONKWd/Spexcon-S1-v0.4-Pilot, a clearly labeled experimental 2,000-example pilot adapter.
• Full Spexcon S1 release: does not exist yet.

v0.3 Smoke Test Result

The first real GPU QLoRA smoke test completed on Kaggle using one Tesla T4 from a T4 x2 session. The run used 300 UltraChat training examples and 50 evaluation examples, sequence length 512, LoRA rank 16, LoRA alpha 32, and QLoRA 4-bit loading.

Measured result summary:

• Training completed: yes, 10 optimizer steps.
• Peak GPU memory allocated: 3.32 GB.
• Recorded training loss: 1.796299934387207.
• Adapter reload: succeeded.
• Fixed-prompt qualitative evaluation: completed for the base model and the experimental adapter.

The qualitative outputs show that the pipeline works, but the adapter is not a quality release. Some generations still repeat role text or switch language, which should be addressed with better formatting checks, cleaner datasets, and a larger v0.4 training run.

v0.4 Pilot Plan

The v0.4 pilot is designed to reduce the rough behavior found in v0.3: unexpected non-English switching, role/template leakage, repeated phrases, strange suffixes, and inconsistent Spexcon identity.

Prepared dataset status:

• Source dataset: HuggingFaceH4/ultrachat_200k
• Dataset license: MIT
• Prepared rows: 2,000 train and 200 held-out eval
• Spexcon identity seed rows: 28
• Raw source rows inspected: 4,068
• Source rows filtered by v0.4 quality checks: 1,896
• Local quality scan: no role-leak, suspicious-artifact, repetition, or rejected-script hits in the prepared train/eval JSONL

The generated JSONL files live under data/processed/ and are intentionally ignored by git. The inspection summary is tracked at evaluation/reports/v0.4-dataset-inspection.md.

v0.4 Pilot Result

The v0.4 pilot QLoRA run completed on Kaggle using the cleaned 2,000-example training subset and 200-example held-out evaluation split. The adapter was saved separately at YONKWd/Spexcon-S1-v0.4-Pilot.

Measured run summary:

• Train examples: 2000
• Eval examples: 200
• Initial training loss: 2.3972280025482178
• Final training loss: 1.3996604919433593
• Runtime seconds: 1954.34
• Peak memory allocated GB: 3.53
• Peak memory reserved GB: 4.01
• Evaluation report: evaluation/reports/v0.4-pilot.md

This result is still experimental. It should be compared qualitatively with v0.3 before any Modal deployment or larger v0.4/v0.5 run.

v0.5 Candidate Baseline Evaluation

Status: evaluation-only baseline completed. No v0.5 training has been started, no v0.5 adapter has been created, and Modal should remain on the v0.3 experimental adapter until a later evaluation gate passes.

Compared with the same six fixed prompts in evaluation/prompts.jsonl:

• Qwen/Qwen3-1.7B-Base
• Qwen/Qwen3-1.7B
• YONKWd/Spexcon-S1-v0.3-Experimental on Qwen/Qwen3-1.7B-Base
• YONKWd/Spexcon-S1-v0.4-Pilot on Qwen/Qwen3-1.7B-Base

Gate result:

• Candidate base: Qwen/Qwen3-1.7B
• Candidate substantially cleaner than base: True
• Base outputs with any objective issue: 5
• Candidate outputs with any objective issue: 0
• Base issue score: 9
• Candidate issue score: 0
• Gate rule: Pass if Qwen3-1.7B has at least 2 fewer outputs with any objective issue, or at least 3 fewer total objective issue flags, than Qwen3-1.7B-Base.
• Full report: evaluation/reports/v0.5-baseline-comparison.md

If the gate is true, training/config_v05.yaml is a prepared pilot config only. Do not run v0.5 training without explicit approval.

v0.5 Pilot Training Result

Status: completed. The v0.5 pilot uses Qwen/Qwen3-1.7B, not Qwen/Qwen3-1.7B-Base, and it does not modify or overwrite the v0.3 or v0.4 adapters.

• Adapter repo: YONKWd/Spexcon-S1-v0.5-Pilot
• Report: evaluation/reports/v0.5-pilot.md
• Training examples: 1000
• Eval examples: 100
• Initial loss: 2.49219012260437
• Final loss: 1.230861282348633
• Eval loss: 1.2230945825576782
• Good enough to replace v0.3 on Modal: True

Modal remains on the v0.3 experimental adapter unless a later deployment step explicitly changes it.

v0.6 Pilot Result

Status: completed. This corrective adapter is separate from v0.5 and has not been deployed automatically.

• Adapter: YONKWd/Spexcon-S1-v0.6-Pilot
• Base: Qwen/Qwen3-1.7B
• Report: evaluation/reports/v0.6-pilot.md
• Training examples: 1000
• Eval examples: 100
• Initial/final loss: 2.312973737716675 / 1.368770217895508
• Eval loss: 1.2137300968170166
• Strict quality gate passed: False
• Recommended to replace live v0.5: False

The public Space continues to use v0.5 unless a separate deployment is explicitly approved.

Intended Use

Spexcon S1 is intended for research, experimentation, and future instruction tuning of compact assistant-style language models. Potential target behaviors include:

• General instruction following.
• Writing and editing assistance.
• Reasoning-style responses after suitable supervised fine-tuning.
• Domain-specialized Spexcon variants trained on carefully licensed datasets.

This repository is intended for development and reproducibility. It is not a finished assistant model.

Limitations

• No full Spexcon S1 training has been run yet.
• The repository does not currently include final model weights.
• The v0.3 adapter is only a smoke-test artifact and should be treated as experimental.
• The configured base model is not a safety-certified system.
• Outputs may be inaccurate, biased, incomplete, or unsafe unless future training and evaluation address those risks.
• The v0.3 qualitative evaluation found rough outputs, including repetition and occasional language switching.
• Downstream users must validate dataset licenses, data quality, privacy constraints, and evaluation results before releasing trained checkpoints.

Training Method

The included training pipeline uses:

• Python
• Hugging Face Transformers
• Hugging Face Datasets
• PEFT
• TRL SFTTrainer
• LoRA or QLoRA
• 4-bit bitsandbytes loading for memory-efficient training

The default method is supervised fine-tuning with QLoRA. The script saves PEFT adapter checkpoints, which can later be merged into the base model if desired.

Dataset Pipeline

The dataset pipeline uses HuggingFaceH4/ultrachat_200k as the first source dataset.

Dataset license: MIT, as listed on the Hugging Face dataset card.

Why this dataset:

• It is English instruction/chat data.
• It includes SFT-ready messages records.
• It has documented train_sft and test_sft splits.
• It is widely used for assistant-style supervised fine-tuning experiments.
• The preparation script streams a small subset, so the full dataset is not committed or downloaded unnecessarily.

The preparation pipeline adds:

• Empty/broken example filtering.
• Role normalization.
• Basic duplicate removal.
• Deterministic shuffling and train/eval split.
• Configurable sample limit and seed.
• Dataset statistics in stats.json.

The v0.4 pipeline adds stricter quality filters:

• Single user/assistant turn extraction.
• English-focused ASCII/script checks.
• Length caps for user, assistant, and total conversation text.
• Rejection of role-label leakage, suspicious debug artifacts, repeated lines, and repeated token runs.
• A small hand-authored Spexcon identity seed set at data/seed/spexcon_identity_v04.jsonl.

Dataset Format

The trainer accepts several common supervised fine-tuning formats:

• messages: a chat-style list of role/content dictionaries.
• text: a fully formatted training string.
• prompt plus response, completion, or answer.
• instruction plus output or response.

For production Spexcon S1 training, prefer a curated dataset with:

• Clear system/user/assistant message turns.
• Permissive licensing.
• Deduplicated examples.
• No private or sensitive data.
• Separate train/validation/test splits.
• Evaluation prompts that are held out from training.

Quick Start

Install dependencies in a fresh GPU notebook or virtual environment:

pip install -r training/requirements.txt

Prepare the v0.2 small dataset subset:

python training/prepare_dataset.py --config training/config.yaml

Validate configuration and prepared dataset loading without training:

python training/train.py --config training/config.yaml --dry-run

Run a tiny QLoRA smoke test with the training entry point:

python training/train.py --config training/config.yaml --smoke-test

Or prepare a tiny 500-example smoke subset and run the smoke test in one command:

python training/train.py --config training/config.yaml --prepare-dataset --smoke-test

Run the full v0.3 smoke-test verification script on a CUDA GPU:

python training/run_v03_smoke.py --config training/config.yaml --push-adapter

Current v0.3 status: the Kaggle T4 QLoRA smoke test completed, the adapter reloaded successfully, fixed-prompt evaluation ran, and the experimental adapter was uploaded to YONKWd/Spexcon-S1-v0.3-Experimental. See evaluation/reports/v0.3-smoke-test.md.

Run an interactive chat against the base model:

python examples/chat.py --model Qwen/Qwen3-1.7B-Base

After a future experimental LoRA adapter has been trained and clearly labeled:

python examples/chat.py --model Qwen/Qwen3-1.7B-Base --adapter path-or-repo-to-adapter

To try the v0.3 experimental adapter explicitly:

python examples/chat.py --model Qwen/Qwen3-1.7B-Base --adapter YONKWd/Spexcon-S1-v0.3-Experimental

Run the fixed-prompt comparison evaluator:

python evaluation/evaluate.py --config training/config.yaml --adapter path-or-repo-to-adapter

Prepare and inspect the v0.4 pilot dataset before GPU training:

python training/run_v04_pilot.py --config training/config_v04.yaml --prepare-only

Run only the v0.4 CUDA preflight on Kaggle after the dataset looks clean:

python training/run_v04_pilot.py --config training/config_v04.yaml --skip-prepare --preflight-only

Run the v0.4 pilot training on one Kaggle Tesla T4 only after explicitly choosing to start training:

python training/run_v04_pilot.py --config training/config_v04.yaml --skip-prepare --push-adapter

Using Spexcon S1 in an App

The server/ module exposes the v0.3 experimental adapter through a small FastAPI backend with an OpenAI-compatible /v1/chat/completions endpoint. This keeps inference separate from frontends, so web, desktop, and mobile apps can all call the same Spexcon API.

For app-development testing, the server applies conservative default stop markers to trim known v0.3 smoke-test artifacts such as debug fragments and repeated chat role labels. This is a temporary inference guardrail, not a substitute for higher-quality v0.4 training.

Install the server dependencies:

pip install -r server/requirements.txt

Start the local API:

uvicorn server.app:app --host 0.0.0.0 --port 8000

By default, the server serves:

• Base model: Qwen/Qwen3-1.7B-Base
• Adapter: YONKWd/Spexcon-S1-v0.3-Experimental
• Model alias: spexcon-s1

The model loads lazily on the first chat request. On CUDA, SPEXCON_QUANTIZATION=auto uses 4-bit bitsandbytes loading to reduce VRAM use. Hugging Face tokens are read only from the server environment, such as HF_TOKEN, and are never sent to client apps.

Health and model list:

curl http://localhost:8000/health
curl http://localhost:8000/v1/models

JavaScript app example using fetch():

const response = await fetch("http://localhost:8000/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "spexcon-s1",
    messages: [{ role: "user", content: "Hello Spexcon" }],
    temperature: 0.7,
    max_tokens: 128,
  }),
});

const data = await response.json();
console.log(data.choices[0].message.content);

OpenAI JavaScript SDK example:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.SPEXCON_API_KEY ?? "local-dev-key",
  baseURL: "http://localhost:8000/v1",
});

const completion = await client.chat.completions.create({
  model: "spexcon-s1",
  messages: [{ role: "user", content: "Hello Spexcon" }],
});

console.log(completion.choices[0].message.content);

For hosted deployment, run this backend on a GPU-capable container or VM, set SPEXCON_API_KEY for client authentication, keep HF_TOKEN server-side only if needed, and put any web/mobile frontend in a separate app that calls the API. This repository does not configure paid hosting automatically.

Modal Deployment

The deploy/modal_app.py module exposes the existing FastAPI app through Modal's ASGI integration. It uses the existing Spexcon model loader, requests a T4 GPU, mounts a persistent Hugging Face cache volume, and is configured to scale to zero when idle.

Create a Modal Secret named huggingface-secret with HF_TOKEN. The same secret can optionally include SPEXCON_API_KEY and SPEXCON_CORS_ORIGINS for private app testing:

modal secret create huggingface-secret \
  HF_TOKEN="$HF_TOKEN" \
  SPEXCON_CORS_ORIGINS="*"

For production, replace * with a comma-separated allowlist such as https://spexcon.ai,https://www.spexcon.ai.

Deploy the public API:

modal deploy -m deploy.modal_app

Use the actual HTTPS URL returned by Modal as the frontend API base:

VITE_SPEXCON_API_URL=https://your-actual-modal-url.modal.run

Do not place HF_TOKEN or SPEXCON_API_KEY in a public frontend build. For details, see deploy/README.md.

Recommended Hardware

For initial Spexcon S1 QLoRA experiments:

• Minimum practical: NVIDIA T4 16 GB with short sequence length and batch size 1.
• Recommended free/low-cost: L4 24 GB or A10G 24 GB.
• Expected VRAM: roughly 8-12 GB for conservative QLoRA settings at 1024-token sequences; 12-18 GB is more comfortable for longer contexts, evaluation, or larger batches.

License

This project scaffold is released under Apache License 2.0. The selected base model, Qwen/Qwen3-1.7B-Base, is listed on Hugging Face with the Apache-2.0 license.

Future trained Spexcon checkpoints should only be released after confirming that the training datasets and any generated or synthetic data are compatible with Apache-2.0 distribution.

Citation and Attribution

Base model:

Qwen/Qwen3-1.7B-Base
https://huggingface.co/Qwen/Qwen3-1.7B-Base
License: Apache-2.0