Downloads · 30 days
22
20% of all-time downloads
Hariharan05/Qwen3.5-0.8B-512K-extended
Qwen3.5-0.8B-512K-extended is a machine learning model from Hariharan05. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
An experimental long-context extension of Qwen3.5-0.8B, extending the context window from approximately 256K tokens to 512K tokens using Rotary Positional Embeddings (RoPE) and YaRN-style positional scaling.
Downloads · 30 days
22
20% of all-time downloads
All-time downloads
112
Public
Repo size
834 MB
Likes
0
Public
Click a slice to open those files.
.gguf834 MB · 100%
From the Hugging Face model README
An experimental long-context extension of Qwen3.5-0.8B, extending the context window from approximately 256K tokens to 512K tokens using Rotary Positional Embeddings (RoPE) and YaRN-style positional scaling.
⚠️ Experimental: This model is intended for research and experimentation with very long context windows. While the model is configured for up to 512K tokens, long-context accuracy and performance may vary depending on the task, inference framework, hardware, quantization, and prompt structure.
Hariharan05/Qwen3.5-0.8B-512K-extended is an experimental context-window extension of the Qwen3.5-0.8B model.
The goal is to retain the small ~0.8B parameter model size while extending its usable context window:
| Base Model | Extended Model | |
|---|---|---|
| Parameters | ~0.8B | ~0.8B |
| Context Window | ~256K | 512K |
| Tokens | 262,144 | 524,288 |
| Positional Encoding | RoPE | RoPE + YaRN-style scaling |
| Model Size | ~0.8B | ~0.8B |
| Status | Base model | Experimental extension |
The extension focuses on positional encoding rather than increasing the number of model parameters.
The model uses Rotary Positional Embeddings (RoPE) with a YaRN-style context extension to support positions beyond the original ~256K context range.
The intended context capacity is:
262,144 tokens → 524,288 tokens
~256K ~512K
This makes the model suitable for experimenting with extremely long documents, large codebases, long transcripts, retrieval tasks, and other workloads where a conventional context window may be insufficient.
A 512K context window should be considered an experimental capability. Supporting a 512K-token input does not guarantee that information will be retrieved or reasoned over equally well at every position in the context.
Long-context inference requires substantially more memory than the raw model weights because the inference engine must maintain the KV cache for the active context.
The following memory breakdown is based on the source data/benchmark used for this model:
| Component | Precision / Quantization | Memory Required | Notes |
|---|---|---|---|
| Model Weights | INT8 / Q8 | ~0.9 GB | The model contains ~0.87B parameters. |
| KV Cache | Q8 | ~6.0 GB | Required to maintain a 512K-token context at 8-bit precision. |
| System Overhead | N/A | ~1.0 GB | Runtime allocation for engines such as llama.cpp or vLLM. |
| Total | Q8 Model + Q8 KV | ~7.9–8.5 GB | Approximate total memory requirement. |
For practical 512K-context experimentation, a GPU with at least 8 GB of available memory may be able to run the Q8 configuration under favorable conditions, but 12 GB or more is recommended to provide additional runtime headroom.
Actual memory usage can vary depending on:
Note: The ~7.9–8.5 GB figure is for the specified Q8 model + Q8 KV-cache configuration. Other precisions or inference configurations will have different memory requirements.
The model can be loaded using compatible Hugging Face Transformers or other inference frameworks that support the model architecture and extended RoPE configuration.
Example:
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "Hariharan05/Qwen3.5-0.8B-512K-extended"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype="auto",
device_map="auto",
)
prompt = "Your long-context prompt here."
inputs = tokenizer(
prompt,
return_tensors="pt"
).to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=512,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For very long prompts, ensure that the selected inference framework correctly recognizes the extended context configuration and has sufficient memory for the KV cache.
This model is primarily intended for:
This is an experimental context extension and should not be assumed to have the same quality at 512K as the base model has within its native context range.
Potential limitations include:
The model's 512K context capacity is therefore best treated as an experimental feature rather than a guarantee of uniform 512K-context understanding.
This model is based on Qwen3.5-0.8B from the Qwen team.
The base model provides the underlying architecture and language capabilities, while this repository provides the experimental context-window extension.
Please refer to the original Qwen3.5 model card for the base model's complete architecture, training information, licensing, and usage guidelines.
| Feature | Qwen3.5-0.8B | Qwen3.5-0.8B-512K-extended |
|---|---|---|
| Parameters | ~0.87B | ~0.87B |
| Context | ~256K | 512K |
| Maximum tokens | 262,144 | 524,288 |
| RoPE | ✓ | ✓ |
| YaRN-style extension | — | ✓ |
| Parameter increase | — | None intended |
| Long-context focus | Standard | Experimental 512K |
| Q8 memory | — | ~7.9–8.5 GB total |
This model builds upon the work of the Qwen team and research into efficient RoPE-based context extension, including YaRN (Yet another RoPE extensioN).
The purpose of this project is to explore how a relatively small language model can be adapted for substantially larger context windows while retaining its original parameter scale.
This is an experimental model.
The advertised 512K context window (524,288 tokens) represents the configured target context length. It should not be interpreted as a guarantee of perfect long-context reasoning or retrieval.
Memory figures are provided as reference values for the specified Q8 model + Q8 KV-cache configuration. Actual requirements may differ depending on the inference engine, hardware, runtime configuration, and workload.