distilgpt2-TFLITE

DistilGPT2 TFLite Model

This repository provides a TensorFlow Lite (TFLite) version of the DistilGPT2 language model for on-device inference. It enables running DistilGPT2 on mobile and embedded devices with reduced size and latency while preserving output quality.

---

Model Overview

DistilGPT2 is a distilled version of GPT-2 offering a smaller footprint and faster inference. This TFLite conversion reduces memory usage and enables offline NLP applications such as text generation, completion, and chatbots directly on edge devices.

• Base model: distilgpt2
• Original framework: PyTorch → TensorFlow → TFLite
• Quantization: Float32 (default), INT8 (optional)
• Model size: ~240 MB (Float32), ~120 MB (INT8 quantized)

---

Features

• Lightweight and optimized for on-device AI
• Supports offline text generation without internet connectivity
• Compatible with Android, iOS, and embedded systems via TFLite
• Optional full-integer (INT8) quantization for smaller footprint
• Example integrations for Python and Flutter

---

Use Cases

This model can be used in scenarios where cloud-based inference is not possible or desirable, such as:

• Mobile assistants – lightweight chatbots that run entirely offline
• Smart IoT devices – conversational AI for smart speakers or appliances
• Privacy-focused apps – local inference without sending data to servers
• Low-connectivity environments – edge devices in rural areas or airplanes
• Education apps – offline writing assistants, summarizers, or quiz generators

---

Performance Matrix

(Benchmarks tested on Snapdragon 870 and Intel i5 CPU; numbers may vary depending on device.)

---

Quick Start Guide

1. Installation

Make sure you have TensorFlow Lite runtime installed:

pip install tflite-runtime tensorFlow 

2. Load and Run Inference (Python)

import numpy as np
import tensorflow as tf

# Load TFLite model
interpreter = tf.lite.Interpreter(model_path="distilgpt2.tflite")
interpreter.allocate_tensors()

# Get input and output details
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Example input (token IDs from tokenizer)
# NOTE: Use Hugging Face's GPT2 tokenizer to prepare inputs
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("distilgpt2")

input_ids = tokenizer.encode("The future of AI is", return_tensors="np")

# Prepare input tensor
interpreter.set_tensor(input_details[0]['index'], input_ids.astype(np.int32))

# Run inference
interpreter.invoke()

# Extract output
output = interpreter.get_tensor(output_details[0]['index'])
decoded_text = tokenizer.decode(output[0])
print(decoded_text)

3. Integration with Flutter

import 'package:tflite_flutter/tflite_flutter.dart';

final interpreter = await Interpreter.fromAsset('distilgpt2.tflite');

// Prepare input tensor and run inference
// (tokenization/decoding handled separately)
// Generate answer based on query

Contributing

Found an issue or want to improve the model? Feel free to:

• Open an issue for bugs or suggestions
• Submit pull requests for improvements
• Share your use cases and results

License

This model follows the same license terms as the original distilgpt2 model.

---

Made with ❤️ for the mobile AI community

Enabling powerful text understanding directly on your device