simpli

About

This project runs a FastAPI server (python) and serves a Large-language-model using the llama_cpp library (llama-cpp-python binding).

File Structure

| app/                  : contains code for the FastAPI
| | config/
| | | model_init.json   : configs for model being served.
| | llm_lib/
| | | chat.py           : Objects for request/response by the server.
| | | llm_logger.py     : logging library [overkill].
| | | llm_models.py     : Wrapper class over "Llama" model. [no real benefit?]
| | | model_init.py     : LLM manager - initializes the model based on config.
| | server/
| | | server.py         : FastAPI server.
| | testing/
| | | api_testing.py    : for testing a set of questions against a llm.
| | | prompt.json       : sample request-data for the API server.
| | | questions.json    : a set of 80 questions for evaluating the model.
| | Makefile            : has functions to execute server/tests.
| models/               : quantized models [using llama.cpp binaries]
| README.md

Getting started (running the API server)

Firstly configure the config/model_init.json file by providing the "model_path" field. This project is built using llama_cpp the model's format is GGUF.

Use the Makefile

1. start FastAPI server with reload (for development use-case): make start_dev

2. start FastAPI server without reload: make start

3. initialize a new model [serve a different model without restart the server] Set the "force_init" to true in config/model_init.json, then run - make initialize NOTE: The server loads the model in the model_init.json during startup.

4. Sanity testing - make sure the server is running. make chat This uses "testing/prompt.json" file for POST request.

TODO :

1. Speculative Decoding using a draft model.
   • support loading via "API" [combination of base model and draft model].
2. Dynamic Batching
   • aka continuous batching : supported by llama.cpp (don't about the python-binding).
3. Medusa Heads
   • goes a step beyond Speculative Decoding and requires training with the Frozen Base model.
   • kind-of integrated with the base-model (becomes the last layer of the LLM)
4. Benchmarking
   • take inspiration from llama.cpp (supports various parameters for testing)
     • batch_size
     • #threads
     • set of models
5. Tidy the code-base. Remove unnecessary code (after learning the RIGHT approach w.r.t LLMs).