NISKA-3b_Q4_K_M

NISKA 3b_Q4_K_M Legal LLM

NISKA 3b_Q4_K_M is a 3-billion parameter Large Language Model (LLM) fine-tuned on a massive dataset of over 70,000 legal documents. This specialization makes it adept at understanding and generating text related to legal concepts, terminology, and documents.

This model is provided in the GGUF (GPT-Generated Unified Format), which is optimized for running efficiently on CPUs and can also be offloaded to GPUs. This guide will walk you through setting up and running the NISKA 3b_Q4_K_M.gguf file using two popular tools: llama.cpp for direct command-line interaction and Ollama for a more integrated, service-based approach.

1. Prerequisites

Before you begin, ensure you have the following installed on your system:

Git: For cloning repositories.

A C++ compiler and build tools:

    On Linux/macOS: make, cmake, g++/clang.

    On Windows: Visual Studio with the "Desktop development with C++" workload.

Ollama: If you plan to use the Ollama setup. You can download it from ollama.com.

The NISKA 3b_Q4_K_M.gguf file itself. Make sure you have downloaded it and placed it in a known directory.

2. Using with llama.cpp

llama.cpp is a high-performance C++ library for running LLMs. It's a great option for running the model directly from your terminal with fine-grained control over its parameters. Step 1: Clone and Build llama.cpp

First, open your terminal and clone the llama.cpp repository from GitHub. Then, navigate into the directory and build the project.

Clone the repository

git clone https://github.com/ggerganov/llama.cpp.git

Navigate into the project directory

cd llama.cpp

Build the project (this command works for most systems)

For specific hardware (like NVIDIA GPUs), see the llama.cpp documentation

make

Step 2: Run the Model

Once llama.cpp is built, you will have an executable named main in the directory. Move your NISKA 3b_Q4_K_M.gguf file into this directory (or provide the full path to it).

You can now run the model using the main executable.

./main -m ./NISKA_3b_Q4_K_M.gguf -n 512 -p "Draft a clause for a confidentiality agreement that covers trade secrets." --color -i

Command Breakdown:

-m ./NISKA_3b_Q4_K_M.gguf: Specifies the path to your model file.

-n 512: Sets the maximum number of tokens to generate in the response.

-p "...": Provides the initial prompt to the model.

--color: Displays the output in color for better readability.

-i: Runs the model in interactive mode, allowing you to have a conversation.

3. Using with Ollama

Ollama provides a simple and powerful way to run LLMs locally as a service. You interact with it through a command-line interface, and it manages the model for you. Step 1: Create a Modelfile

To import a GGUF file into Ollama, you need to create a Modelfile. This is a plain text file that tells Ollama where to find the model and how to configure it.

Create a new file named Modelfile (no extension) in the same directory as your NISKA_3b_Q4_K_M.gguf file. Add the following content:

Modelfile for NISKA 3b Legal LLM

FROM ./NISKA_3b_Q4_K_M.gguf

Optional: Set default parameters for the model

PARAMETER temperature 0.7

PARAMETER top_k 40

PARAMETER top_p 0.9

Set the system prompt (optional, but recommended for specialized models)

SYSTEM """You are NISKA, a specialized legal assistant. Your purpose is to provide information and draft text based on legal concepts. Do not provide legal advice. All information is for educational purposes only."""

Step 2: Create the Model in Ollama

Open your terminal and use the ollama create command to register the model from your Modelfile.

Make sure you are in the directory containing the Modelfile and the .gguf file

ollama create niska-legal -f Modelfile

niska-legal: This is the name you are giving the model within Ollama. You can choose any name you like.

-f Modelfile: This points to your Modelfile.

Ollama will process the file and add niska-legal to your list of available models. Step 3: Run the Model

Now you can run your model using the ollama run command.

ollama run niska-legal

This will start an interactive session where you can chat with the NISKA legal model. Example Prompts

Here are a few examples of prompts you can use to test the model:

"Explain the concept of 'res judicata' in simple terms."

"What are the key elements required to form a valid contract?"

"Summarize the main points of the GDPR regarding data subject rights."

"Draft a basic non-disclosure agreement (NDA) between two parties."

Disclaimer

This model is for informational and educational purposes only. The output generated by NISKA 3b_Q4_K_M is not a substitute for professional legal advice. Always consult with a qualified legal professional for any legal issues or concerns. The creators of this model are not liable for any actions taken based on its output.

---

license: mit