granite-3.3-8b-rag-agent-lib

Granite 3.3 RAG Agent Library

The LLM Intrinsics RAG Library includes six intrinsics implemented as LoRA adapters for ibm-granite/granite-3.3-8b-instruct, each of which expects as input a (single-turn or multi-turn) conversation between a user and an AI assistant, and most of which also expect a set of grounding passages. Each intrinsic has been developed for a specific task that is likely to be useful in Agentic RAG pipelines. We give a brief overview of the functionality of each intrinsic, as the details can be found in each individual intrinsic readme.

Intrinsics implemented as LoRA adapters

The six intrinsics that have been implemented as LoRA adapters for ibm-granite/granite-3.3-8b-instruct and made available in this HF repository are:

Query Rewrite (QR): Given a conversation ending with a user query, QR will decontextualize that last user query by rewriting it (whenever necessary) into an equivalent version that is standalone and can be understood by itself. While this adapter is general purpose for any multi-turn conversation, it is especially effective in RAG settings where its ability to rewrite a user query into a standalone version directly improves the retriever performance, which in turn improves the answer generation performance. This is a pre-retrieval intrinsic since its suggested use is before invoking retrieval.

Context Relevance (CR): Given a conversation ending with a user query, and an individual passage, CR classifies whether the passage is relevant, partially relevant, or irrelevant for answering the last user query - or if the passage may instead mislead or harm the downstream generator model’s response quality. This is a pre-generation intrinsic.

Answerability Determination (AD): Given a conversation ending with a user query, and a set of passages, AD classifies whether that final user query is answerable or unanswerable based on the available information in the passages. It is valuable for restraining over-eager models by identifying unanswerable queries and preventing the generation of hallucinated responses. It can also be used to indicate that the system should re-query the retriever with alternate formulations, to fetch more relevant passages. This is a pre-generation intrinsic.

Uncertainty Quantification (UQ): Given a conversation ending with an assistant response, UQ calculates a certainty percentage to reflect how certain it is about the answer generated to the previous user query. UQ can also take as input a conversation ending with an user query and predicting the certainty score based solely on the query, prior to generating an answer. UQ is also calibrated on document-based question answering datasets, and hence can be applied to give certainty scores for RAG responses created using grounding passages. This intrinsic could be used in a post-generation or pre-generation step.

Hallucination Detection (HD): Given a conversation ending with an assistant response, and a set of passages, HD outputs a hallucination risk range for each sentence in the last assistant response, with respect to the set of passages. This could be used in concert with sampling techniques that yield multiple generated responses, some of which could then be filtered according to their HD scores. This is a post-generation intrinsic since its expected use is after invoking the LLM to create the response.

Citation Generation (CG): Given a conversation ending with an assistant response, and a set of passages, CG generates citations for that last assistant response from the provided passages. Citations are generated for each sentence in the response (when available), where each citation consists of a set of sentences from the supporting passages. This is a post-generation intrinsic since its expected use is after invoking the LLM, and therefore can be used to create citations for responses generated by any model.

Recommended Use

The recommended use of these LoRA adapters is through Granite IO Processing, a framework which enables transforming how a user calls or infers an IBM Granite model and how the output from the model is returned to the user. In other words, the framework allows extended functionality of calling the model. This is particularly valuable as the downstream use of intrinsics relies on correctly structured output. Although we have made the individual LoRAs available, we strongly suggest that everyone use the implementations in Granite IO and we have made example notebooks available. Two additional intrinsics, Query Expansion and Passage Reranking, are pure software-based implementations and are only available through Granite IO.

Intrinsics implemented as functions

Query Expansion (QE): Given a conversation ending with a user query, QE is designed to probe the retriever from multiple angles by generating a set of semantically diverse versions of that last user query. This expanded set of queries provides diverse retrieval paths, and thus this intrinsic is particularly effective in RAG settings, especially with terse, general, or underspecified queries. Like Query Rewrite, this is a pre-retrieval intrinsic.

Quickstart Example

As explained above, it is highly recommended to use these LoRA adapters through Granite IO Processing.

However, if you prefer to invoke the LoRA adapters directly, you can follow the following process.

1. Select the LoRA adapter that you want to experiment with from here.
2. Download the LoRA adapter to a local directory. Following example shows how to download the "hallucination_detection_lora" intrinsic to the local directory intrinsics/hallucination_detection

from huggingface_hub import HfApi, hf_hub_download
import os
from tqdm import tqdm

def download_intrinsic(
    repo_id: str,
    intrinsic: str,
    local_dir: str,
    token: str = None,
):
    api = HfApi(token=token)
    files = api.list_repo_files(repo_id=repo_id, token=token)

    # Keep only files under your desired subfolder
    files = [f for f in files if f.startswith(intrinsic.rstrip("/") + "/")]

    os.makedirs(local_dir, exist_ok=True)

    for file_path in tqdm(files, desc="Downloading files"):
        rel_path = os.path.relpath(file_path, intrinsic)
        dest_path = os.path.join(local_dir, rel_path)
        os.makedirs(os.path.dirname(dest_path), exist_ok=True)

        hf_hub_download(
            repo_id=repo_id,
            filename=file_path,
            local_dir=local_dir,
            token=token,
        )


download_intrinsic(
    repo_id="ibm-granite/granite-3.3-8b-rag-agent-lib",
    intrinsic="hallucination_detection_lora",
    local_dir="intrinsics/hallucination_detection",
    token="YOUR_HUGGINGFACE_TOKEN",  # omit if not needed
)

1. Load the LoRA adapter from the downloaded local directory and run the intrinsic model. Each intrinsic contains a README file inside the LoRA adapter directory which explains how to run the model. Here is an example.