Phi3-Image-Question-Generator-2

Model Card for Model ID

A model purpose made for MCQ generation on cartoon images for lower primary language education. Phi3-mini is used as the LM and CLIP is used as the vision encoder.

Requires a CUDA enabled devices to run. Please run the model using Google Colab with a T4 card, or a local CUDA device with at least 15GB of available VRAM. Ensure that CUDA is built according to your device requirements.

Uses

The model is meant to be used with a custom pipeline. It returns a dict Object with with the following keys

1. questions (str)
2. choices (list)
3. answer (str)
4. desc (str)

Docs

Use the generateQn() function to generate the above. The function takes in the following parameters:

1. img_path (str): file path to to image file
2. n (int): numbers of question set to output

from transformers import AutoModelForCausalLM, AutoProcessor
import torch
import json

processor = AutoProcessor.from_pretrained("Clyine1/Phi3-Image-Question-Generator-2")
model = AutoModelForCausalLM.from_pretrained("Clyine1/Phi3-Image-Question-Generator-2",
                                              trust_remote_code=True, proc=processor,
                                              torch_dtype=torch.float16).to(0)


out = model.generateQn(<img_path>, 1)
print(json.dumps(out, indent=4))
"""
Generated output:
[
    {
        "desc": "In this vibrant and lively illustration, a young boy with black hair and a frown on his face sits at a desk, looking disheartened. He is wearing a blue shirt with a P logo and a pink headband. The boy is holding a pencil in his hand, but his expression suggests that he is struggling with his homework.\n\nThe desk is cluttered with various objects, including a pink stuffed animal with a P logo, a stack of books, and a pile of papers. The boy's attention seems to be focused on a piece of paper with a math problem, but he appears to be having difficulty understanding it.\n\nIn the background, a colorful and cheerful classroom setting can be seen. The walls are adorned with posters and paintings, including a picture of a smiling teacher, a cartoon of a happy lion, and a drawing of a smiling sun. The room is filled with other children, some of whom are sitting at their desks, looking bored or disinterested.\n\nOne boy in particular, sitting at a nearby desk, appears to be sleeping with his head resting on his arms. Another boy is sitting on a chair, looking frustrated as he tries to solve a math problem on a piece of paper. A girl with a pink headband and a pink shirt is sitting at a different desk, looking confused as she stares at her own math problem.\n\nThe overall atmosphere of the classroom is one of disengagement and frustration, with the students struggling to understand and complete their homework. The boy at the desk with the pencil appears to be the most affected by this, as he looks sad and defeated.",
        "question": "Why is the boy with the pencil looking sad and defeated while trying to solve his math problem?",
        "Choices": [
            "1) The boy is upset because he lost his favorite toy.",
            "2) The boy is sad because he didn't get a good grade on his test.",
            "3) The boy is struggling to understand the math problem.",
            "4) The boy is frustrated because he can't find his homework."
        ],
        "Answers": "Correct Answer: 3) The boy is struggling to understand the math problem."
    }
]
"""