Woven-City-AI-Vision-Engine

Woven City AI Vision Engine

🧠 Model Summary

Woven City AI Vision Engine is a cutting-edge, 8B parameter long-context video modeling solution that enables efficient and high-quality video understanding. The model introduces a hierarchical compression method to process long/short videos without the typical quality degradation or high computational costs. This allows for a comprehensive analysis of videos, capturing both fine-grained details and overarching narratives.

✨ Key Features

• Hierarchical Compression: Employs a sophisticated two-tier compression strategy. A fine-grained compressor that processes short-term temporal information, while a coarse-grained compressor handles long-term context.
• Long-Context Understanding: Capable of processing and understanding videos of extended duration.
• Efficient and High-Fidelity: Maintains high-fidelity video representation with a high compression ratio ensuring both efficiency and accuracy.
• Cutting-edge Performance: Outperforms existing models in long-video question answering and excels in the video understanding benchmark MVBench with a score of 73.81%

---

🏆 MVBench Evaluation Leaderboard

Results confirmed as of 2025.09. Evaluated on the standard MVBench test set.

⚙️ How to Use

This model is distributed as a Model Package on the Amazon Marketplace. To use it, you must first subscribe to the model and then deploy it to a SageMaker endpoint. The following steps guide you through deploying the model and performing inference for text, image, and video analysis. Please refer to the inference.ipynb notebook for using this model using Amazon Marketplace.

1. Deploy the Model to a SageMaker Endpoint

After subscribing to the model package on the AWS Marketplace, you can use the following Python code to create a SageMaker endpoint. You will need your AWS account's execution role and the Model Package ARN (Amazon Resource Name) from the subscription page.

Deployment Script

import sagemaker
from sagemaker import ModelPackage
from sagemaker.predictor import Predictor
from datetime import datetime

# --- Configuration ---
# Get your execution role
try:
    role = sagemaker.get_execution_role()
except ValueError:
    # If not in a SageMaker environment, specify the ARN directly
    role = "arn:aws:iam::YOUR_ACCOUNT_ID:role/YourSageMakerExecutionRole"

# Find your Model Package ARN on the AWS Marketplace subscription page
model_package_arn = "arn:aws:sagemaker:REGION:ACCOUNT_ID:model-package/YourModelPackageName" 

# Define deployment instance type and count
instance_type = "ml.g5.2xlarge" # Recommended instance type
instance_count = 1

# Create a unique endpoint name
endpoint_name = f"woven-city-ai-vision-engine-endpoint-{datetime.now().strftime('%H%M%S')}"

# --- Deployment ---
sagemaker_session = sagemaker.Session()

# Create a model package object
model_package = ModelPackage(
    role=role,
    model_package_arn=model_package_arn,
    sagemaker_session=sagemaker_session
)

# Deploy the model
print(f"Deploying endpoint '{endpoint_name}'... This may take 10-15 minutes.")
predictor = model_package.deploy(
    initial_instance_count=instance_count,
    instance_type=instance_type,
    endpoint_name=endpoint_name
)

print(f"Model deployed successfully to endpoint: {predictor.endpoint_name}")

2. Setup and Helper Functions

Once the endpoint is InService, set up your environment to connect to it. Define helper functions for encoding media and sending requests.

Connect to the Endpoint

import boto3
import json
import base64
import sagemaker
from sagemaker.predictor import Predictor

# Configure your AWS session if needed
# boto3.setup_default_session(profile_name = 'your-profile')
session = boto3.Session()
sagemaker_session = sagemaker.Session()
region = session.region_name

# Use the endpoint name from the deployment step
# endpoint_name = "your-sagemaker-endpoint-name" 

predictor = Predictor(
    endpoint_name=endpoint_name,
    sagemaker_session=sagemaker_session
)

print(f"Connected to endpoint: {predictor.endpoint_name}")

Helper Functions

def encode_image_to_base64(image_path):
    """Encode image file to base64 data URL"""
    with open(image_path, 'rb') as f:
        image_data = base64.b64encode(f.read()).decode('utf-8')
    
    if image_path.lower().endswith(('.jpg', '.jpeg')):
        mime_type = 'image/jpeg'
    elif image_path.lower().endswith('.png'):
        mime_type = 'image/png'
    else:
        mime_type = 'image/jpeg'
    
    return f"data:{mime_type};base64,{image_data}"

def encode_video_to_base64(video_path):
    """Encode video file to base64 data URL"""
    with open(video_path, 'rb') as f:
        video_data = base64.b64encode(f.read()).decode('utf-8')
    
    return f"data:video/mp4;base64,{video_data}"

def send_request(predictor, payload):
    """Send request to SageMaker endpoint with error handling"""
    try:
        payload_json = json.dumps(payload)
        
        response = predictor.predict(
            payload_json,
            initial_args={"ContentType": "application/json", "Accept": "application/json"}
        )
        
        if isinstance(response, bytes):
            response = response.decode('utf-8')
        
        if isinstance(response, str):
            response = json.loads(response)
            
        return response
        
    except Exception as e:
        print(f"An error occurred: {e}")
        return None

3. 📄 Text Analysis

You can use the model for standard text generation tasks by providing a text-only prompt.

Example Request

text_payload = {
    "instances": [{
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Explain the importance of workplace safety in manufacturing environments."
                }
            ]
        }],
        "parameters": {
            "max_new_tokens": 120,
            "temperature": 0.7,
            "do_sample": True
        }
    }]
}

print("=== Text Analysis ===")
response = send_request(predictor, text_payload)
if response:
    print("Generated Text:")
    print(response.get('generated_text', 'No response'))

Generated Text: Workplace safety in manufacturing environments is paramount for the well-being of employees and the smooth operation of the facility. It involves a range of measures, from the use of personal protective equipment to the implementation of safety protocols and training programs. Ensuring a safe workplace not only protects the physical health of the employees but also enhances productivity, reduces costs associated with accidents, and fosters a positive work environment.

4. 🖼️ Image Analysis

Provide an image along with a text prompt to perform visual question answering or description tasks.

Example Request

# Note: Replace 'sample1.jpg' with your image file path
image_path = "media/sample1.jpg" 

image_payload = {
    "instances": [{
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What do you see in this picture."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": encode_image_to_base64(image_path),
                        "detail": "high"
                    }
                }
            ]
        }],
        "parameters": {
            "max_new_tokens": 150,
            "temperature": 0.6
        }
    }]
}

print("=== Image Analysis ===")
response = send_request(predictor, image_payload)
if response:
    print("Analysis:")
    print(response.get('generated_text', 'No response'))

Example Input (sample1.jpg)

<img src="https://huggingface.co/wovenbytoyota-vai/Woven-City-AI-Vision-Engine/resolve/main/media/sample1.jpg" alt="Breathtaking mountainous landscape" width="360">

The video showcases a breathtaking mountainous landscape. The scene is dominated by a majestic mountain in the background, its peak capped with snow, indicating a high altitude. The mountain's slopes are a mix of rocky terrain and patches of snow, suggesting a rugged and possibly challenging environment. In the foreground, there are several small lakes, their calm waters reflecting the surrounding scenery. The lakes are nestled among lush green forests, with trees of varying heights and densities. The sky above is clear and blue, adding to the serene and picturesque quality of the scene. The overall composition of the video highlights the natural beauty and tranquility of the landscape, with the mountain, lakes, and forest creating a harmonious and visually appealing scene.

5. 📹 Video Analysis

For video analysis, provide a video file and a prompt to ask questions about its content, summarize events, or identify actions.

Example Request

video_path = "media/sample2.mp4"

video_payload = {
    "instances": [{
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe what happens in this video and identify any notable events or actions."
                },
                {
                    "type": "video_url",
                    "video_url": {
                        "url": encode_video_to_base64(video_path),
                        "fps": 2.0,
                        "max_frames": 16
                    }
                }
            ]
        }],
        "parameters": {
            "max_new_tokens": 180,
            "temperature": 0.6,
            "fps": 2.0,
            "max_num_frames": 16
        }
    }]
}

print("=== Video Analysis ===")
response = send_request(predictor, video_payload)
if response:
    print("Analysis:")
    print(response.get('generated_text', 'No response'))

Example Input (sample.mp4)

<video src="https://huggingface.co/wovenbytoyota-vai/Woven-City-AI-Vision-Engine/resolve/main/media/sample3.mp4" controls preload="metadata" width="360" > Sorry — your viewer doesn’t support embedded video. </video>

The video presents a sweeping aerial view of London's skyline, showcasing a variety of architectural styles and building heights. The sequence begins with a wide shot of the city, gradually panning in to reveal more details. The buildings are predominantly modern, with glass facades reflecting the sunlight, and some have distinctive shapes, such as the curved glass structure and the spire of the Shard. The color palette is dominated by the blue of the sky and the grey of the buildings, with occasional greenery on rooftops. There are no visible characters or movement, suggesting a focus on the city's static beauty. The light is bright and natural, indicating daytime with clear weather conditions. The video captures the essence of London's urban landscape, emphasizing its density and architectural diversity.

📝 Parameters Description

Reproducibility: for deterministic, repeatable output set do_sample=false. Sampling parameters are then ignored.

⚠️ Common Mistakes

Additional Resources

• Amazon Marketplace — Amazon Marketplace
  Visit our Amazon Bedrock Marketplace listing to deploy Woven City AI Vision Engine in your AWS account

• InstVL — InstVL Dataset
  InstVL is a large-scale dataset of images and videos designed to bridge the gap between holistic scene understanding and fine-grained, instance-level comprehension.

• InstQA — InstQA Dataset
  InstQA is a large-scale dataset of images and videos. The dataset contains dense instance-level captions and Visual Question Answers.

• WTS Dataset — WTS: Woven Traffic Safety Dataset
  A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding

🙏 Acknowledgements

This dataset is based on results obtained from a project, JPNP20017, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).