text-qa-openenv

Text QA OpenEnv Environment

🎮 A Robust Question Answering ENVIRONMENT for Meta PyTorch Hackathon 2026 - Round 1

Built for Environment Evaluation, Not Agent Training!

This is a production-grade RL environment (the "playground") that provides standardized question-answering tasks for AI agents. Focus is on environment robustness, not agent implementation.

🎯 Round 1 Submission - Environment Components

This environment implements ALL required components for Round 1:

1. Observation Space (What the agent sees)

{
    'question': str,      # The question to answer
    'context': str,       # Context containing the answer
    'question_id': int    # Unique question identifier
}

2. Action Space (What the agent can do)

• Type: Text string
• Description: Agent provides an answer as a string

3. Reward Function (Scoring system)

• +1.0: Correct answer (exact match)
• +0.5: Partial match (contains key information)
• -0.1: Incorrect answer

4. Automated Grader (Success evaluation)

• Programmatic evaluation of agent performance
• Pass/Fail criteria: ≥70% accuracy OR ≥85% partial accuracy
• Detailed per-question feedback
• JSON export for evaluation

5. Robust Error Handling

• Never crashes on invalid actions
• Handles None, empty strings, wrong types
• Graceful fallbacks for all edge cases

Key Features

• 🎮 Robust Environment: Never crashes on invalid actions - comprehensive error handling
• 🌐 Real-World Datasets: SQuAD (10k+ questions), TriviaQA (95k questions)
• 📋 Built-in Questions: 33 curated questions for quick testing
• 🔄 Gymnasium Compliant: Full OpenEnv/Gymnasium API implementation
• ✅ Automated Grader: Programmatic agent evaluation system
• 🏅 Multi-tier Rewards: Smart reward shaping for learning
• 🐳 Dockerized: Containerized for consistent evaluation
• 📊 Validated: Passes all robustness tests for Round 1

Installation

Quick Install (Minimal - for Round 1)

pip install -r requirements.txt

Build time: ~2 minutes | Size: ~100MB

This includes only the essentials:

• gymnasium (environment framework)
• numpy (numerical operations)

Full Install (with External Datasets - Optional)

pip install -r requirements-full.txt

Build time: ~60 minutes | Size: ~2GB

Additional features:

• SQuAD dataset (10k+ questions)
• TriviaQA dataset (95k+ questions)
• Transformer models support

Quick Start - Testing the Environment

Validate Environment Robustness (Run This First!)

pip install -e .
python validate_environment.py

This runs comprehensive tests to ensure the environment:

• ✅ Never crashes on invalid actions
• ✅ Handles all edge cases
• ✅ Complies with Gymnasium API
• ✅ Grader works correctly

Basic Environment Usage

from text_qa_env import TextQAEnv

# Create environment
env = TextQAEnv(difficulty='easy')

# Reset to get initial observation
observation, info = env.reset()

# Agent takes an action (provides an answer)
action = "Paris"
observation, reward, terminated, truncated, info = env.step(action)

# Check results
print(f"Reward: {reward}")
print(f"Correct: {info['correct']}")

Using External Datasets (SQuAD, TriviaQA)

from text_qa_env import TextQAEnv

# Use SQuAD dataset with 10,000+ questions
env = TextQAEnv(
    difficulty='external',
    external_dataset='squad',  # or 'squad_v2', 'trivia_qa'
    max_questions=1000
)

observation, info = env.reset()
action = "agent's answer"
observation, reward, terminated, truncated, info = env.step(action)

Using the Automated Grader (Required for Hackathon)

from text_qa_env import run_automated_grading

# Define your agent
class MyAgent:
    def select_action(self, observation):
        # Your agent logic here
        return "answer"

# Run automated grading
agent = MyAgent()
report = run_automated_grading(agent, difficulty='easy')

# Results include:
# - Pass/Fail status
# - Accuracy metrics
# - Detailed per-question results
# - Letter grade (A+ to F)

Training an Agent (NOT Required for Round 1)

Note: Agent training is for Round 2. Round 1 focuses on environment quality.

# This is optional - shows the environment can be used for training
from examples.train_agent import train_simple_agent

agent = train_simple_agent(episodes=100, difficulty='easy')

Environment Specification

Observation Space

• Type: Dictionary
  • question: String - The question to answer
  • context: String - Context containing the answer
  • question_id: Integer - Unique question identifier

Action Space

• Type: String - The agent's answer

Rewards

• +1.0: Correct answer (exact match)
• +0.5: Partial match (contains correct keywords)
• -0.1: Incorrect answer

Episode Termination

• Episode ends after answering the question
• Maximum 1 question per episode

Grading Criteria (Automated)

• Pass: ≥70% accuracy OR ≥85% partial accuracy
• Grades: A+ (90%+), A (80%+), B (70%+), C (60%+), D (50%+), F (<50%)

Datasets

Built-in Questions (33 total)

• Easy (15 questions): Simple factual questions (capitals, famous people, basic science)
• Medium (10 questions): Multi-step reasoning, historical dates, scientific concepts
• Hard (8 questions): Complex inference, advanced science, mathematical theorems

External Datasets (Production-Ready)

• SQuAD: 10,570 questions from Wikipedia paragraphs
• SQuAD v2: Includes unanswerable questions for robustness
• TriviaQA: 95,000 question-answer pairs from trivia sources

Usage:

# SQuAD (recommended for research)
env = TextQAEnv(difficulty='external', external_dataset='squad')

# TriviaQA (larger dataset)
env = TextQAEnv(difficulty='external', external_dataset='trivia_qa', max_questions=1000)

Project Structure

Open-Env/
├── text_qa_env/
│   ├── __init__.py
│   ├── environment.py         # Main Gymnasium environment
│   ├── dataset.py             # Built-in QA questions
│   ├── external_datasets.py   # SQuAD/TriviaQA integration 🌐
│   └── grader.py              # Automated grading system ✅
├── examples/
│   ├── basic_usage.py         # Simple usage example
│   ├── train_agent.py         # Training script with Q-learning agent
│   ├── test_grader.py         # Grader demonstration
│   └── external_dataset_demo.py  # SQuAD integration demo 🌐
├── Dockerfile
├── requirements.txt
├── setup.py
└── README.md

Hackathon Requirements Checklist ✅

This environment fully implements all Meta PyTorch Hackathon Round 1 requirements:

• ✅ Environment (The Playground): TextQAEnv with complete task logic
• ✅ Observation Space: Clearly defined (question, context, question_id)
• ✅ Action Space: Text string answers
• ✅ Reward Function: Multi-tier scoring (+1.0, +0.5, -0.1)
• ✅ Automated Graders: Programmatic success evaluation (QAGrader)
• ✅ OpenEnv/Gymnasium: Full API compliance (reset, step, render)
• ✅ Error Handling: Never crashes on any action (tested with 7 edge cases)
• ✅ Docker: Containerized for consistent execution
• ✅ Documentation: Comprehensive README and examples
• ✅ Validation: Passes all robustness tests

Round 1 Focus: Environment quality, not agent training!

Contributing

This project was created for the Meta PyTorch Hackathon. Contributions are welcome!

License

MIT License

Author

Built with ❤️ for Meta PyTorch Hackathon 2026