arcgis-test-agent

ArcGIS Test Script Generator Agent

An AI agent that reads ArcGIS tool documentation and generates test scripts following your team's template and conventions.

Quick Start

# 1. Install
pip install -e .

# 2. Set credentials
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/"
export AZURE_OPENAI_KEY="your-key"

# 3. Set up your directories
export DOCS_DIR="./docs"           # Tool documentation (.md files)
export TEMPLATE_DIR="./templates"  # Your test template + validators.json
export EXAMPLES_DIR="./examples"   # Example test scripts by category
export TEST_DATA_ROOT="./test_data" # With manifest.json

# 4. Generate a test
python -m arcgis_test_agent generate Buffer

How It Works

User: "Generate test for Buffer"
         │
         ▼
┌─────────────────────────────────────────────┐
│  PHASE 1: PLANNING                          │
│  • read_tool_docs("Buffer")                 │
│  • search_data_index("polygon shapefile")   │
│  • search_data_index("polyline features")   │
│  • read_output_validators("feature_class")  │
│  • read_example_tests("analysis")           │
├─────────────────────────────────────────────┤
│  PHASE 2: CODING                            │
│  • read_test_template()                     │
│  • [LLM writes complete test script]        │
├─────────────────────────────────────────────┤
│  PHASE 3: REVIEW                            │
│  • validate_script(code)                    │
│  • [Fix issues if any, retry up to 3x]     │
└─────────────────────────────────────────────┘
         │
         ▼
  generated_tests/test_buffer.py

Directory Setup

You need to provide 4 things:

1. docs/ — Tool Documentation

One markdown file per tool:

docs/
├── Buffer.md
├── Clip.md
├── ReconstructSurface.md
└── ...

2. templates/test_template.py — Your Test Template

The exact structure your generated tests must follow.

3. templates/validators.json — Output Validators Registry

{
    "feature_class": [
        {"name": "assertFeatureClassExists", "signature": "self.assertFeatureClassExists(path)", "description": "..."}
    ],
    "raster": [...],
    "json": [...]
}

4. test_data/manifest.json — Available Test Data Index

[
    {
        "path": "vector/polygon/parcels.shp",
        "data_type": "vector",
        "geometry_type": "polygon",
        "description": "Polygon shapefile with 500 parcel features in NAD83",
        "spatial_reference": "NAD83 UTM Zone 11N",
        "feature_count": 500
    }
]

CLI Usage

python -m arcgis_test_agent generate Buffer
python -m arcgis_test_agent generate Buffer --output tests/test_buffer.py
python -m arcgis_test_agent generate Buffer --context "focus on 3D polygon inputs"
python -m arcgis_test_agent generate Buffer --verbose
python -m arcgis_test_agent info

Python API

from arcgis_test_agent import TestGeneratorAgent, AzureConfig, AgentConfig

agent = TestGeneratorAgent()
result = agent.generate("Buffer")

if result.success:
    print(result.script)

Evaluation

from arcgis_test_agent.eval import evaluate_structural

with open("generated_tests/test_buffer.py") as f:
    code = f.read()

score = evaluate_structural(code)
print(f"Composite score: {score.composite:.3f}")

Demo (No Azure Credentials Required)

python arcgis_test_agent/demo/run_demo.py

Project Structure

arcgis_test_agent/
├── __init__.py          # Package entry point
├── __main__.py          # CLI
├── agent.py             # Core agent loop (tool-calling)
├── config.py            # Configuration management
├── eval.py              # Evaluation framework
├── tools/
│   ├── schemas.py       # OpenAI function schemas (6 tools)
│   └── handlers.py      # Tool implementations
├── prompts/
│   └── system.py        # System prompt
└── demo/
    ├── run_demo.py      # End-to-end demo
    ├── docs/            # Sample tool docs
    ├── templates/       # Sample template + validators
    ├── examples/        # Sample test scripts
    └── test_data/       # Sample manifest

<!-- ml-intern-provenance -->

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "dduseja/arcgis-test-agent"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.