OpenEvidence Production Checklist

OpenEvidence Production Checklist

Overview

Comprehensive checklist for deploying OpenEvidence clinical AI integrations to production healthcare environments.

Prerequisites

• Completed development and testing phases
• Signed BAA with OpenEvidence
• Security review completed
• Compliance team sign-off

Production Readiness Checklist

1. Legal & Compliance

• Business Associate Agreement (BAA) signed

  • Contact: [email protected]
  • Required for any PHI handling

• API Terms of Service accepted

  • Review: https://www.openevidence.com/policies/api

• Clinical use disclaimers in place

  • All clinical outputs must include appropriate disclaimers
  • "For informational purposes only. Verify with current guidelines."

• Liability and malpractice considerations reviewed

  • Legal review of clinical decision support usage

• Data processing agreement documented

  • What data is sent, stored, processed

2. Security

• Production API keys generated

  • Separate keys from development/​staging
  • Stored in secret manager (not environment variables)

• Key rotation schedule established

  • Recommend: 90-day rotation cycle

• IP allowlist configured (if available)

  • Restrict API access to known IP ranges

• TLS 1.2+ enforced

  • No fallback to older protocols

• PHI sanitization implemented

  • Review openevidence-security-basics

• Webhook signature verification active

  • All webhook endpoints validate signatures

• Audit logging enabled

  • HIPAA-compliant access logging
  • Minimum 6-year retention

3. Infrastructure

• High availability configuration

  // Multiple API endpoints for failover
  const config = {
    primaryEndpoint: 'https://api.openevidence.com',
    fallbackEndpoint: 'https://api-fallback.openevidence.com',
    timeout: 30000,
    retries: 3,
  };
  

• Circuit breaker implemented

  import CircuitBreaker from 'opossum';
  
  const breaker = new CircuitBreaker(clinicalQuery, {
    timeout: 30000,
    errorThresholdPercentage: 50,
    resetTimeout: 30000,
  });
  

• Rate limit handling configured

  • Review openevidence-rate-limits
  • Request queuing for traffic spikes

• Timeout configuration appropriate

  • Clinical queries: 15-30s
  • DeepConsult: 5-10 minutes

• Health check endpoint implemented

  app.get('/​health/​openevidence', async (req, res) => {
    try {
      await client.health.check();
      res.json({ status: 'healthy' });
    } catch (error) {
      res.status(503).json({ status: 'unhealthy', error: error.message });
    }
  });
  

4. Monitoring & Alerting

• Metrics collection active

  • Request count, latency, error rate
  • Review openevidence-observability

• Alert rules configured

  Alert	Threshold	Severity
  Error rate > 5%	5 min window	Warning
  Error rate > 20%	5 min window	Critical
  P95 latency > 15s	5 min window	Warning
  Health check failed	2 consecutive	Critical

• Dashboard created

  • Real-time visibility into OpenEvidence integration health

• Log aggregation configured

  • Centralized logging for debugging
  • PHI excluded from logs

5. Error Handling & Resilience

• Graceful degradation implemented

  async function clinicalQueryWithFallback(question: string) {
    try {
      return await openEvidenceQuery(question);
    } catch (error) {
      // Log error, alert, return cached or default response
      await alertService.send('OpenEvidence unavailable');
      return {
        answer: 'Clinical evidence service temporarily unavailable. Please consult current guidelines directly.',
        fallback: true,
      };
    }
  }
  

• Retry logic tested

  • Exponential backoff with jitter
  • Respects Retry-After headers

• Error classification implemented

  • Review openevidence-common-errors

6. Performance

• Response caching strategy

  • Cache frequently queried clinical information
  • Cache TTL appropriate for clinical data freshness

• Connection pooling configured

  • Reuse HTTP connections

• Query optimization reviewed

  • Clear, specific clinical questions
  • Appropriate context provided

• Load testing completed

  • Test at 2x expected peak traffic
  • Verify rate limits not exceeded

7. Documentation & Training

• Runbook created

  • Incident response procedures
  • Review openevidence-incident-runbook

• On-call training completed

  • Team knows how to triage OpenEvidence issues

• User documentation updated

  • How to use clinical decision support
  • Limitations and appropriate use

• Clinical staff training

  • Understanding AI-assisted decision support
  • When to rely on vs. question AI recommendations

8. Testing & Validation

• Integration tests passing

  • Against sandbox environment

• End-to-end tests passing

  • Full workflow validation

• Clinical validation completed

  • Sample queries reviewed by clinical team
  • Responses appropriate and accurate

• Performance benchmarks met

  • P50 < 5s, P95 < 15s for clinical queries
  • P50 < 3min, P95 < 5min for DeepConsult

9. Deployment

• Blue-green or canary deployment planned

  • Gradual rollout with monitoring

• Rollback procedure documented

  • Quick revert if issues detected

• Feature flags configured

  • Ability to disable OpenEvidence quickly

• Database migrations completed

  • Audit log tables, cache tables ready

10. Post-Launch

• Monitoring dashboard reviewed daily (first week)

• Error reports triaged immediately

• User feedback collection active

• Performance baseline established

Production Configuration Example

// config/​openevidence.production.ts
export const productionConfig = {
  // Credentials from secret manager
  credentials: {
    source: 'secret-manager',
    apiKeyPath: 'projects/​prod/​secrets/​openevidence-api-key',
    orgIdPath: 'projects/​prod/​secrets/​openevidence-org-id',
  },

  // Endpoints
  api: {
    baseUrl: 'https://api.openevidence.com',
    timeout: 30000,
    retries: 3,
  },

  // Rate limiting
  rateLimits: {
    enabled: true,
    maxConcurrent: 10,
    requestsPerMinute: 100,
  },

  // Circuit breaker
  circuitBreaker: {
    enabled: true,
    errorThreshold: 50,
    resetTimeout: 30000,
  },

  // Caching
  cache: {
    enabled: true,
    ttlSeconds: 3600,
    maxEntries: 10000,
  },

  // Monitoring
  monitoring: {
    metricsEnabled: true,
    tracingEnabled: true,
    logLevel: 'info',
  },
};

Output

• Completed production readiness checklist
• Configuration validated
• All stakeholders signed off
• Go-live approved

Resources

• OpenEvidence
• OpenEvidence Security

Next Steps

For version upgrades, see openevidence-upgrade-migration.