FireCrawl Data Handling

FireCrawl Data Handling

Overview

Handle sensitive data correctly when integrating with FireCrawl.

Prerequisites

• Understanding of GDPR/​CCPA requirements
• FireCrawl SDK with data export capabilities
• Database for audit logging
• Scheduled job infrastructure for cleanup

Data Classification

PII Detection

const PII_PATTERNS = [
  { type: 'email', regex: /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/​g },
  { type: 'phone', regex: /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/​g },
  { type: 'ssn', regex: /\b\d{3}-\d{2}-\d{4}\b/​g },
  { type: 'credit_card', regex: /\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/​g },
];

function detectPII(text: string): { type: string; match: string }[] {
  const findings: { type: string; match: string }[] = [];

  for (const pattern of PII_PATTERNS) {
    const matches = text.matchAll(pattern.regex);
    for (const match of matches) {
      findings.push({ type: pattern.type, match: match[0] });
    }
  }

  return findings;
}

Data Redaction

function redactPII(data: Record<string, any>): Record<string, any> {
  const sensitiveFields = ['email', 'phone', 'ssn', 'password', 'apiKey'];
  const redacted = { ...data };

  for (const field of sensitiveFields) {
    if (redacted[field]) {
      redacted[field] = '[REDACTED]';
    }
  }

  return redacted;
}

// Use in logging
console.log('FireCrawl request:', redactPII(requestData));

Data Retention Policy

Retention Periods

Automatic Cleanup

async function cleanupFireCrawlData(retentionDays: number): Promise<void> {
  const cutoff = new Date();
  cutoff.setDate(cutoff.getDate() - retentionDays);

  await db.firecrawlLogs.deleteMany({
    createdAt: { $lt: cutoff },
    type: { $nin: ['audit', 'compliance'] },
  });
}

// Schedule daily cleanup
cron.schedule('0 3 * * *', () => cleanupFireCrawlData(30));

GDPR/​CCPA Compliance

Data Subject Access Request (DSAR)

async function exportUserData(userId: string): Promise<DataExport> {
  const firecrawlData = await firecrawlClient.getUserData(userId);

  return {
    source: 'FireCrawl',
    exportedAt: new Date().toISOString(),
    data: {
      profile: firecrawlData.profile,
      activities: firecrawlData.activities,
      // Include all user-related data
    },
  };
}

Right to Deletion

async function deleteUserData(userId: string): Promise<DeletionResult> {
  // 1. Delete from FireCrawl
  await firecrawlClient.deleteUser(userId);

  // 2. Delete local copies
  await db.firecrawlUserCache.deleteMany({ userId });

  // 3. Audit log (required to keep)
  await auditLog.record({
    action: 'GDPR_DELETION',
    userId,
    service: 'firecrawl',
    timestamp: new Date(),
  });

  return { success: true, deletedAt: new Date() };
}

Data Minimization

// Only request needed fields
const user = await firecrawlClient.getUser(userId, {
  fields: ['id', 'name'], // Not email, phone, address
});

// Don't store unnecessary data
const cacheData = {
  id: user.id,
  name: user.name,
  // Omit sensitive fields
};

Instructions

Step 1: Classify Data

Categorize all FireCrawl data by sensitivity level.

Step 2: Implement PII Detection

Add regex patterns to detect sensitive data in logs.

Step 3: Configure Redaction

Apply redaction to sensitive fields before logging.

Step 4: Set Up Retention

Configure automatic cleanup with appropriate retention periods.

Output

• Data classification documented
• PII detection implemented
• Redaction in logging active
• Retention policy enforced

Error Handling

Examples

Quick PII Scan

const findings = detectPII(JSON.stringify(userData));
if (findings.length > 0) {
  console.warn(`PII detected: ${findings.map(f => f.type).join(', ')}`);
}

Redact Before Logging

const safeData = redactPII(apiResponse);
logger.info('FireCrawl response:', safeData);

GDPR Data Export

const userExport = await exportUserData('user-123');
await sendToUser(userExport);

Resources

• GDPR Developer Guide
• CCPA Compliance Guide
• FireCrawl Privacy Guide

Next Steps

For enterprise access control, see firecrawl-enterprise-rbac.