Flaky Fix Skill

Flaky Fix Skill

Purpose

Analyze flaky test patterns and suggest or auto-apply fixes. Based on FlakyFix research showing LLMs can automatically repair flaky tests with targeted prompts.

Research Foundation

When This Skill Applies

• After flaky-detect identifies flaky tests
• User asks to "fix flaky test" or "make test reliable"
• CI is failing intermittently on specific tests
• Test marked as flaky needs repair

Trigger Phrases

Fix Patterns by Category

1. Timing Issues (45% of flaky tests)

Problem: Uses Real Time

// FLAKY: Time-dependent
it('should expire after 1 hour', () => {
  const token = createToken();
  expect(token.expiresAt).toBeGreaterThan(Date.now());
});

Fix: Mock Time

// FIXED: Mocked time
it('should expire after 1 hour', () => {
  const fixedTime = new Date('2024-01-01T00:00:00Z');
  vi.setSystemTime(fixedTime);

  const token = createToken();

  expect(token.expiresAt).toBe(fixedTime.getTime() + 3600000);
  vi.useRealTimers();
});

Problem: Explicit Sleep/​Delay

// FLAKY: Arbitrary delay
it('should complete async operation', async () => {
  startAsyncOperation();
  await sleep(100);  // Race condition!
  expect(result).toBeDefined();
});

Fix: Proper Async Handling

// FIXED: Wait for actual completion
it('should complete async operation', async () => {
  const result = await startAsyncOperation();
  expect(result).toBeDefined();
});

// Or use waitFor for DOM
it('should show loading state', async () => {
  render(<Component />);
  await waitFor(() => {
    expect(screen.getByText('Loaded')).toBeInTheDocument();
  });
});

2. Async Issues (25% of flaky tests)

Problem: Missing Await

// FLAKY: Promise not awaited
it('should fetch data', () => {
  const promise = fetchData();
  promise.then(data => {
    expect(data).toBeDefined();  // May not run before test ends
  });
});

Fix: Proper Async/​Await

// FIXED: Awaited promise
it('should fetch data', async () => {
  const data = await fetchData();
  expect(data).toBeDefined();
});

Problem: Race Condition

// FLAKY: Order not guaranteed
it('should process items', async () => {
  const results = [];
  items.forEach(async item => {
    results.push(await process(item));
  });
  expect(results.length).toBe(3);  // Race!
});

Fix: Sequential or Parallel Await

// FIXED: Guaranteed order
it('should process items', async () => {
  const results = await Promise.all(
    items.map(item => process(item))
  );
  expect(results.length).toBe(3);
});

3. Test Order Dependencies (20% of flaky tests)

Problem: Shared State

// FLAKY: Shared state between tests
let counter = 0;

it('should increment', () => {
  counter++;
  expect(counter).toBe(1);  // Fails if other test runs first
});

Fix: Test Isolation

// FIXED: Isolated state
describe('counter', () => {
  let counter;

  beforeEach(() => {
    counter = 0;  // Fresh state each test
  });

  it('should increment', () => {
    counter++;
    expect(counter).toBe(1);
  });
});

4. Non-deterministic Values (10% of flaky tests)

Problem: Random/​UUID Values

// FLAKY: Random ID
it('should create user with ID', () => {
  const user = createUser();
  expect(user.id).toBe('expected-id');  // Random!
});

Fix: Mock Random Generation

// FIXED: Deterministic ID
it('should create user with ID', () => {
  vi.mock('uuid', () => ({
    v4: () => 'test-uuid-1234'
  }));

  const user = createUser();
  expect(user.id).toBe('test-uuid-1234');
});

5. Environment Dependencies (15% of flaky tests)

Problem: Network Calls

// FLAKY: Real network
it('should fetch from API', async () => {
  const data = await fetch('https://api.example.com/​data');
  expect(data).toBeDefined();  // Network failures!
});

Fix: Mock Network

// FIXED: Mocked network
it('should fetch from API', async () => {
  vi.mock('node-fetch', () => ({
    default: vi.fn().mockResolvedValue({
      json: () => ({ success: true })
    })
  }));

  const data = await fetchFromApi();
  expect(data.success).toBe(true);
});

Auto-Fix Rules

Safe to Auto-Fix (Apply Automatically)

Requires Review (Suggest Only)

Output Format

## Flaky Test Fix Report

### Test: `test/​api/​login.test.ts:45`

**Root Cause**: Timing - uses `Date.now()` in assertion
**Confidence**: HIGH (pattern match)
**Auto-fixable**: YES

#### Original Code
```typescript
it('should create token with expiry', () => {
  const token = createToken();
  expect(token.expiresAt).toBeGreaterThan(Date.now());
});

Suggested Fix

it('should create token with expiry', () => {
  const now = new Date('2024-01-01T12:00:00Z');
  vi.setSystemTime(now);

  const token = createToken();

  expect(token.expiresAt).toBe(now.getTime() + TOKEN_LIFETIME);
  vi.useRealTimers();
});

Changes Summary

• Added: vi.setSystemTime() for deterministic time
• Added: vi.useRealTimers() cleanup
• Changed: Assertion to exact value match

Verification

Run 10x to confirm fix:

for i in {1..10}; do npm test -- test/​api/​login.test.ts:45; done

Batch Fix Summary

Auto-fixed: 2 tests Manual review: 2 tests Estimated stability improvement: +1.5%

## Integration Points

- Works with `flaky-detect` for test identification
- Reports to Test Engineer for complex fixes
- Feeds into CI stability metrics
- Updates `.aiwg/​testing/​flaky-fixes.md`

## Script Reference

### flaky_fixer.py
Analyze and fix flaky tests:
```bash
python scripts/​flaky_fixer.py --test test/​api/​login.test.ts --auto-fix

batch_fix.py

Fix multiple flaky tests:

python scripts/​batch_fix.py --input flaky-report.json --safe-only