Jest Flaky Test Detection

In software development, flaky tests are a significant impediment to a smooth CI/CD pipeline and developer confidence. These are tests that sometimes pass and sometimes fail without any apparent changes to the code being tested. This challenge focuses on implementing a mechanism to detect such flaky tests within a Jest testing environment. Successfully identifying flaky tests will allow teams to prioritize their fixing, leading to more reliable test suites.

Problem Description

Your task is to create a Jest test runner that automatically identifies and flags tests that exhibit flaky behavior. A flaky test is defined as a test that has failed at least once and then passed in a subsequent run, or vice-versa, within a defined number of consecutive runs.

Key Requirements:

1. Test Execution: The runner must execute a given set of Jest tests multiple times.
2. State Tracking: It needs to keep track of the pass/fail status of each individual test across these runs.
3. Flakiness Detection: Implement logic to identify a test as flaky if its status changes between consecutive runs, and it has a history of both passing and failing.
4. Configuration: The number of consecutive runs to perform and a threshold for determining flakiness should be configurable.
5. Reporting: A clear report should be generated listing all identified flaky tests.

Expected Behavior:

• The runner should accept a Jest configuration and a list of test files.
• It should execute the specified tests for a configurable number of consecutive runs.
• For each test, it should record its outcome (passed or failed) in each run.
• A test is considered flaky if, within the configured number of runs, it has shown inconsistent behavior (e.g., passed in run X, failed in run X+1, or vice-versa). A more robust definition could be: a test is flaky if it has failed at least once and then passed, or passed at least once and then failed, within the observation window.
• The output should be a list of test names (or paths) that are deemed flaky.

Edge Cases to Consider:

• Tests that consistently fail.
• Tests that consistently pass.
• Tests that fail for the first time in the last run.
• Tests that pass for the first time in the last run.
• Handling of tests that are skipped or have errors other than assertion failures.

Examples

Let's assume we have a simple Jest test file (__tests__/my.test.ts) with two tests:

// __tests__/my.test.ts
describe('My Feature', () => {
  it('should pass consistently', () => {
    expect(true).toBe(true);
  });

  // This test will randomly fail
  it('should be flaky', () => {
    expect(Math.random() > 0.3).toBe(true); // Fails ~30% of the time
  });

  it('should fail consistently', () => {
    expect(false).toBe(true);
  });
});

Example 1:

Configuration:

• numberOfRuns: 10
• flakyThreshold: 2 (meaning a test is flaky if its status changes at least twice within the runs, or a simpler definition: if it has both passed and failed at least once)

Simulated Test Run Outcomes (for should be flaky): Run 1: Pass Run 2: Pass Run 3: Fail Run 4: Pass Run 5: Fail Run 6: Pass Run 7: Pass Run 8: Fail Run 9: Pass Run 10: Fail

Output:

Flaky Tests:
- __tests__/my.test.ts: My Feature should be flaky

Explanation: The should be flaky test has shown inconsistent behavior, passing and failing across multiple runs. The other tests (should pass consistently and should fail consistently) have stable outcomes throughout the 10 runs.

Example 2:

Configuration:

• numberOfRuns: 5
• flakyThreshold: 1 (meaning a test is flaky if its status changes even once within the runs)

Simulated Test Run Outcomes (for should be flaky): Run 1: Pass Run 2: Pass Run 3: Pass Run 4: Fail Run 5: Pass

Output:

Flaky Tests:
- __tests__/my.test.ts: My Feature should be flaky

Explanation: Even with just one failure after consistently passing, this test is flagged as potentially flaky because its status changed. This highlights a more sensitive detection.

Constraints

• The number of runs (numberOfRuns) will be an integer between 3 and 20.
• The flaky threshold (flakyThreshold) will be an integer between 1 and numberOfRuns - 1.
• The solution should be implemented in TypeScript.
• The solution should integrate with Jest's programmatic API, not just by running jest --runInBand --maxWorkers=1 multiple times externally.
• The focus is on test detection, not automatic test fixing.

Notes

• You will need to leverage Jest's programmatic API to execute tests multiple times and capture their results.
• Consider how to uniquely identify each test. Jest test titles combined with their describe blocks can serve this purpose.
• Think about how to store the state of each test across multiple runs.
• The "flaky threshold" can be interpreted in different ways. A simpler approach is to flag any test that has both passed and failed at least once within the observation window. A more complex approach might involve looking for specific patterns of state changes. For this challenge, a test is flaky if its outcome has changed at least once and it has a history of both passing and failing within the observation window.
• You may want to disable parallel test execution (--maxWorkers=1) to ensure consistent ordering of test runs and to simplify state management if Jest's internal worker management is complex to track individual test outcomes across runs.