Master/Worker Pattern with Jest Testing

This challenge focuses on implementing a fundamental concurrency pattern: the master/worker model. You will create a system where a "master" process distributes tasks to multiple "worker" processes and collects their results. This pattern is crucial for efficient parallel processing and can significantly improve performance in CPU-bound or I/O-bound applications.

Problem Description

Your task is to build a simplified master/worker system in TypeScript and then write Jest tests to ensure its correct functionality.

The system should consist of:

1. Master: A main process responsible for creating worker processes, distributing tasks to them, and aggregating their results.
2. Worker: Child processes that receive tasks, perform a specific operation (e.g., a calculation), and send their results back to the master.

You will need to use Node.js's built-in child_process module to spawn worker processes. Communication between the master and workers will be handled via message passing (e.g., process.send and process.on('message')).

Key Requirements:

• The master should be able to spawn a configurable number of worker processes.
• Tasks should be distributed to workers in a round-robin fashion or by selecting an available worker.
• Workers should be able to receive task data, process it, and return a result.
• The master must collect all results from the workers.
• The system should handle cases where a worker might error.
• Your solution should be written in TypeScript.

Expected Behavior: The master process will take a list of tasks. It will distribute these tasks to available workers. Each worker will process its assigned task and send back a result. The master will wait until all tasks are completed and then return an aggregated list of results.

Edge Cases to Consider:

• What happens if there are more tasks than workers?
• What happens if there are fewer tasks than workers?
• What happens if a worker process crashes or throws an error during task processing?
• What if no tasks are provided?

Examples

Example 1: Simple Task Distribution

// In your main application file (e.g., master.ts)
import { spawn } from 'child_process';
import path from 'path';

interface Task {
    id: number;
    data: number;
}

interface Result {
    taskId: number;
    workerId: number;
    processedData: number;
    error?: string;
}

async function runMaster(tasks: Task[], numWorkers: number): Promise<Result[]> {
    // ... implementation details ...
    return []; // Placeholder
}

// --- Example Usage ---
const tasks = [
    { id: 1, data: 5 },
    { id: 2, data: 10 },
    { id: 3, data: 3 },
    { id: 4, data: 7 },
];
const numberOfWorkers = 2;

// Assuming runMaster is implemented correctly, the expected output would be:
// Example Output (order of results may vary):
// [
//   { taskId: 1, workerId: 0, processedData: 10 }, // Worker 0 processed task 1 (5 * 2)
//   { taskId: 2, workerId: 1, processedData: 20 }, // Worker 1 processed task 2 (10 * 2)
//   { taskId: 3, workerId: 0, processedData: 6 },  // Worker 0 processed task 3 (3 * 2)
//   { taskId: 4, workerId: 1, processedData: 14 }  // Worker 1 processed task 4 (7 * 2)
// ]

Example 2: Handling More Tasks than Workers

// ... (same interfaces as Example 1) ...

// --- Example Usage ---
const tasks = [
    { id: 1, data: 2 },
    { id: 2, data: 4 },
    { id: 3, data: 6 },
    { id: 4, data: 8 },
    { id: 5, data: 10 },
];
const numberOfWorkers = 2;

// Expected Output (order of results may vary):
// [
//   { taskId: 1, workerId: 0, processedData: 4 },
//   { taskId: 2, workerId: 1, processedData: 8 },
//   { taskId: 3, workerId: 0, processedData: 12 },
//   { taskId: 4, workerId: 1, processedData: 16 },
//   { taskId: 5, workerId: 0, processedData: 20 }
// ]

Example 3: Worker Error Handling

Suppose a worker encounters an error when processing a task.

// --- Example Usage ---
const tasks = [
    { id: 1, data: 10 },
    { id: 2, data: 'invalid_data' }, // This task will cause a worker to error
    { id: 3, data: 20 },
];
const numberOfWorkers = 2;

// Expected Output (order of results may vary, and one result will indicate an error):
// [
//   { taskId: 1, workerId: 0, processedData: 20 },
//   { taskId: 2, workerId: 1, error: "Invalid input data for processing" }, // Or a similar error message
//   { taskId: 3, workerId: 0, processedData: 40 }
// ]

Constraints

• The master and worker processes should be managed using Node.js child_process.fork for inter-process communication.
• Task processing in the worker should be a simple operation (e.g., multiplying the input data by 2). For error handling, simulate an error for a specific input.
• Your Jest tests should cover the cases described in the examples, including successful completion, handling a variable number of tasks, and worker errors.
• The maximum number of worker processes to spawn should not exceed 16.
• The number of tasks processed should not exceed 1000 for performance testing.

Notes

• You will need to create at least two files: one for the master logic and one for the worker logic.
• The worker script will need to listen for messages from the parent process and send messages back.
• Consider using Promises and async/await to manage the asynchronous nature of inter-process communication.
• For testing, you'll likely want to create a mock or simplified version of your worker script for unit tests, and then use actual fork calls in integration tests.
• Think about how to gracefully shut down worker processes when all tasks are done.