Go Log Aggregator Service

Build a simple log aggregation service in Go. This service will listen for incoming log messages on a defined port, store them, and allow clients to query for logs based on certain criteria. This is a fundamental building block for many distributed systems, enabling centralized monitoring and debugging.

Problem Description

Your task is to create a Go application that acts as a log aggregator. The application should:

1. Listen for Incoming Logs: Accept incoming log messages via a network interface (e.g., UDP or TCP). Each log message will be a simple string.
2. Store Logs: Store the received log messages in memory. For simplicity, we won't require persistent storage for this challenge.
3. Provide a Query Interface: Expose an API endpoint that allows clients to query for stored logs. The query should support filtering by a specific keyword.
4. Handle Multiple Clients: The service should be able to handle multiple log producers and multiple query clients concurrently.

Key Requirements:

• Network Protocol: Choose a suitable network protocol (UDP is suggested for simplicity and fire-and-forget nature of logs, but TCP is also acceptable).
• Data Structure: Decide on an appropriate in-memory data structure to store the logs.
• API Design: Design a simple API for querying logs. A RESTful approach is common, but not strictly required.
• Concurrency: Ensure the service can handle concurrent incoming logs and query requests without data corruption or blocking issues.

Expected Behavior:

• When a log message is received, it should be added to the internal storage.
• When a query is made for a keyword, the service should return all stored logs that contain that keyword.
• If no logs match the keyword, an empty result set should be returned.

Edge Cases:

• Empty Log Messages: How should the service handle receiving empty log strings? (Ideally, ignore or log them).
• No Matching Logs: What happens when a query is made for a keyword that doesn't exist in any logs?
• Concurrent Access: Ensure data consistency when multiple goroutines are writing and reading logs simultaneously.

Examples

Example 1: Basic Logging and Querying

• Scenario:

  • The log aggregator is running on port 8080.
  • A log producer sends the following logs to port 8080:
    • "INFO: User logged in successfully."
    • "ERROR: Database connection failed."
    • "INFO: Processing request #123."
  • A client then queries for logs containing the keyword "INFO".

• Input (Logs to port 8080):

  INFO: User logged in successfully.
  ERROR: Database connection failed.
  INFO: Processing request #123.
  

• Query Request (e.g., HTTP GET to /logs?keyword=INFO)

• Output (Logs matching "INFO"):

  [
    "INFO: User logged in successfully.",
    "INFO: Processing request #123."
  ]
  

• Explanation: The aggregator receives three log messages. The query for "INFO" returns the two messages that contain this string. The "ERROR" message is excluded.

Example 2: Querying for a Non-existent Keyword

• Scenario:

  • The log aggregator has received the following logs:
    • "DEBUG: Starting initialization."
    • "WARN: Configuration value not found."
  • A client queries for logs containing the keyword "FATAL".

• Input (Logs previously received):

  DEBUG: Starting initialization.
  WARN: Configuration value not found.
  

• Query Request (e.g., HTTP GET to /logs?keyword=FATAL)

• Output:

  []
  

• Explanation: No logs contain the keyword "FATAL", so an empty array is returned.

Example 3: Handling Empty or Different Protocols

• Scenario:

  • A log producer sends an empty string and then a valid log.
  • A client queries for logs containing "system".

• Input (Logs to port 8080 - assuming UDP):

  (empty packet)
  "INFO: System rebooting in 5 minutes."
  

• Query Request (e.g., HTTP GET to /logs?keyword=system)

• Output:

  [
    "INFO: System rebooting in 5 minutes."
  ]
  

• Explanation: The empty log message is ignored. The query for "system" correctly retrieves the relevant log.

Constraints

• The log aggregator should listen on a configurable UDP port (default to 8080).
• The query API should be accessible via HTTP on a different port (e.g., 8081).
• The in-memory storage should be able to hold at least 10,000 log entries.
• The service must handle at least 100 concurrent log producers.
• The query API should respond within 200ms for a dataset of up to 10,000 logs.

Notes

• Consider using Go's built-in net package for network communication and sync package for concurrency primitives.
• For the query API, you can use Go's net/http package.
• Think about how to make your log storage thread-safe.
• You can use libraries like encoding/json for API responses.
• For this challenge, error handling can be basic (e.g., logging errors and continuing).
• A good starting point for the query API could be a simple GET request like /logs?keyword=<your_keyword>.