Structured Logging with Python's logging Module

Modern applications generate a significant amount of log data. While traditional plain-text logs are helpful, they can be difficult to parse and analyze programmatically. Structured logging, where log messages are formatted as key-value pairs (often in JSON), makes logs machine-readable and significantly improves the ability to filter, search, and aggregate log data. This challenge will guide you to implement structured logging in your Python application.

Problem Description

Your task is to create a Python logging system that outputs logs in a structured, JSON-like format. This will involve configuring the standard Python logging module to emit messages with predefined fields such as timestamp, log level, module name, and a custom message payload. You should be able to pass arbitrary data as part of the log message, which will be included in the structured output.

Key Requirements:

• Structured Output: Log messages must be formatted as JSON strings.
• Standard Fields: Each log entry must include:
  • timestamp: ISO 8601 formatted timestamp of when the log was generated.
  • level: The logging level (e.g., INFO, WARNING, ERROR).
  • module: The name of the Python module where the log originated.
  • message: The main log message string.
• Custom Data: The logger should support passing additional arbitrary key-value data with each log message. This data should be merged into the JSON output.
• Configuration: The logging setup should be configurable.

Expected Behavior:

When you log a message, the output to the console (or wherever the handler is directed) should be a single line representing a JSON object containing the standard fields and any custom data provided.

Edge Cases:

• Handling of missing custom data.
• Logging at different severity levels.

Examples

Example 1:

import logging
from datetime import datetime

# Assuming logging is configured to output structured logs

logging.basicConfig(level=logging.INFO) # Placeholder, actual configuration will be done in the solution

# --- Simulate logging with custom data ---
extra_data = {"user_id": 123, "request_id": "abc-xyz"}
logging.info("User logged in successfully.", extra=extra_data)

Output:

{"timestamp": "2023-10-27T10:30:00.123456", "level": "INFO", "module": "__main__", "message": "User logged in successfully.", "user_id": 123, "request_id": "abc-xyz"}

Explanation: The logging.info call was made with a message and extra data. The structured logger formats this into a JSON string, including the standard fields and the custom user_id and request_id. The timestamp will reflect the actual time of execution.

Example 2:

import logging
from datetime import datetime

# Assuming logging is configured to output structured logs

logging.basicConfig(level=logging.WARNING) # Placeholder

# --- Simulate logging without custom data ---
logging.warning("Database connection timed out.")

Output:

{"timestamp": "2023-10-27T10:35:15.987654", "level": "WARNING", "module": "__main__", "message": "Database connection timed out."}

Explanation: A warning message is logged without any additional custom data. The output JSON includes only the standard fields.

Example 3: (Edge case: Logging an error with complex data)

import logging
from datetime import datetime

# Assuming logging is configured to output structured logs

logging.basicConfig(level=logging.ERROR) # Placeholder

# --- Simulate logging an error with complex data ---
error_details = {
    "error_code": 500,
    "details": {"type": "DatabaseError", "message": "Connection refused"}
}
logging.error("An internal server error occurred.", extra={"request_info": {"url": "/api/users", "method": "POST"}, "error_data": error_details})

Output:

{"timestamp": "2023-10-27T10:40:40.555666", "level": "ERROR", "module": "__main__", "message": "An internal server error occurred.", "request_info": {"url": "/api/users", "method": "POST"}, "error_data": {"error_code": 500, "details": {"type": "DatabaseError", "message": "Connection refused"}}}

Explanation: An error is logged with nested custom data. The structured logger correctly serializes the nested dictionaries into the JSON output.

Constraints

• Python Version: Solution must be compatible with Python 3.7+.
• Standard Library Only: You should primarily use Python's built-in logging module and standard library modules (like json and datetime). Avoid external logging libraries like loguru for this challenge.
• Single Line Output: Each log message must be output as a single, valid JSON line.
• Performance: The logging setup should not introduce significant performance overhead for typical application workloads.

Notes

• The logging.basicConfig() function is a convenient way to set up a basic logger, but for more control over handlers and formatters, you'll need to create Logger, Handler, and Formatter objects explicitly.
• Consider how to format the timestamp to be ISO 8601 compliant.
• The extra parameter of logging methods is key to passing custom data. You'll need to ensure this data is merged correctly into your structured output.
• Think about how to handle potential non-serializable data if it were to be passed in extra. For this challenge, assume basic JSON-serializable data types.