Python Configuration Manager

Many applications require external configuration to define their behavior, such as database credentials, API keys, or feature flags. This challenge asks you to build a robust and flexible configuration management system in Python that can load and access these settings from various sources.

Problem Description

Your task is to implement a Python class, ConfigManager, that can load configuration settings from different file formats and provide a unified interface for accessing these settings. The system should be able to:

1. Load configurations from multiple sources: Initially, support loading from JSON and YAML files.
2. Handle hierarchical configurations: Allow nested settings (e.g., database.host, api.keys.primary).
3. Provide a clear access interface: Enable easy retrieval of configuration values using dot notation or dictionary-like access.
4. Merge configurations: If multiple configuration files are loaded, the settings from later files should override those from earlier files for duplicate keys.
5. Support default values: Allow specifying default values that are used if a configuration key is not found in any loaded file.
6. Error handling: Gracefully handle cases where configuration files are missing, malformed, or if a requested setting does not exist and has no default.

Key Requirements

• The ConfigManager class should have a constructor that accepts an optional list of configuration file paths.
• It should have a method to load configurations from a given file path.
• It should provide a way to access configuration values.
• It should handle merging configurations from multiple files.
• It should support default values.

Expected Behavior

• When a configuration file is loaded, its contents should be parsed and added to the manager's internal state.
• Accessing a configuration value should return the value associated with the given key, respecting the merging order and defaults.
• If a key is not found and no default is provided, an appropriate error should be raised.

Edge Cases to Consider

• Empty configuration files.
• Configuration files with invalid JSON or YAML syntax.
• Non-existent configuration files.
• Accessing keys that do not exist and have no defaults.
• Deeply nested configuration structures.
• Overlapping keys across multiple configuration files.

Examples

Example 1: Basic JSON Loading and Access

Input Files: config.json:

{
  "database": {
    "host": "localhost",
    "port": 5432
  },
  "api_key": "12345abc"
}

Python Usage:

manager = ConfigManager(["config.json"])
db_host = manager.get("database.host")
api_key = manager.get("api_key")

Output: db_host should be "localhost" api_key should be "12345abc"

Explanation: The ConfigManager loads the JSON file. The get method successfully retrieves values using dot notation.

Example 2: YAML Loading, Merging, and Defaults

Input Files: defaults.yaml:

logging:
  level: INFO
  format: "%(asctime)s - %(levelname)s - %(message)s"
timeout: 30

override.yaml:

logging:
  level: DEBUG
timeout: 60
feature_flags:
  new_dashboard: true

Python Usage:

manager = ConfigManager(["defaults.yaml", "override.yaml"])
log_level = manager.get("logging.level")
timeout = manager.get("timeout")
feature_enabled = manager.get("feature_flags.new_dashboard", default=False)
non_existent = manager.get("database.host", default="127.0.0.1")

Output: log_level should be "DEBUG" timeout should be 60 feature_enabled should be True non_existent should be "127.0.0.1"

Explanation: defaults.yaml is loaded first, then override.yaml. Settings in override.yaml (like logging.level and timeout) overwrite those from defaults.yaml. The feature_flags.new_dashboard is present in override.yaml. non_existent uses the provided default value.

Example 3: Handling Missing Keys and Errors

Input Files: settings.json:

{
  "user": "admin"
}

Python Usage:

manager = ConfigManager(["settings.json"])

# Accessing a key without a default
try:
    manager.get("database.port")
except KeyError as e:
    print(f"Error: {e}")

# Accessing a key that exists but is None (if supported by parser)
# Assuming a valid parser treats null as None
# file_with_null.json: {"nullable_setting": null}
# manager_null = ConfigManager(["file_with_null.json"])
# print(manager_null.get("nullable_setting")) # Should print None

Output: Error: 'database.port'

Explanation: When manager.get("database.port") is called, the key is not found in settings.json and no default is provided. A KeyError is raised, as expected.

Constraints

• The ConfigManager must support loading from .json and .yaml (or .yml) files.
• The maximum depth of nested configurations is not strictly limited, but efficient handling of deeply nested structures is encouraged.
• The number of configuration files to load at initialization should not exceed 100.
• The total size of all configuration files combined should not exceed 1MB.
• The solution should be implemented in Python 3.8 or later.
• External libraries like PyYAML and Python's built-in json module are permitted.

Notes

• Consider how you will parse different file formats. You will likely need to import libraries such as json and PyYAML.
• The get method could potentially accept a list of keys for deeper nesting or a single string with dot notation.
• Think about how to efficiently merge dictionaries, especially with nested structures. Python's dictionary update method might be useful, but you'll need a recursive approach for nested dictionaries.
• For accessing nested keys like database.host, you'll need to split the key string and traverse the configuration dictionary.
• When handling missing keys, returning None might be an option for get if no default is provided, or raising a KeyError is also a common and acceptable pattern. The examples suggest KeyError is preferred.