Implementing Robust Health Checks in a Go Microservice

In modern software development, especially in microservices architectures, ensuring the health and readiness of your services is paramount. Health checks are a fundamental mechanism for orchestration systems (like Kubernetes, Docker Swarm, or even simple load balancers) to monitor service availability, detect failures early, and facilitate graceful restarts or deployments. This challenge focuses on implementing these checks effectively in a Go application.

Problem Description

You are tasked with building a simple Go microservice that exposes health check endpoints. This service should provide different types of health checks to allow orchestrators to differentiate between a service that is simply running but not yet ready to serve traffic, and a service that is fully operational.

Key Requirements:

Liveness Probe: Implement an endpoint (e.g., /healthz) that indicates if the service is alive. This check should be fast and should only verify that the core process is running and hasn't crashed. It doesn't necessarily mean the service is ready to handle requests.
Readiness Probe: Implement an endpoint (e.g., /readyz) that indicates if the service is ready to serve traffic. This check should verify not only that the service is running but also that all its dependencies (like a database connection, external API availability) are healthy and operational.
Startup Probe (Optional but recommended): Implement an endpoint (e.g., /startupz) that indicates if the service has started successfully. This is particularly useful for services that have a non-trivial startup time (e.g., loading large datasets, establishing multiple connections). The orchestrator can use this to delay sending traffic until the service is fully initialized.
Response Format: All health check endpoints should return a JSON response with a status field. The status should be either "ok" or "error".
HTTP Status Codes:
- Liveness probe: Should return 200 OK if alive, 500 Internal Server Error otherwise.
- Readiness probe: Should return 200 OK if ready, 500 Internal Server Error otherwise.
- Startup probe: Should return 200 OK if started, 500 Internal Server Error otherwise.
Dependency Simulation: For the readiness and startup probes, you will need to simulate dependencies. For example, you might simulate a database connection that can be in a healthy or unhealthy state. You should provide a mechanism (e.g., a configuration flag or another endpoint) to toggle the health of these simulated dependencies.

Expected Behavior:

When the service starts, the /healthz endpoint should immediately return "ok".
The /readyz and /startupz endpoints might initially return "error" until their simulated dependencies are healthy.
An external mechanism should be able to change the state of the simulated dependency, causing /readyz and /startupz to transition from "error" to "ok".

Edge Cases:

What happens if a dependency becomes unhealthy after the service has been declared ready? The /readyz endpoint should reflect this change.
What if the service encounters a critical error during startup that prevents it from ever becoming ready? The /startupz endpoint should continue to indicate an error.

Examples

Example 1: Initial State (Service Running, Dependencies Not Ready)

Request: GET http://localhost:8080/healthz
Response:
```
{
  "status": "ok"
}
```
HTTP Status Code: 200 OK
Request: GET http://localhost:8080/readyz
Response:
```
{
  "status": "error"
}
```
HTTP Status Code: 500 Internal Server Error
Request: GET http://localhost:8080/startupz
Response:
```
{
  "status": "error"
}
```
HTTP Status Code: 500 Internal Server Error

Example 2: Dependencies Healthy

Assume a mechanism (e.g., calling an internal function or an endpoint) has been used to make the simulated dependency healthy.

Request: GET http://localhost:8080/healthz
Response:
```
{
  "status": "ok"
}
```
HTTP Status Code: 200 OK
Request: GET http://localhost:8080/readyz
Response:
```
{
  "status": "ok"
}
```
HTTP Status Code: 200 OK
Request: GET http://localhost:8080/startupz
Response:
```
{
  "status": "ok"
}
```
HTTP Status Code: 200 OK

Example 3: Dependency Becomes Unhealthy After Being Ready

Assume the service was ready, but a simulated dependency suddenly fails.

Request: GET http://localhost:8080/healthz
Response:
```
{
  "status": "ok"
}
```
HTTP Status Code: 200 OK
Request: GET http://localhost:8080/readyz
Response:
```
{
  "status": "error"
}
```
HTTP Status Code: 500 Internal Server Error

Constraints

The web server should listen on port 8080.
The service should be implemented using Go's standard library (net/http) or a popular lightweight framework like gin-gonic or echo.
The simulated dependency should be controllable via a simple mechanism (e.g., a global variable, a small internal HTTP endpoint, or a command-line flag).
The health check endpoints should be idempotent.
Responses should be handled efficiently, without significant latency.

Notes

Consider using Go's sync package for managing the state of your simulated dependencies, especially if multiple goroutines might access it.
Think about how you would structure your code to easily add more complex health checks in the future (e.g., checking a database connection pool, validating API keys, etc.).
For readiness and startup probes, the check should ideally be asynchronous if the underlying dependency check is slow, to avoid blocking the HTTP request handler. However, for this challenge, a synchronous check is acceptable.
The primary goal is to demonstrate understanding of different health check types and their purpose in a microservice context.