Designing Production-Ready Microservice Dependency Health Checks

Modern microservices need more than just REST endpoints—they require deep, production-ready health checks to ensure reliability across complex dependency chains. With distributed architectures and third-party integrations growing, dependency health checks are now critical to uptime and fast incident resolution.

What Is a Dependency Health Check in Microservices?

A dependency health check is a programmatic endpoint or probe that actively checks the status of a microservice’s critical dependencies—such as databases, caches, message brokers, HTTP services, or external APIs. Unlike a simple liveness probe (which only checks if the microservice’s process responds), a dependency health check drills into the actual health of outbound connections and infrastructure.

Here’s an example of a production-grade health check endpoint in a Node.js microservice using express and mongodb (Node 20.x, Express 4.18, MongoDB Node Driver 5.x):

// health.js
const express = require('express');
const { MongoClient } = require('mongodb');

const router = express.Router();
const mongoUrl = process.env.MONGO_URL;

router.get('/healthz', async (req, res) => {
  try {
    // Deep dependency probe: ensure DB is reachable and a ping returns
    const client = new MongoClient(mongoUrl, { serverSelectionTimeoutMS: 1500 });
    await client.connect();
    await client.db('admin').command({ ping: 1 });
    await client.close();
    res.status(200).json({ status: 'ok', db: 'ok' });
  } catch (err) {
    res.status(503).json({ status: 'fail', db: 'unreachable', error: err.message });
  }
});

module.exports = router;

This example demonstrates a deep readiness probe, not just process health. It confirms the database is available, with fast timeouts and explicit error reporting.

Key insight: Shallow health checks miss real-world outages; deep probes catch failures before they impact end users.

Step 1: Identify All Critical Service Dependencies

Why You Must Map Dependencies Explicitly

Every microservice typically depends on multiple external resources: databases (like PostgreSQL 15 or MongoDB 6.0), caches (Redis 7.x), message queues (RabbitMQ 3.12), APIs, or cloud-managed services (AWS RDS, SQS, or Google Pub/Sub). Missing even one can lead to silent partial outages—services may appear healthy, but are unable to fulfill requests due to a broken dependency.

How to Inventory Dependencies

1. Catalog all outbound network connections: Use your service code, deployment manifests, and infrastructure-as-code to list every dependency your microservice needs to start up and serve traffic.
2. Classify dependencies as critical vs. optional: For example, authentication DBs are usually critical; analytics sinks may be optional.
3. Document connection details and health endpoints: Record connection strings, authentication methods, and expected health endpoints for each dependency.

Example dependency list:

• PostgreSQL 15 (primary DB) — critical
• Redis 7.0 (cache) — critical
• Stripe API (payments) — critical
• Elasticsearch 8 (search index) — optional

Key insight: Without a precise inventory, you can’t build comprehensive health checks or avoid cascading failures.

Step 2: Implement Deep Readiness and Liveness Probes

Liveness vs. Readiness: Why Both Are Essential

• Liveness probe: Answers “Is my process running?” Used by orchestrators like Kubernetes to restart crashed containers.
• Readiness probe: Answers “Can I actually serve requests now?”—including dependency status. Used to gate traffic routing.

In Kubernetes (v1.29+), a typical deployment manifest with both probes looks like this:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 5
  failureThreshold: 2

• /healthz endpoint: Returns 200 if the process is alive (no deep checks, fast response).
• /readyz endpoint: Returns 200 only if all critical dependencies (DB, cache, etc.) are reachable and healthy.

Implementing Deep Readiness Checks

• For each dependency, implement a fast, explicit check (e.g., DB ping, cache set/get, HTTP HEAD).
• Aggregate results; return HTTP 503 if any critical dependency fails.
• Use aggressive timeouts (<=1s per dependency) to avoid probe pile-ups.

Key insight: Production readiness checks must test real dependency health, not just process status.

Step 3: Automate Dependency Health Checks in CI/CD Pipelines

Why CI/CD Health Check Automation Matters

Automated dependency health checks catch integration failures before code reaches production. By running deep health checks during deployment, you prevent broken code from being promoted.

How to Integrate in CI/CD (GitHub Actions Example)

1. Spin up service and dependencies as containers: Use docker-compose with healthcheck configs.
2. Run health check scripts: Use HTTP requests to /readyz and /healthz endpoints with curl or httpie.
3. Fail build if any critical check returns non-200.

Example docker-compose.yml healthcheck section:

services:
  redis:
    image: redis:7.0
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 1s
      retries: 3
  app:
    build: .
    depends_on:
      redis:
        condition: service_healthy

Add this CI step in .github/workflows/ci.yml:

- name: Check readiness endpoint
  run: |
    for i in {1..10}; do
      http --check-status http://localhost:8080/readyz && exit 0
      sleep 2
    done
    exit 1

Key insight: Automated health checks in CI/CD block bad deployments—don’t wait for production to discover issues.

Step 4: Monitor and Alert on Dependency Health in Production

Why Real-Time Monitoring Is Non-Negotiable

Once deployed, your health endpoints become vital observability signals. Modern ops stacks like Prometheus (2.48+) and Grafana (10.x) can scrape and visualize health status. Alerting on status codes and response latency gives you early warning of dependency issues.

Setting Up Monitoring and Alerting

1. Export health status as metrics: Use a /metrics endpoint with Prometheus exposition format, or add custom labels to health check responses.
2. Scrape endpoints: Configure Prometheus to scrape /readyz every 10-30 seconds.
3. Alerting: Define alert rules for non-200 responses, high probe latency, and retries.

Example Prometheus alert rule:

- alert: ServiceDependencyDown
  expr: probe_success{job="my-service-readyz"} == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Dependency health check failed"
    description: "The /readyz endpoint is returning non-200 responses."

4. Log failures with context: Include dependency name, error message, and timestamps in logs for each failed check.

Key insight: Real-time dependency health monitoring enables teams to react before failures cascade to end users.

Step 5: Harden Health Check Endpoints for Security and Performance

Why Unprotected Health Endpoints Are a Risk

Health endpoints often leak sensitive infrastructure status. Exposing them publicly can aid attackers or leak incident details. Additionally, poorly implemented endpoints can DOS your dependencies if hit at scale by external monitors.

Best Practices for Secure, Efficient Health Checks

1. Restrict access: Use network policies, ingress rules, or API gateways (e.g., Kong 3.x, AWS API Gateway) to expose health endpoints only to trusted sources (internal load balancers, monitoring tools).
2. Rate limit probes: Apply rate limiting (e.g., Envoy, NGINX) to prevent health check endpoints from overwhelming dependencies.
3. Avoid expensive operations: Never run full-table scans or complex queries as part of health checks.
4. Redact error details: In non-internal environments, avoid leaking stack traces or connection strings in health check responses.

Example: Restricting health endpoints in NGINX Ingress for Kubernetes 1.29+

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
spec:
  rules:
    - http:
        paths:
          - path: /readyz
            backend:
              service:
                name: app-service
                port:
                  number: 8080

Key insight: Hardened health endpoints stay safe and fast—don’t expose internals or create new production risks.

Comparison Table: Health Check Tools and Patterns

Tool/Pattern	Deep Dependency Checks	Kubernetes Support	Security Features	Observability	Complexity
Custom HTTP Endpoints	Yes	Native	Low	High	Medium
Spring Boot Actuator 3.x	Yes (auto/extend)	Native	Medium	High	Low
ASP.NET Core HealthChecks	Yes (plug-ins)	Native	Medium	High	Low
Kubernetes Probes	Indirect (uses above)	Built-in	N/A	Low	Low
HashiCorp Consul 1.16	Yes (scriptable)	Integrates	Medium	High	Medium
AWS Elastic Beanstalk	Basic (process only)	No	Low	Low	Low

Custom HTTP endpoints (as in the Node.js and Python ecosystems) offer the most flexibility for deep dependency probing, but require explicit coding and security hardening.

Key insight: Choose health check tooling based on your stack’s dependency complexity, cloud environment, and security needs.

Frequently Asked Questions

Q: What’s the difference between a liveness probe and a readiness probe in Kubernetes? A: A liveness probe checks if the app process is running and restarts the container if it fails. A readiness probe checks if the app is ready to receive traffic (including healthy dependencies); only ready pods get traffic from Services.

Q: How often should I run dependency health checks in production? A: Most teams run readiness probes every 10-30 seconds. For critical, latency-sensitive services, you may tune this down to 5 seconds, but balance probe frequency against dependency load.

Q: Can health checks impact performance or cause outages? A: Yes, poorly designed health checks (e.g., expensive DB queries, unbounded retries) can overload dependencies or even trigger cascading failures. Always use fast, lightweight operations and aggressive timeouts.

Key Takeaways

• Inventory all critical and optional service dependencies for each microservice.
• Implement deep readiness checks that verify actual dependency health, not just process status.
• Integrate health check verification into CI/CD to catch failures before production.
• Monitor and alert on health check endpoints in production with tools like Prometheus and Grafana.
• Harden health endpoints with network restrictions, rate limiting, and redacted error responses.
• Choose health check tools and patterns based on your tech stack, security, and operational needs.