healthz sidecar — Image & Manifests

Build artefacts for the healthz Kubernetes sidecar (Python 3.13 / FastAPI). Exposes per-container livenessProbe / readinessProbe endpoints that observe a sibling main container's process, TCP socket, and HTTP endpoint, and reports detailed self/target/system snapshots on demand.

Two deliverables:

1. Container image — Dockerfile + requirements.txt + healthz/app/
2. Kubernetes manifests — manifests/samples/*.yaml (raw / kustomize) and manifests/helm/ (Helm chart)

Layout

.
├── Dockerfile                     # multi-stage Python 3.13-slim build
├── .dockerignore
├── .gitignore
├── requirements.txt               # pinned Python deps
├── README.md
├── healthz/                       # source root (CWD for `python -m app.main`)
│   └── app/                       # Python package (entrypoint: app.main)
│       ├── __init__.py
│       ├── main.py                # entry point
│       ├── factory.py             # FastAPI app factories (was app.py)
│       ├── state.py
│       ├── process_identifier.py
│       ├── checks/                # tcp / http / process / scheduler
│       ├── collectors/            # detail collectors (system/network/process/...)
│       ├── config/                # YAML loader + schema
│       ├── endpoints/             # FastAPI route handlers
│       └── utils/                 # logging / masking / proc / time helpers
├── manifests/
│   ├── samples/                   # raw manifests (kustomize)
│   │   ├── 00-namespace.yaml
│   │   ├── 10-rbac.yaml
│   │   ├── 20-configmap.yaml
│   │   ├── 25-alloy-config.yaml
│   │   ├── 30-statefulset.yaml
│   │   ├── 40-services.yaml
│   │   ├── 50-networkpolicy.yaml
│   │   ├── 60-pdb.yaml
│   │   ├── healthz_config.yaml    # local-dev config (mirrors ConfigMap.data)
│   │   └── kustomization.yaml
│   └── helm/                      # Helm chart (NetworkPolicy excluded)
│       ├── Chart.yaml
│       ├── values.yaml
│       ├── .helmignore
│       └── templates/
└── scripts/
    └── validate.py                # manifest validator (PyYAML only)

Build the image

podman build \
  --build-arg GIT_SHA="$(git rev-parse --short HEAD 2>/dev/null || echo dev)" \
  --build-arg BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -t myregistry.example.internal/healthz:1.0.0 .

podman push myregistry.example.internal/healthz:1.0.0

The base image is pinned via the BASE_IMAGE build arg (default python:3.13-slim-bookworm). For RHEL provenance, override:

podman build --build-arg BASE_IMAGE=registry.access.redhat.com/ubi9/python-312:latest ...

Verify availability of UBI image tags at https://catalog.redhat.com/.

Run locally (development)

# from project root, source layout
cd healthz
python -m app.main --config ../manifests/samples/healthz_config.yaml

Or via container:

podman run --rm \
  -p 9001:9001 -p 9000:9000 \
  -v $PWD/manifests/samples/healthz_config.yaml:/etc/healthz/config.yaml:ro \
  myregistry.example.internal/healthz:1.0.0

# in another shell
curl -s http://127.0.0.1:9001/health           | jq
curl -s http://127.0.0.1:9001/health/detail    | jq '.self.os_info.kernel'
curl -s http://127.0.0.1:9000/healthz          | jq

The TCP/HTTP/process checks will of course report FAIL because there is no nifi-app inside the standalone container. /health (self) should return 200 once startup completes.

Deploy with raw manifests (kustomize)

Update the image reference in manifests/samples/30-statefulset.yaml, then:

kubectl apply -k manifests/samples/

# Wait for pod to be ready
kubectl -n data rollout status statefulset/nifi --timeout=5m

# Verify probes
kubectl -n data exec nifi-0 -c nifi-app -- \
  curl -s http://127.0.0.1:9001/health | jq

kubectl -n data exec nifi-0 -c nifi-app -- \
  curl -s http://127.0.0.1:9000/healthz/detail | jq '.target.checks'

Deploy with Helm

helm install nifi-healthz manifests/helm/ \
  --namespace data --create-namespace \
  --set image.repository=myregistry.example.internal/healthz \
  --set image.tag=1.0.0

values.yaml exposes image/replica/resources/probes/target-config knobs. NetworkPolicy is intentionally not part of this chart (manage with your cluster-wide CNP/NP framework). Service type defaults to ClusterIP.

Validate manifests pre-deploy

python scripts/validate.py

Performs:

• A. YAML well-formedness for every doc under manifests/samples/
• B. ConfigMap → loader round-trip (app.config.loader.load_config) — catches coercion errors at apply time, not first-pod-up
• C. Cross-document consistency (volumes ↔ volumeMounts, Service / PDB / NetworkPolicy selectors ↔ pod labels, ConfigMap references, named ports)
• D. Pod-spec invariants (shareProcessNamespace=true, healthz declared as native sidecar, nifi-app probes use NUMERIC port 9000)

Optional schema validation against the K8s OpenAPI:

kubeconform -strict -summary -kubernetes-version 1.33.0 -schema-location default \
  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  manifests/samples/

ConfigMap drift

The healthz loader records the SHA-256 of the config it loaded at startup. Subsequent ConfigMap edits surface in /healthz/detail under config_info.drift_detected: true. Apply changes by recreating the pod:

kubectl -n data rollout restart statefulset/nifi

K8s version requirements

• Kubernetes ≥ 1.29 for native sidecars (initContainer + restartPolicy: Always). The feature gate SidecarContainers is beta default-on in 1.29 and stable (GA) in 1.33. Reference: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/

• shareProcessNamespace: true is required so healthz can /proc-scan the target's PIDs. Reference: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/

• CAP_SYS_PTRACE is required to read /proc/<target_pid>/{environ, root, io, ...} when the target runs as a different UID. This capability is NOT in the Pod Security Standards baseline allow list, which is why the namespace is labelled pod-security.kubernetes.io/enforce: privileged. Reference: https://kubernetes.io/docs/concepts/security/pod-security-standards/

Endpoints

Port	Path	Purpose
9001	/health	healthz self liveness (sidecar's own kubelet probe)
9001	/health/detail	healthz self detailed info
9000	/healthz	target (nifi-app) liveness/readiness — kubelet probes nifi via this path
9000	/healthz/detail	target detailed info
{.dense}

Operational defaults

Knob	Default
TCP/HTTP/process check interval	10 s
TCP/HTTP timeout	5 s
failure_threshold	3 cycles → DOWN
Drain on SIGTERM	15 s + preStop sleep 5 s
terminationGracePeriodSeconds	60 s
Probe latency window (deque)	1 000 samples
recent_failures buffer	20
{.dense}

Sources

• Kubernetes Sidecar Containers (GA 1.33): https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/
• shareProcessNamespace task: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/
• Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/
• Pod Security Admission: https://kubernetes.io/docs/concepts/security/pod-security-admission/
• Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
• Helm chart best practices: https://helm.sh/docs/chart_best_practices/
• CIS Kubernetes Benchmark 5.1.5 (SA token automounting): https://www.cisecurity.org/benchmark/kubernetes
• UBI Python image catalog: https://catalog.redhat.com/software/containers/search?q=python
• Python 3.13 release notes: https://docs.python.org/3.13/whatsnew/3.13.html
• FastAPI: https://fastapi.tiangolo.com/
• Grafana Alloy components: https://grafana.com/docs/alloy/latest/reference/components/