# Blameless Postmortem: 2026-07-15 Kafka file-download consumer-group replay and Nearmap throttling Incident: `INC-2026-07150002` Status: **Monitoring / Remediation Pending** ## What Happened A production rollout changed kafka-file-download from the established FILE-DOWNLOAD-group consumer identity to FILE-DOWNLOAD-rmuqop-group while retaining fromBeginning:true. The new group had no committed offsets and replayed retained historical download requests during a 66-second old/new rolling overlap. Duplicate traffic caused Nearmap HTTP 429 throttling, file-download DLQ growth, and customer-facing iframe design-generation failures. Production was rolled back to version 2.0.0 and is currently healthy, but permanent remediation, exact impact enumeration, and recovery of failed jobs remain pending. ## What Went Well - Production was rolled back to the original image and offset lineage. - The investigator kept Kafka inspection read-only and did not persist payloads or signed URLs. - The failure mechanics and bad-release outcome counts were reconstructed with precise timing. - The current production deployment returned to 3/3 Ready with a clean 15-minute log sample. ## What Went Poorly - Consumer-group identity changed without an offset-continuity gate. - CD cancellation occurred after the GitOps commit and did not prevent reconciliation. - Provider traffic had no shared rate budget or circuit breaker across replicas. - The customer saw code undefined instead of a correlation ID. - Deleted-pod logs were unavailable, and adjacent credential logging and path traversal risks were found during audit. ## Where We Got Lucky - Rollback restored a still-valid original group offset lineage. - No source-data corruption or unauthorized access was evidenced. - The provider throttle subsided after rollback and queue drain without a broader portal outage. ## Root Cause And Contributing Factors ## Root Causes - **RCA-ROOT-0001** Production consumer-group identity changed across a release, abandoning committed offsets while fromBeginning:true was enabled. Evidence: EV-0003. Actions: ACT-0001, ACT-0005. ## Contributing Factors - **RCA-CF-0001** All replicas shared INSTANCE_ID=rmuqop, creating one new shared group that ran concurrently with the old group during rollout. Evidence: EV-0003. Actions: ACT-0001. - **RCA-CF-0002** No request-level idempotency or deduplication prevented duplicate downstream provider calls. Evidence: EV-0003. Actions: ACT-0002. - **RCA-CF-0003** Retries were process-local and no provider-wide rate limiter or circuit breaker enforced Nearmap capacity across replicas. Evidence: EV-0003, EV-0007. Actions: ACT-0002, ACT-0006. - **RCA-CF-0004** CD cancellation occurred after the GitOps image commit, creating ambiguity about whether production would still reconcile the change. Evidence: EV-0003. Actions: ACT-0005. - **RCA-CF-0005** A 30-second per-attempt timeout plus retries can exceed KafkaJS's default 30-second session timeout, while eachMessage does not heartbeat; eviction and rebalance can create duplicate processing. Evidence: EV-0003. Actions: ACT-0010. - **RCA-CF-0006** Provider responses are fully buffered with arrayBuffer and no size cap. Evidence: EV-0003. Actions: ACT-0010. - **RCA-CF-0007** Kafka-derived folder/name paths can traverse outside the intended root, and the container runs as root. Evidence: EV-0003. Actions: ACT-0011. - **RCA-CF-0008** Startup logging of Kafka SASL/SSL objects may expose credentials or private-key material in source or retained logs. Evidence: EV-0003. Actions: ACT-0012. ## Detection Gaps - **RCA-DG-0001** No deployment gate compared production consumer-group identity and committed offset continuity before rollout. Evidence: EV-0003. Actions: ACT-0005. - **RCA-DG-0002** The customer UI returned code undefined instead of a traceable correlation ID and actionable failure state. Evidence: EV-0002. Actions: ACT-0008. - **RCA-DG-0003** Health checks can remain green after run-loop failure because liveness does not await the consumer Promise and the registry check is a one-time settled Promise. Evidence: EV-0003. Actions: ACT-0014. ## Monitoring Gaps - **RCA-MG-0001** No sufficiently early alert combined group drift, replay pressure, 429 rate, and DLQ growth. Evidence: EV-0003. Actions: ACT-0006. - **RCA-MG-0002** Deleted-pod stdout was unavailable because the Cloud Logging _Default sink was disabled. Evidence: EV-0003. Actions: ACT-0007. ## Documentation And Release Gaps - **RCA-DOC-0001** The production offset-identity invariant and safe consumer-group migration procedure were not enforced as a documented release contract. Evidence: EV-0003, EV-0004. Actions: ACT-0001, ACT-0005, ACT-0006. - **RCA-DOC-0002** Release CI does not make the incident model tests a hard dependency of release, and on-prem overlay patching can report success after string matches become stale. Evidence: EV-0003. Actions: ACT-0013. ## Corrective And Preventive Actions | ID | Priority | Status | Owner | Due | Action | Verification | | --- | --- | --- | --- | --- | --- | --- | | ACT-0001 | P0 | in_progress | Kafka File Download Service Owners | 2026-07-16 | Complete, merge, release, and verify the #4241 remediation set | Required adjacent blockers are resolved or linked to explicit approved follow-ups; PR merged; exact commit built and promoted; production group remains FILE-DOWNLOAD-group; valid offset continuity is proven; rollout has no replay surge, 429 spike, DLQ growth, credential/signed-URL leakage, or stale on-prem overlay; CI and client-like checks pass. | | ACT-0002 | P0 | accepted | Aerialytic Data Recovery and Kafka Platform | 2026-07-16 | Recover failed jobs with deduplication and provider-wide rate bounds | Signed recovery manifest; deduplicated request count; provider-rate budget; dry-run approval; bounded execution receipts; customer workflow verification; residual DLQ disposition. | | ACT-0003 | P0 | in_progress | Aerialytic Incident Response and Data Engineering | 2026-07-16 | Produce exact impacted customer and job inventory | Versioned deduplicated inventory with query hashes, inclusion/exclusion rules, unresolved records, and customer/job counts. | | ACT-0004 | P0 | accepted | Aerialytic Customer Success and Support | 2026-07-16 | Complete customer recovery and approved communications | Approved communication log and per-customer recovery confirmation linked to the deduplicated impact inventory. | | ACT-0005 | P1 | accepted | Aerialytic Release Engineering | 2026-07-17 | Add offset-continuity and cancellation deployment gates | Negative CI fixtures for group drift; approved migration override contract; cancellation/reconcile integration test; production release evidence includes before/after group and offset checks. | | ACT-0006 | P1 | in_progress | Aerialytic SRE and Observability | 2026-07-17 | Ship 429, DLQ, replay-pressure, and group-drift alerts with runbook #4244 | Issue #4244 closed with alert-rule tests, synthetic threshold exercises, dashboards, paging destinations, and runbook drill evidence. | | ACT-0007 | P1 | accepted | Aerialytic Cloud Platform and Security | 2026-07-22 | Restore production workload log retention | Retention policy approved; deleted-pod log retrieval test passes; signed URL and credential redaction tests pass; access and cost controls documented. | | ACT-0008 | P1 | accepted | Aerialytic Iframe Product and Frontend | 2026-07-22 | Add customer-facing error correlation and actionable design failure states | Browser test reproduces a controlled provider failure and verifies a stable safe code, correlation ID, support path, and matching server trace. | | ACT-0009 | P0 | accepted | Kafka File Download Service Owners and SRE | 2026-07-17 | Add provider-wide header-aware rate control and circuit breaking | Multi-replica load tests prove one shared budget, header-aware pause/reset behavior, bounded attempts, circuit-open behavior, and no provider overrun at 1/3/5-pod scale. | | ACT-0010 | P0 | accepted | Kafka Platform and File Download Service Owners | 2026-07-17 | Harden Kafka consumer ownership, buffering, and crash consistency | Tests cover responses over the size cap, 30-second-plus provider stalls, rebalances during eachMessage, crash points around output/DLQ and offset commit, and duplicate request identifiers. | | ACT-0011 | P0 | accepted | Kafka File Download Service Owners and Application Security | 2026-07-16 | Remove file path traversal and root-container exposure | Traversal, encoded traversal, absolute-path, symlink-escape, and valid nested-path tests pass; runtime securityContext proves non-root and least-privilege mounts. | | ACT-0012 | P0 | in_progress | Aerialytic Security and Cloud Platform | 2026-07-16 | Remove credential-bearing startup logs and rotate exposed Kafka/TLS material | Code/log scans show no secret-bearing object logs; exposure assessment is signed; affected credentials are scoped and rotated; old credentials fail; workloads recover without printing secret values. | | ACT-0013 | P1 | accepted | Aerialytic Release Engineering | 2026-07-17 | Make incident contracts hard release gates on the actual artifact | A deliberately broken group, retry, secret-log, health, and stale-overlay fixture each blocks release before deployment; promoted image digest and runtime overlay hashes match the tested release evidence. | | ACT-0014 | P1 | accepted | Kafka File Download Service Owners | 2026-07-18 | Make health checks reflect the live consumer run loop | Fault-injection tests terminate or stall the consumer and prove readiness/liveness transitions, alerting, and bounded restart behavior. | ## Closure Criteria - #4241 and adjacent blockers are released and verified against the exact production artifact. - Exact customer/job impact inventory is complete. - Failed jobs are recovered through a deduplicated, rate-bounded process. - Customer communications and recovery confirmations are recorded. - Security exposure assessment and scoped credential rotation are complete. - Offset, provider-rate, idempotency, health, log-retention, and customer-error actions have verification evidence.