[PLA4-44978] Remove Celery autoscaler timing logs

[samuelshih]

Linear

• https://linear.app/scale-epd/issue/PLA4-44978

Finding

The celery-autoscaler-sqs loop emits four INFO timing statements on every iteration and for every scanned namespace:

• list_namespaced_deployment ... took ... seconds
• list_deployments took ... seconds
• get_metrics took ... seconds
• Iteration length: ... seconds

This change removes only those timing statements and their now-unused timers. Queue state, scaling actions, errors, and health telemetry remain unchanged.

Datadog evidence

Frozen window: 2026-09-04 20:08:12 UTC through 2026-09-11 20:08:12 UTC.

• Exact matching logs and seven-day volume: 116,003,024 INFO events
• All service indexed-event volume: 118.00M events
• All service ingested-byte volume: 216.27 GB
• The selector returned only info events and represents approximately 98% of indexed service volume.

Estimated savings

Contract rates: $0.98 per million indexed events (7-day retention) and $0.10 per ingested GB. Monthly estimates use 30.4375 / 7.

Cost	Seven-day removable usage	Monthly value
Indexed events	116,003,024	$494.32
Ingestion	212.612 GB estimated	$92.45
Total		$586.77/month ($7,041.19/year)

The ingestion estimate allocates measured service bytes using the exact selector's share of service events. The full census projects 44.46B indexed events against a 26B monthly commitment and 158,475 GB ingestion against a 105,000 GB commitment, so gross usage value and projected invoice impact are currently the same.

Safety and provenance

• Removal confidence: 95/100.
• The four timing messages were introduced with the autoscaler in PR #378, merged November 17, 2023.
• Queue-size gauges, worker-count gauges, connection gauges, celery_autoscaler.heartbeat, scaling success/failure metrics, deployment-update logs, queue-state logs, and every exception/error path remain intact.
• pup monitors search found eight Celery-autoscaler-related monitors and zero exact references to the removed strings. Four log monitors alert on status:error; four health monitors use celery_autoscaler.heartbeat. Neither dependency is changed.
• Security Monitoring Rules and saved Log Explorer views could not be audited because the current app key receives 403 for those APIs. The PR remains draft for owner review.

The repository has no CODEOWNERS file. The autoscaler chart/workload metadata uses team: infra; ownership should be confirmed during review.

Tests

• Ruff 0.6.8 check on the changed file.
• Black 24.8.0 formatting check.
• Python bytecode compilation.
• git diff --check.

Rollout

Draft only. No deployment or reviewers assigned. After deploy, verify the exact query falls while status:error alerts, heartbeat monitors, scaling metrics, and deployment-update logs remain healthy.

Confidence Score: 5/5

This PR appears safe to merge.

What we checked:

• Loop pacing stays intact: No. The loop still records its start time, measures the full pass, and sleeps long enough to keep the three-second minimum interval.

Summary

The Celery autoscaler stops writing timing logs while keeping its scaling loop and health reporting unchanged. It removes timers used only for those messages, reducing repeated INFO log volume.

Diagram

sequenceDiagram
    participant A as Autoscaler
    participant K as Kubernetes API
    participant B as Queue broker
    participant D as Datadog
    loop Autoscaler pass
        A->>K: "List deployments"
        K-->>A: "Return deployments"
        A->>B: "Read queue and worker state"
        B-->>A: "Return queue metrics"
        A->>D: "Send queue and worker gauges"
        A->>K: "Update worker counts"
        opt Pass took under three seconds
            A->>A: "Sleep for the remaining time"
        end
        A->>D: "Send heartbeat"
        Note over A,D: Timing INFO logs are no longer sent
    end

Loading

_{Reviews (1) · Last reviewed commit: "[PLA4-44978] Remove Celery autoscaler ti..."}