0x0010044F ZIP_EXECUTIVE_V4_LARGE_LOG_VOLUME_EXCEEDED_REMOTE
Applicable alarms (2)
Likely outcomes — click to expand
Based on 0 missions where this alarm fired in the last
90 days
(for 0x0010044F) as of 2026-10-13
Aggregated from fleet operations and refreshed daily. Individual outcomes may vary by site and configuration.
Background
Section titled “Background”This alarm triggers when disk usage on the ZIP_B node’s /application partition rises above the large-log threshold. It remains triggered until disk usage drops below a different, lower clear threshold. For the common ~180 GB /application partition, the alarm sets at roughly 124 GB used and clears below roughly 90 GB used. The critical storage alarm sets later, at roughly 144 GB used.
The onboard-log-cleanup service monitors for missions rejected due to this alarm code. On finding such a mission, the service will SSH onto the appropriate node and clear disk space by deleting low-priority logs.
When onboard-log-cleanup runs successfully, it posts a summary output to the #data-platform-alerts Slack channel. Recent cleanup tasks can also be viewed through the web interface.
The service also provides a web interface that can be used for manual intervention. Note that you will need to be in the tailscale-dock-survey-fixture-users Tailscale group to be able to access the webpage. Post a request in the #ask-it Slack channel if you need to be added to this group.
Troubleshooting Steps
Section titled “Troubleshooting Steps”graph TD
wait-for-onboard-log-cleanup["<small><b>Step 1</b></small><br/>Action: Wait 20 minutes for the onboard-log-..."]
send-a-new-mission["<small><b>Step 2</b></small><br/>Action: Send a new mission to the Zip."]
did-the-new-mission-fail{"<small><b>Step 3</b></small><br/>Question: Did the new mission fail?"}
did-the-mission-fail-due-to-this-alarm{"<small><b>Step 4</b></small><br/>Question: Did the mission fail due to ZIP_EXEC..."}
run-manual-cleanup["<small><b>Step 5</b></small><br/>Action: Use the onboard-log-cleanup web inte..."]
did-manual-cleanup-complete-successfully{"<small><b>Step 6</b></small><br/>Question: Did manual cleanup complete successfu..."}
send-another-new-mission["<small><b>Step 7</b></small><br/>Action: Send another new mission to the Zip."]
did-the-second-new-mission-fail{"<small><b>Step 8</b></small><br/>Question: Did the new mission fail?"}
did-the-second-mission-fail-due-to-this-alarm{"<small><b>Step 9</b></small><br/>Question: Did the mission fail due to ZIP_EXEC..."}
follow-the-failure-guide["<small><b>Step 10</b></small><br/>Action: Follow the troubleshooting guide for ..."]
return-to-service["<small><b>Step 11</b></small><br/>Action: Return the Zip to service."]
escalate-to-data-platform{{"<small><b>Step 12</b></small><br/>Escalation: Escalate to P2 Onboard Data Platform ..."}}
wait-for-onboard-log-cleanup --> send-a-new-mission
send-a-new-mission --> did-the-new-mission-fail
did-the-new-mission-fail -->|Yes| did-the-mission-fail-due-to-this-alarm
did-the-new-mission-fail -->|No| return-to-service
did-the-mission-fail-due-to-this-alarm -->|Yes| run-manual-cleanup
did-the-mission-fail-due-to-this-alarm -->|No| follow-the-failure-guide
run-manual-cleanup --> did-manual-cleanup-complete-successfully
did-manual-cleanup-complete-successfully -->|Yes| send-another-new-mission
did-manual-cleanup-complete-successfully -->|No| escalate-to-data-platform
send-another-new-mission --> did-the-second-new-mission-fail
did-the-second-new-mission-fail -->|Yes| did-the-second-mission-fail-due-to-this-alarm
did-the-second-new-mission-fail -->|No| return-to-service
did-the-second-mission-fail-due-to-this-alarm -->|Yes| escalate-to-data-platform
did-the-second-mission-fail-due-to-this-alarm -->|No| follow-the-failure-guide
style wait-for-onboard-log-cleanup fill:#dcfce7,stroke:#22c55e,color:#14532d
style send-a-new-mission fill:#dcfce7,stroke:#22c55e,color:#14532d
style did-the-new-mission-fail fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style did-the-mission-fail-due-to-this-alarm fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style run-manual-cleanup fill:#dcfce7,stroke:#22c55e,color:#14532d
style did-manual-cleanup-complete-successfully fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style send-another-new-mission fill:#dcfce7,stroke:#22c55e,color:#14532d
style did-the-second-new-mission-fail fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style did-the-second-mission-fail-due-to-this-alarm fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style follow-the-failure-guide fill:#dcfce7,stroke:#22c55e,color:#14532d
style return-to-service fill:#dcfce7,stroke:#22c55e,color:#14532d
style escalate-to-data-platform fill:#fef3c7,stroke:#f59e0b,color:#78350f Hover or focus a block to preview it. Select the block to keep the preview open.
onboard-log-cleanup service to clear low-priority logs onboard.ZIP_EXECUTIVE_V4_LARGE_LOG_VOLUME_EXCEEDED_REMOTE?You will need to be in the
tailscale-dock-survey-fixture-users Tailscale group to be able to access the webpage. Post a request in the #ask-it Slack channel if you need to be added to this group.ZIP_EXECUTIVE_V4_LARGE_LOG_VOLUME_EXCEEDED_REMOTE?Resolution note
This plaintext block is ready to paste into the Resolution Note field on the ZERP issue you're working. The permalink below gives engineering review access to the full step-by-step timeline including any back-navigation; share it if the case needs a deeper look.
Permalink:
Session is too large to fit in a URL. Use Download session JSON to share the timeline out-of-band.
Did this guide help you get unblocked?
Open suggestions
There are no open suggestions.
Open pull requests
There are no open pull requests.