Zip Offline
Viewing this guide for
Background
Section titled “Background”This guide covers the first recovery steps for a Zip that is showing offline in FleetWrangler. It power-cycles the Aircraft, checks onboard disk usage, and cleans up logs when needed before checking compute reachability, installation state, boot mode, and OTA results.
Troubleshooting Steps
Section titled “Troubleshooting Steps”graph TD
power-cycle-the-aircraft["<small><b>Step 1</b></small><br/>Action: Power-cycle the Aircraft: 1. Follow ..."]
is-the-zip-online-after-power-cycle{"<small><b>Step 2</b></small><br/>Question: Is the Zip online in FleetWrangler af..."}
zip-is-back-online["<small><b>Step 3</b></small><br/>Action: The issue is resolved. The Aircraft m..."]
check-onboard-disk-space["<small><b>Step 4</b></small><br/>Action: **Check available disk space** Access..."]
what-is-the-disk-space-status{"<small><b>Step 5</b></small><br/>Question: What status does the get-disk-space-..."}
is-either-compute-near-full{"<small><b>Step 6</b></small><br/>Question: Does the result show that /applicati..."}
clean-up-onboard-logs["<small><b>Step 7</b></small><br/>Action: **Clean up onboard logs** In the Rem..."]
what-is-the-sleep-cleanup-status{"<small><b>Step 8</b></small><br/>Question: What status does the manual sleep t..."}
recheck-onboard-disk-space["<small><b>Step 9</b></small><br/>Action: **Recheck available disk space** In t..."]
what-is-the-disk-space-status-after-sleep{"<small><b>Step 10</b></small><br/>Question: What status does the get-disk-space-..."}
is-either-compute-still-near-full{"<small><b>Step 11</b></small><br/>Question: Does the result still show that /app..."}
delete-log-folders["<small><b>Step 12</b></small><br/>Action: **Delete log folders** In the Remote..."]
did-log-folder-cleanup-complete{"<small><b>Step 13</b></small><br/>Question: Does the log-folders task show com..."}
power-cycle-after-log-cleanup["<small><b>Step 14</b></small><br/>Action: Power-cycle the Aircraft: 1. Follow ..."]
is-the-zip-online-after-log-cleanup{"<small><b>Step 15</b></small><br/>Question: Is the Zip online in FleetWrangler af..."}
zip-is-back-online-after-log-cleanup["<small><b>Step 16</b></small><br/>Action: The issue is resolved. The Aircraft m..."]
escalate-log-cleaner-failed{{"<small><b>Step 17</b></small><br/>Escalation: Escalate to P2 Onboard Data Platform ..."}}
attempt-ssh-after-disk-task-unavailable["<small><b>Step 18</b></small><br/>Action: Use SSH into Compute A/B/topics/do..."]
are-both-computes-reachable-after-disk-task-unavailable{"<small><b>Step 19</b></small><br/>Question: Are both Aircraft computes reachable ..."}
report-log-manager-node-unavailable{{"<small><b>Step 20</b></small><br/>Escalation: Escalate to P2 Onboard Data Platform ..."}}
manually-check-application-disk-space["<small><b>Step 21</b></small><br/>Action: On Compute A and Compute B, check dis..."]
is-either-compute-near-full-manually{"<small><b>Step 22</b></small><br/>Question: Is /application usage 90% or higher..."}
escalate-manual-disk-cleanup{{"<small><b>Step 23</b></small><br/>Escalation: Escalate to P2 Onboard Data Platform ..."}}
attempt-ssh-after-power-cycle["<small><b>Step 24</b></small><br/>Action: Use SSH into Compute A/B/topics/do..."]
are-both-computes-reachable-after-power-cycle{"<small><b>Step 25</b></small><br/>Question: Are both Aircraft computes reachable ..."}
escalate-compute-unreachable-after-power-cycle{{"<small><b>Step 26</b></small><br/>Escalation: Escalate to P2 Embedded Linux On Call..."}}
check-production-mode-and-stack-installed["<small><b>Step 27</b></small><br/>Action: Run this command after SSHing into Co..."]
enable-production-mode["<small><b>Step 28</b></small><br/>Action: On Compute A, enable production mode ..."]
confirm-production-mode-after-reboot["<small><b>Step 29</b></small><br/>Action: On Compute A, confirm production mode..."]
is-zip-online-after-enabling-production-mode{"<small><b>Step 30</b></small><br/>Question: Is the Zip online in FleetWrangler?"}
zip-is-online-after-boot-mode-recovery["<small><b>Step 31</b></small><br/>Action: The issue is resolved. The Aircraft m..."]
escalate-boot-mode-recovery-failed{{"<small><b>Step 32</b></small><br/>Escalation: Escalate to P2 Embedded Linux On Call..."}}
check-recent-ota-results["<small><b>Step 33</b></small><br/>Action: Open the OTA results Slack channel, s..."]
was-there-a-recent-failed-ota{"<small><b>Step 34</b></small><br/>Question: Does the most recent OTA deployment t..."}
follow-ota-deployment-failures["<small><b>Step 35</b></small><br/>Action: Follow OTA Deployment Failures for A..."]
escalate-stack-not-running{{"<small><b>Step 36</b></small><br/>Escalation: Escalate to P2 Embedded Platform On C..."}}
continue-shared-offline-troubleshooting["<small><b>Step 37</b></small><br/>Action: Return to Asset Offline Troubleshoot..."]
power-cycle-the-aircraft --> is-the-zip-online-after-power-cycle
is-the-zip-online-after-power-cycle -->|Yes| zip-is-back-online
is-the-zip-online-after-power-cycle -->|No| check-onboard-disk-space
check-onboard-disk-space --> what-is-the-disk-space-status
what-is-the-disk-space-status -->|Completed| is-either-compute-near-full
what-is-the-disk-space-status -->|Node unavailable| attempt-ssh-after-disk-task-unavailable
what-is-the-disk-space-status -->|Failed| escalate-log-cleaner-failed
is-either-compute-near-full -->|Yes| clean-up-onboard-logs
is-either-compute-near-full -->|No| attempt-ssh-after-power-cycle
clean-up-onboard-logs --> what-is-the-sleep-cleanup-status
what-is-the-sleep-cleanup-status -->|Completed / completed with ...| recheck-onboard-disk-space
what-is-the-sleep-cleanup-status -->|Ineffective| delete-log-folders
what-is-the-sleep-cleanup-status -->|Node unavailable| attempt-ssh-after-disk-task-unavailable
what-is-the-sleep-cleanup-status -->|Failed| delete-log-folders
recheck-onboard-disk-space --> what-is-the-disk-space-status-after-sleep
what-is-the-disk-space-status-after-sleep -->|Completed| is-either-compute-still-near-full
what-is-the-disk-space-status-after-sleep -->|Node unavailable| attempt-ssh-after-disk-task-unavailable
what-is-the-disk-space-status-after-sleep -->|Failed| escalate-log-cleaner-failed
is-either-compute-still-near-full -->|Yes| delete-log-folders
is-either-compute-still-near-full -->|No| power-cycle-after-log-cleanup
delete-log-folders --> did-log-folder-cleanup-complete
did-log-folder-cleanup-complete -->|Yes| power-cycle-after-log-cleanup
did-log-folder-cleanup-complete -->|No| escalate-log-cleaner-failed
power-cycle-after-log-cleanup --> is-the-zip-online-after-log-cleanup
is-the-zip-online-after-log-cleanup -->|Yes| zip-is-back-online-after-log-cleanup
is-the-zip-online-after-log-cleanup -->|No| attempt-ssh-after-power-cycle
attempt-ssh-after-disk-task-unavailable --> are-both-computes-reachable-after-disk-task-unavailable
are-both-computes-reachable-after-disk-task-unavailable -->|Yes| report-log-manager-node-unavailable
are-both-computes-reachable-after-disk-task-unavailable -->|No| escalate-compute-unreachable-after-power-cycle
report-log-manager-node-unavailable --> manually-check-application-disk-space
manually-check-application-disk-space --> is-either-compute-near-full-manually
is-either-compute-near-full-manually -->|Yes| escalate-manual-disk-cleanup
is-either-compute-near-full-manually -->|No| check-production-mode-and-stack-installed
attempt-ssh-after-power-cycle --> are-both-computes-reachable-after-power-cycle
are-both-computes-reachable-after-power-cycle -->|Yes| check-production-mode-and-stack-installed
are-both-computes-reachable-after-power-cycle -->|No| escalate-compute-unreachable-after-power-cycle
check-production-mode-and-stack-installed -->|Maintenance mode| enable-production-mode
check-production-mode-and-stack-installed -->|Production mode and stack i...| continue-shared-offline-troubleshooting
check-production-mode-and-stack-installed -->|Production mode and stack n...| check-recent-ota-results
enable-production-mode --> confirm-production-mode-after-reboot
confirm-production-mode-after-reboot --> is-zip-online-after-enabling-production-mode
is-zip-online-after-enabling-production-mode -->|Yes| zip-is-online-after-boot-mode-recovery
is-zip-online-after-enabling-production-mode -->|No| escalate-boot-mode-recovery-failed
check-recent-ota-results --> was-there-a-recent-failed-ota
was-there-a-recent-failed-ota -->|Yes| follow-ota-deployment-failures
was-there-a-recent-failed-ota -->|No| escalate-stack-not-running
style power-cycle-the-aircraft fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-the-zip-online-after-power-cycle fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style zip-is-back-online fill:#dcfce7,stroke:#22c55e,color:#14532d
style check-onboard-disk-space fill:#dcfce7,stroke:#22c55e,color:#14532d
style what-is-the-disk-space-status fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style is-either-compute-near-full fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style clean-up-onboard-logs fill:#dcfce7,stroke:#22c55e,color:#14532d
style what-is-the-sleep-cleanup-status fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style recheck-onboard-disk-space fill:#dcfce7,stroke:#22c55e,color:#14532d
style what-is-the-disk-space-status-after-sleep fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style is-either-compute-still-near-full fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style delete-log-folders fill:#dcfce7,stroke:#22c55e,color:#14532d
style did-log-folder-cleanup-complete fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style power-cycle-after-log-cleanup fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-the-zip-online-after-log-cleanup fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style zip-is-back-online-after-log-cleanup fill:#dcfce7,stroke:#22c55e,color:#14532d
style escalate-log-cleaner-failed fill:#fef3c7,stroke:#f59e0b,color:#78350f
style attempt-ssh-after-disk-task-unavailable fill:#dcfce7,stroke:#22c55e,color:#14532d
style are-both-computes-reachable-after-disk-task-unavailable fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style report-log-manager-node-unavailable fill:#fef3c7,stroke:#f59e0b,color:#78350f
style manually-check-application-disk-space fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-either-compute-near-full-manually fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style escalate-manual-disk-cleanup fill:#fef3c7,stroke:#f59e0b,color:#78350f
style attempt-ssh-after-power-cycle fill:#dcfce7,stroke:#22c55e,color:#14532d
style are-both-computes-reachable-after-power-cycle fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style escalate-compute-unreachable-after-power-cycle fill:#fef3c7,stroke:#f59e0b,color:#78350f
style check-production-mode-and-stack-installed fill:#dcfce7,stroke:#22c55e,color:#14532d
style enable-production-mode fill:#dcfce7,stroke:#22c55e,color:#14532d
style confirm-production-mode-after-reboot fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-zip-online-after-enabling-production-mode fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style zip-is-online-after-boot-mode-recovery fill:#dcfce7,stroke:#22c55e,color:#14532d
style escalate-boot-mode-recovery-failed fill:#fef3c7,stroke:#f59e0b,color:#78350f
style check-recent-ota-results fill:#dcfce7,stroke:#22c55e,color:#14532d
style was-there-a-recent-failed-ota fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style follow-ota-deployment-failures fill:#dcfce7,stroke:#22c55e,color:#14532d
style escalate-stack-not-running fill:#fef3c7,stroke:#f59e0b,color:#78350f
style continue-shared-offline-troubleshooting fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
1 action
Power-cycle the Aircraft:
1. Follow Power OFF Aircraft.
2. Follow Power ON Aircraft.
When complete, return here to continue.
1. Follow Power OFF Aircraft.
2. Follow Power ON Aircraft.
When complete, return here to continue.
2 question
Is the Zip online in FleetWrangler after the power cycle?
3 action
The issue is resolved. The Aircraft may return to service.
4 action
Check available disk space
Access to the Remote Log Manager requires membership in the
In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
3. Under Nodes, select
4. Click Queue and wait for the task to finish.
Access to the Remote Log Manager requires membership in the
tailscale-dock-survey-fixture-users Tailscale group. If you need access, post a request in the #ask-it Slack channel.In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
get-disk-space-onboard.3. Under Nodes, select
ZIP_A and ZIP_B.4. Click Queue and wait for the task to finish.
5 question
What status does the
get-disk-space-onboard task show? 6 question
Does the result show that
/application usage is 90% or higher on either compute? 7 action
Clean up onboard logs
In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
3. Under Nodes, select
4. Uncheck Dry Run.
5. Click Queue and wait for the task to finish.
In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
sleep.3. Under Nodes, select
ZIP_A and ZIP_B.4. Uncheck Dry Run.
5. Click Queue and wait for the task to finish.
8 question
What status does the manual
sleep task show? 9 action
Recheck available disk space
In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
3. Under Nodes, select
4. Click Queue and wait for the task to finish.
In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
get-disk-space-onboard.3. Under Nodes, select
ZIP_A and ZIP_B.4. Click Queue and wait for the task to finish.
10 question
What status does the
get-disk-space-onboard task show? 11 question
Does the result still show that
/application usage is 90% or higher on either compute? 12 action
Delete log folders
In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
3. Under Nodes, select
4. Uncheck Dry Run.
5. Click Queue and wait for the task to finish.
In the Remote Log Manager:
1. Enter or confirm the Zip Asset ID.
2. Set Target to
log-folders.3. Under Nodes, select
ZIP_A and ZIP_B.4. Uncheck Dry Run.
5. Click Queue and wait for the task to finish.
13 question
Does the
log-folders task show completed or completed with warnings? 14 action
Power-cycle the Aircraft:
1. Follow Power OFF Aircraft.
2. Follow Power ON Aircraft.
When complete, return here to continue.
1. Follow Power OFF Aircraft.
2. Follow Power ON Aircraft.
When complete, return here to continue.
15 question
Is the Zip online in FleetWrangler after the power cycle?
16 action
The issue is resolved. The Aircraft may return to service.
17 escalation
Escalate to P2 Onboard Data Platform On Call. Include the Zip ID, Remote Log Manager task ID, status, and output.
21 action
On Compute A and Compute B, check disk usage:
Check the
df -h /applicationCheck the
Use% value for /application in each result. 22 question
Is
/application usage 90% or higher on either compute? 23 escalation
Escalate to P2 Onboard Data Platform On Call to clear logs. Include the Zip ID, affected compute, and the output of
df -h /application from both computes. 24 action
Use SSH into Compute A/B to attempt to connect to Compute A and Compute B.
25 question
Are both Aircraft computes reachable via SSH?
26 escalation
Escalate to P2 Embedded Linux On Call. Include the Zip ID and which compute is unreachable.
27 action
Run this command after SSHing into Compute A:
Compute A in maintenance mode will not start the flight stack. Select the result shown by the command.
boot_mode=$(sudo boot-mode --check); if [ "$boot_mode" = production ] && [ -f /install/stack_installed ]; then echo "RESULT: production mode and stack installed"; elif [ "$boot_mode" = production ]; then echo "RESULT: production mode and stack not installed"; else echo "RESULT: $boot_mode mode"; fiCompute A in maintenance mode will not start the flight stack. Select the result shown by the command.
28 action
On Compute A, enable production mode and reboot:
sudo boot-mode --enable-production
sudo reboot 29 action
On Compute A, confirm production mode:
Confirm that the command returns
sudo boot-mode --checkConfirm that the command returns
production, then check the Zip in FleetWrangler. 30 question
Is the Zip online in FleetWrangler?
31 action
The issue is resolved. The Aircraft may return to service.
32 escalation
Escalate to P2 Embedded Linux On Call and include the output of
boot-mode --check from both computes. 33 action
Open the OTA results Slack channel, search for the Zip ID, and open the most recent deployment thread. Check the final result and Error section to determine whether the OTA failed.
34 question
Does the most recent OTA deployment thread show a failure?
35 action
36 escalation
Escalate to P2 Embedded Platform On Call to determine why
/install/stack_installed is missing.Resolution note
This plaintext block is ready to paste into the Resolution Note field on the ZERP issue you're working. The permalink below gives engineering review access to the full step-by-step timeline including any back-navigation; share it if the case needs a deeper look.
Permalink:
Session is too large to fit in a URL. Use Download session JSON to share the timeline out-of-band.
Did this guide help you get unblocked?
Open suggestions
There are no open suggestions.