Droid Log Offloading Troubleshooting
Background
Section titled “Background”./scripts/check_log_offloading.sh is a wrapper script that reads the output of the automated droid log offloader to give the technician an idea for the amount of logs still left to offload from the Droid. If the script is not detecting a Droid or the number of logs increase or fail to decrease over the course of 5 minutes, something might be wrong with the log offloader.
Nominal operation should follow: Log Offloading (Aircraft/Droid).
Troubleshooting Steps
Section titled “Troubleshooting Steps”graph TD
follow-the-steps-for-the-type-of-issue{"<small><b>Step 1</b></small><br/>Question: Follow the steps for the type of issu..."}
is-the-droid-connected-to-the-correct-l{"<small><b>Step 2</b></small><br/>Question: Is the Droid connected to the correct..."}
the-droid-log-offloader-software-may-be{{"<small><b>Step 3</b></small><br/>Escalation: The Droid log offloader software may ..."}}
the-droid-is-not-on-the-correct-system["<small><b>Step 4</b></small><br/>Action: The Droid is not on the correct syste..."]
does-ping-devices-return-all-5-nodes-of{"<small><b>Step 5</b></small><br/>Question: Does ping_devices return all 5 node..."}
escalate-to-p2-smart-carts{{"<small><b>Step 6</b></small><br/>Escalation: Escalate to P2 Smart Carts."}}
see-ping-devices-not-seeing-aircraft-or["<small><b>Step 7</b></small><br/>Action: See: Ping_devices not Seeing Aircraf..."]
the-automatic-offloader-did-not-kick-of["<small><b>Step 8</b></small><br/>Action: The automatic offloader did not kick ..."]
wait-5-min["<small><b>Step 9</b></small><br/>Action: Wait 5 min."]
if-the-issue-is-not-resolved-escalate-t{{"<small><b>Step 10</b></small><br/>Escalation: If the issue is not resolved, escalat..."}}
the-offloader-may-have-gotten-stuck-or["<small><b>Step 11</b></small><br/>Action: The offloader may have gotten stuck o..."]
follow-the-steps-for-the-type-of-issue -->|check_log_offloading - “n...| is-the-droid-connected-to-the-correct-l
follow-the-steps-for-the-type-of-issue -->|check_log_offloading - “n...| does-ping-devices-return-all-5-nodes-of
follow-the-steps-for-the-type-of-issue -->|check_log_offloading - lo...| the-automatic-offloader-did-not-kick-of
follow-the-steps-for-the-type-of-issue -->|check_log_offloading - st...| the-offloader-may-have-gotten-stuck-or
is-the-droid-connected-to-the-correct-l -->|Yes| the-droid-log-offloader-software-may-be
is-the-droid-connected-to-the-correct-l -->|No| the-droid-is-not-on-the-correct-system
does-ping-devices-return-all-5-nodes-of -->|Yes| escalate-to-p2-smart-carts
does-ping-devices-return-all-5-nodes-of -->|No| see-ping-devices-not-seeing-aircraft-or
the-automatic-offloader-did-not-kick-of --> wait-5-min
wait-5-min --> if-the-issue-is-not-resolved-escalate-t
style follow-the-steps-for-the-type-of-issue fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style is-the-droid-connected-to-the-correct-l fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style the-droid-log-offloader-software-may-be fill:#fef3c7,stroke:#f59e0b,color:#78350f
style the-droid-is-not-on-the-correct-system fill:#dcfce7,stroke:#22c55e,color:#14532d
style does-ping-devices-return-all-5-nodes-of fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style escalate-to-p2-smart-carts fill:#fef3c7,stroke:#f59e0b,color:#78350f
style see-ping-devices-not-seeing-aircraft-or fill:#dcfce7,stroke:#22c55e,color:#14532d
style the-automatic-offloader-did-not-kick-of fill:#dcfce7,stroke:#22c55e,color:#14532d
style wait-5-min fill:#dcfce7,stroke:#22c55e,color:#14532d
style if-the-issue-is-not-resolved-escalate-t fill:#fef3c7,stroke:#f59e0b,color:#78350f
style the-offloader-may-have-gotten-stuck-or fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
Escalate to P2 Smart Carts.
ping_devices return all 5 nodes of the Droid?Run the following command:
pkill -f auto_rsync1. Run
ssh_droid "du -h --max-depth=1 /application/log".2. Find the log directory that is over 10G.
3. Delete it with a
rm -rf command.Example:
ssh_droid "rm -rf /application/log/20260427-025821__p2-droid-922__droid__rated-shop-maintain".Example:
20260427-025821__p2-droid-922__droid__rated-shop-maintain is 25G and should be deleted.
4. Kill the previous iteration of the offload script with
pkill -f auto_rsync.5. Wait 5 min and check status again. A new job should have started.
Resolution note
This plaintext block is ready to paste into the Resolution Note field on the ZERP issue you're working. The permalink below gives engineering review access to the full step-by-step timeline including any back-navigation; share it if the case needs a deeper look.
Permalink:
Session is too large to fit in a URL. Use Download session JSON to share the timeline out-of-band.
General Troubleshooting
Section titled “General Troubleshooting”To check the latest direct output from the log offloader, run tail /tmp/auto_rsync.out. Any errors from the offload step will show here.
To check the latest output from the cloud upload, run tail /tmp/log_upload.out. Any errors from the cloud upload step will show here.
Did this guide help you get unblocked?
Open suggestions
There are no open suggestions.
Open pull requests
There are no open pull requests.