Asset Offline Due To Comms OR Provisioning Agent Error
Viewing this guide for
Related search words: Offline asset, cloud comms, provisioning agent, call home.
Background
Section titled “Background”This guide fits into the larger Asset Offline Flow described here: Asset Offline Troubleshooting Landing Page. Before making edits to this page, please review the Landing Page and ensure that the steps here fit into the overall flow.
Asset Offline vs C2 Loss
Section titled “Asset Offline vs C2 Loss”There is a distinction between an Asset being offline and C2 Loss. C2 Loss can only occur on Zips, and only while they are assigned a mission. See the C2 Loss guide: C2_EXTENDED_LOSS.
This guide focuses on situations where an individual asset is offline in Fleet Wrangler, and we have ruled out a site or asset networking issue.
Troubleshooting Steps
Section titled “Troubleshooting Steps”graph TD
check-fleetwrangler-fleetview-in-the-in["<small><b>Step 1</b></small><br/>Action: Check FleetWrangler/FleetView in the ..."]
verify-in-zerp-this-asset-is-at-the-app["<small><b>Step 2</b></small><br/>Action: In addition to checking Fleetwrangler..."]
ssh-into-the-zip-or-dock["<small><b>Step 3</b></small><br/>Action: SSH into the zip or dock. For steps s..."]
check-provisioning-mtls-certificate-validity["<small><b>Step 4</b></small><br/>Action: On the offline node, check the provis..."]
is-provisioning-mtls-certificate-expired{"<small><b>Step 5</b></small><br/>Question: Is today's date past the certificate'..."}
retrieve-fresh-provisioning-certificates["<small><b>Step 6</b></small><br/>Action: Pull fresh certificates on your own m..."]
stage-fresh-provisioning-certificates{"<small><b>Step 7</b></small><br/>Question: Which offline asset needs the fresh c..."}
install-fresh-dock-provisioning-certificates["<small><b>Step 8</b></small><br/>Action: Stage the fresh certificate files to ..."]
install-fresh-aircraft-provisioning-certificates["<small><b>Step 9</b></small><br/>Action: Stage the fresh certificate files to ..."]
restart-affected-agents["<small><b>Step 10</b></small><br/>Action: Restart the affected services on each..."]
verify-that-the-provisioning-agent-step["<small><b>Step 11</b></small><br/>Action: Verify that the provisioning_agent st..."]
has-provisioning-succeeded{"<small><b>Step 12</b></small><br/>Question: Has provisioning succeeded?"}
debug-provisioning-failure-steps-below["<small><b>Step 13</b></small><br/>Action: Debug provisioning using Provisionin..."]
continue-to-cloud-communication-check["<small><b>Step 14</b></small><br/>Action: Verify that the cloud comms agent has..."]
check-fleetwrangler-fleetview-in-the-in --> verify-in-zerp-this-asset-is-at-the-app
verify-in-zerp-this-asset-is-at-the-app --> ssh-into-the-zip-or-dock
ssh-into-the-zip-or-dock --> check-provisioning-mtls-certificate-validity
check-provisioning-mtls-certificate-validity --> is-provisioning-mtls-certificate-expired
is-provisioning-mtls-certificate-expired -->|Yes| retrieve-fresh-provisioning-certificates
is-provisioning-mtls-certificate-expired -->|No| verify-that-the-provisioning-agent-step
retrieve-fresh-provisioning-certificates --> stage-fresh-provisioning-certificates
stage-fresh-provisioning-certificates -->|Dock| install-fresh-dock-provisioning-certificates
stage-fresh-provisioning-certificates -->|Aircraft| install-fresh-aircraft-provisioning-certificates
install-fresh-dock-provisioning-certificates --> restart-affected-agents
install-fresh-aircraft-provisioning-certificates --> restart-affected-agents
restart-affected-agents --> verify-that-the-provisioning-agent-step
verify-that-the-provisioning-agent-step --> has-provisioning-succeeded
has-provisioning-succeeded -->|Yes| continue-to-cloud-communication-check
has-provisioning-succeeded -->|No| debug-provisioning-failure-steps-below
style check-fleetwrangler-fleetview-in-the-in fill:#dcfce7,stroke:#22c55e,color:#14532d
style verify-in-zerp-this-asset-is-at-the-app fill:#dcfce7,stroke:#22c55e,color:#14532d
style ssh-into-the-zip-or-dock fill:#dcfce7,stroke:#22c55e,color:#14532d
style check-provisioning-mtls-certificate-validity fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-provisioning-mtls-certificate-expired fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style retrieve-fresh-provisioning-certificates fill:#dcfce7,stroke:#22c55e,color:#14532d
style stage-fresh-provisioning-certificates fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style install-fresh-dock-provisioning-certificates fill:#dcfce7,stroke:#22c55e,color:#14532d
style install-fresh-aircraft-provisioning-certificates fill:#dcfce7,stroke:#22c55e,color:#14532d
style restart-affected-agents fill:#dcfce7,stroke:#22c55e,color:#14532d
style verify-that-the-provisioning-agent-step fill:#dcfce7,stroke:#22c55e,color:#14532d
style has-provisioning-succeeded fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style debug-provisioning-failure-steps-below fill:#dcfce7,stroke:#22c55e,color:#14532d
style continue-to-cloud-communication-check fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
1 action
Check FleetWrangler/FleetView in the
Integration environment rather than the production environment - or vice versa if the asset in question is a test asset. This will confirm that there wasn’t an issue with provisioning or bringup that has the asset communicating with the wrong environment. Continue to the next step after confirming the asset is offline in both environments 2 action
In addition to checking Fleetwrangler in the
Integration environment, verify in Zerp this asset is at the appropriate commercial location, not a test site. Search for the asset ID in Zerp and check its location. If it is not clear that the location is correct or may be an integration test site, escalate to the Zerp team. 3 action
SSH into the zip or dock. For steps see SSH into Compute A/B or SSH into Dock.
4 action
On the offline node, check the provisioning mTLS certificate validity:
cd /application/certs/provisioning
openssl x509 -in service-0_certificate.pem -text -noout | grep -A2 "Validity" 5 question
Is today's date past the certificate's
Not After time? 6 action
Pull fresh certificates on your own machine using the required AWS role:
If you do not have the required AWS role or this command fails, escalate to P2 Embedded Platform On Call.
aws secretsmanager get-secret-value \
--region us-west-2 \
--secret-id "cloud-infra/cert-mtls-client.flyzipline.com-data" \
--query "SecretString" \
--output textIf you do not have the required AWS role or this command fails, escalate to P2 Embedded Platform On Call.
7 question
Which offline asset needs the fresh certificates?
8 action
Stage the fresh certificate files to the Dock, replacing
If staging, SSH, or moving the files fails, escalate to P2 Embedded Platform On Call.
xxx with the affected Dock number, then SSH into it:scp service-0* zipline@p2-dock-xxx:~/
ssh zipline@p2-dock-xxx
sudo mv ~/service-0* /application/certs/provisioning/If staging, SSH, or moving the files fails, escalate to P2 Embedded Platform On Call.
9 action
Stage the fresh certificate files to the affected Dock, replacing
xxx with the Dock number, then SSH into the Dock and copy the files to both Aircraft computes over ZipWAN:scp service-0* zipline@p2-dock-xxx:~/
ssh zipline@p2-dock-xxx
scp ~/service-0* 'zipline@[fe80::50%zipwan]:/application/certs/provisioning/'
scp ~/service-0* 'zipline@[fe80::51%zipwan]:/application/certs/provisioning/'fe80::50%zipwan is Compute A and fe80::51%zipwan is Compute B. If staging, SSH, or either transfer fails, escalate to P2 Embedded Platform On Call. 10 action
Restart the affected services on each affected node:
If either restart fails, escalate to P2 Embedded Platform On Call.
sudo systemctl restart provisioning_agent.service
sudo systemctl restart cloud_comms_agent.serviceIf either restart fails, escalate to P2 Embedded Platform On Call.
11 action
Verify that the provisioning_agent step has run successfully with the command:
The output of the above command will be displayed in two parts. First, it will show the status of the service, and be in this format:
The second part of the output will be the logs of the
sudo systemctl status provisioning_agent.service.The output of the above command will be displayed in two parts. First, it will show the status of the service, and be in this format:
sudo systemctl status provisioning_agent.service
○ provisioning_agent.service - Fetch boot up data from the cloud
Loaded: loaded (/etc/systemd/system/provisioning_agent.service; static)
Active: inactive (dead) since Tue 2025-06-10 20:32:26 UTC; 21min ago
Duration: 6min 30.232s
Main PID: 2294 (code=exited, status=0/SUCCESS)The second part of the output will be the logs of the
provisioning_agent service itself. If the provisioning agent ran successfully you should see these lines at the end of the output:Jun 10 20:25:56 p2-ev-zp-hil-5-compute-a systemd[1]: Finished Fetch boot up data from the cloud.
Jun 10 20:32:26 p2-ev-zp-hil-5-compute-a systemd[1]: provisioning_agent.service: Deactivated successfully.
Jun 10 20:32:26 p2-ev-zp-hil-5-compute-a systemd[1]: Stopped Fetch boot up data from the cloud.Notice: journal has been rotated since unit was started, output may be incomplete. 12 question
Has provisioning succeeded?
13 action
Debug provisioning using Provisioning Failure: Troubleshooting Steps below.
14 action
Verify that the cloud comms agent has run successfully with the command using the Cloud Comms Agent: Troubleshooting Steps section below.
Resolution note
This plaintext block is ready to paste into the Resolution Note field on the ZERP issue you're working. The permalink below gives engineering review access to the full step-by-step timeline including any back-navigation; share it if the case needs a deeper look.
Permalink:
Session is too large to fit in a URL. Use Download session JSON to share the timeline out-of-band.
Determining if an asset is communicating to the cloud
Section titled “Determining if an asset is communicating to the cloud”graph TD
open-fleet-debugger["<small><b>Step 1</b></small><br/>Action: If the zip/dock is online in FleetWra..."]
select-status-checkbox["<small><b>Step 2</b></small><br/>Action: Scroll down to ZipStatus or DockSt..."]
check-message-recency{"<small><b>Step 3</b></small><br/>Question: Are the most recent messages from the..."}
asset-is-communicating-with-cloud["<small><b>Step 4</b></small><br/>Action: If the most recent messages are from ..."]
asset-is-not-communicating-with-cloud["<small><b>Step 5</b></small><br/>Action: If the most recent messages are older..."]
status-checkbox-missing["<small><b>Step 6</b></small><br/>Action: If you do not see a ZipStatus or D..."]
open-fleet-debugger --> select-status-checkbox
select-status-checkbox --> check-message-recency
check-message-recency -->|Most recent messages are fr...| asset-is-communicating-with-cloud
check-message-recency -->|Most recent messages are older| asset-is-not-communicating-with-cloud
check-message-recency -->|I do not see a ZipStatus/Do...| status-checkbox-missing
style open-fleet-debugger fill:#dcfce7,stroke:#22c55e,color:#14532d
style select-status-checkbox fill:#dcfce7,stroke:#22c55e,color:#14532d
style check-message-recency fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style asset-is-communicating-with-cloud fill:#dcfce7,stroke:#22c55e,color:#14532d
style asset-is-not-communicating-with-cloud fill:#dcfce7,stroke:#22c55e,color:#14532d
style status-checkbox-missing fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
1 action
If the zip/dock is online in FleetWrangler, the cloud is likely hearing from it. If it is offline, open Fleet Debugger:
- https://fleet-debugger.platform.flyzipline.com/#!/details/dock/<dock_id>
- https://fleet-debugger.platform.flyzipline.com/#!/details/zip/<zip_id>
- https://fleet-debugger.platform.flyzipline.com/#!/details/dock/<dock_id>
- https://fleet-debugger.platform.flyzipline.com/#!/details/zip/<zip_id>
2 action
Scroll down to
ZipStatus or DockStatus and click the checkbox. 3 question
Are the most recent messages from the last few seconds, or are they older?
4 action
If the most recent messages are from the last few seconds, the asset is communicating with the cloud.
If the asset still appears offline in FleetWrangler, escalate to ASW Delivery Support On Call.
If the asset still appears offline in FleetWrangler, escalate to ASW Delivery Support On Call.
5 action
If the most recent messages are older, the asset is likely not communicating with the cloud.
In this case, follow the troubleshooting guides below.
In this case, follow the troubleshooting guides below.
6 action
If you do not see a
In this case, follow the troubleshooting guides below.
ZipStatus or DockStatus checkbox, the asset is likely not communicating with the cloud.In this case, follow the troubleshooting guides below.
Cloud Comms Agent: Troubleshooting Steps
Section titled “Cloud Comms Agent: Troubleshooting Steps”graph TD
run-cloud-comms-agent-status["<small><b>Step 1</b></small><br/>Action: Run: text sudo systemctl status cl..."]
match-cloud-comms-output["<small><b>Step 2</b></small><br/>Action: Follow the troubleshooting steps in t..."]
run-cloud-comms-agent-status --> match-cloud-comms-output
style run-cloud-comms-agent-status fill:#dcfce7,stroke:#22c55e,color:#14532d
style match-cloud-comms-output fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
1 action
Run:
A running comms agent will only have these messages every few seconds:
sudo systemctl status cloud_comms_agent.serviceA running comms agent will only have these messages every few seconds:
Apr 05 15:41:59 p2-dock-317 comms_agent[28913]: 2026-04-05T15:41:59.170822Z INFO bin: Send thread busy, waiting till next cycle
Apr 05 15:44:18 p2-dock-317 comms_agent[28913]: 2026-04-05T15:44:18.169336Z INFO bin: Send thread busy, waiting till next cycle
Apr 05 15:47:38 p2-dock-317 comms_agent[28913]: 2026-04-05T15:47:38.367959Z INFO bin: Send thread busy, waiting till next cycle 2 action
Follow the troubleshooting steps in the table below that relate to the text that you see.
| If you see this: | Troubleshooting Steps |
|---|---|
| This error means the asset has tried to send or receive a larger than allowed message to/from the cloud per our transport protocol limits.
|
|
|
If on a zip: |
|
The cloud_comms_agent has crashed. Run sudo systemctl status cloud_comms_agent.service to determine. |
|
| The cloud_comms_agent is healthy. |
|
Provisioning Failure: Troubleshooting Steps
Section titled “Provisioning Failure: Troubleshooting Steps”graph TD
restart-provisioning-agent["<small><b>Step 1</b></small><br/>Action: Restart the provisioning agent: te..."]
verify-inventory-location["<small><b>Step 2</b></small><br/>Action: Make sure the Zip/Dock has been corre..."]
run-provisioning-journal-logs["<small><b>Step 3</b></small><br/>Action: The provisioning agent can fail for a..."]
follow-applicable-asset{"<small><b>Step 4</b></small><br/>Question: Follow the steps for the applicable a..."}
aircraft-provisioning-cert-missing["<small><b>Step 5</b></small><br/>Action: If the provisioning_agent service r..."]
is-aircraft-visible-in-fleetwrangler{"<small><b>Step 6</b></small><br/>Question: Is the Aircraft visible in FleetWrang..."}
aircraft-may-return-to-service("<small><b>Step 7</b></small><br/>Reference: The Aircraft may return to service.")
escalate-application-software{{"<small><b>Step 8</b></small><br/>Escalation: Escalate to Embedded On Call."}}
dock-provisioning-identity-or-cert["<small><b>Step 9</b></small><br/>Action: If the provisioning_agent service r..."]
restart-provisioning-agent --> verify-inventory-location
verify-inventory-location --> run-provisioning-journal-logs
run-provisioning-journal-logs --> follow-applicable-asset
follow-applicable-asset -->|Aircraft| aircraft-provisioning-cert-missing
follow-applicable-asset -->|Dock| dock-provisioning-identity-or-cert
aircraft-provisioning-cert-missing --> is-aircraft-visible-in-fleetwrangler
is-aircraft-visible-in-fleetwrangler -->|Yes| aircraft-may-return-to-service
is-aircraft-visible-in-fleetwrangler -->|No| escalate-application-software
style restart-provisioning-agent fill:#dcfce7,stroke:#22c55e,color:#14532d
style verify-inventory-location fill:#dcfce7,stroke:#22c55e,color:#14532d
style run-provisioning-journal-logs fill:#dcfce7,stroke:#22c55e,color:#14532d
style follow-applicable-asset fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style aircraft-provisioning-cert-missing fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-aircraft-visible-in-fleetwrangler fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style aircraft-may-return-to-service fill:#f3e8ff,stroke:#a855f7,color:#581c87
style escalate-application-software fill:#fef3c7,stroke:#f59e0b,color:#78350f
style dock-provisioning-identity-or-cert fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
1 action
Restart the provisioning agent:
sudo systemctl restart provisioning_agent 2 action
Make sure the Zip/Dock has been correctly received in Odoo to the correct inventory location.
A Zip/Dock receives a Commercial Ops certificate only if it is in a Commercial Ops inventory location.
If it is not in a Commercial Ops inventory location, it will receive a Test Ops certificate and will be visible in Fleet Wrangler's integration environment instead.
A Zip/Dock receives a Commercial Ops certificate only if it is in a Commercial Ops inventory location.
If it is not in a Commercial Ops inventory location, it will receive a Test Ops certificate and will be visible in Fleet Wrangler's integration environment instead.
3 action
The provisioning agent can fail for a number of reasons. Run:
Then copy the output. It should look something like the example below.
sudo journalctl --lines 50 -u provisioning_agent.service | tail -n 50Then copy the output. It should look something like the example below.
Example provisioning agent failure output
Jun 10 17:24:45 p2-ev-zp-hil-1-compute-a systemd[1]: Starting Fetch boot up data from the cloud...
Jun 10 17:24:45 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: Attempt 1 failed: Failed to create channel: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))). Retrying in 100ms...
Jun 10 17:24:45 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: Attempt 2 failed: Failed to create channel: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))). Retrying in 200ms...
Jun 10 17:24:45 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: Attempt 3 failed: Failed to create channel: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))). Retrying in 400ms...
Jun 10 17:24:45 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: Attempt 4 failed: Failed to create channel: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))). Retrying in 800ms...
Jun 10 17:24:46 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: Attempt 5 failed: Failed to create channel: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))). Retrying in 1600ms...
Jun 10 17:24:48 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: Attempt 6 failed: Failed to create channel: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))). Retrying in 3200ms...
Jun 10 17:24:51 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: Retry failed after 7 attempts in 6.3052517s
Jun 10 17:24:51 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: thread 'main' panicked at cloud_comms/provisioning_agent/src/main.rs:204:34:
Jun 10 17:24:51 p2-ev-zp-hil-1-compute-a provisioning_agent[5185]: called `Result::unwrap()` on an `Err` value: Failed to create channel: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))
Jun 10 17:24:51 p2-ev-zp-hil-1-compute-a systemd[1]: provisioning_agent.service: Main process exited, code=exited, status=101/n/a
Jun 10 17:24:51 p2-ev-zp-hil-1-compute-a systemd[1]: provisioning_agent.service: Failed with result 'exit-code'.
Jun 10 17:24:51 p2-ev-zp-hil-1-compute-a systemd[1]: Failed to start Fetch boot up data from the cloud. 4 question
Follow the steps for the applicable asset.
5 action
If the
Using a smart-cart, run the
If you are unable to update via a smart-cart, escalate to OTA Support On Call to add the requisite certs to the Aircraft.
provisioning_agent service reads Unable to read ca_cert file: Os { code: 2, kind: NotFound, message: "No such file or directory" }, the Aircraft does not have updated certificates.Using a smart-cart, run the
provision_vehicle test on it to get the latest certs.If you are unable to update via a smart-cart, escalate to OTA Support On Call to add the requisite certs to the Aircraft.
6 question
Is the Aircraft visible in FleetWrangler?
7 reference
The Aircraft may return to service.
8 escalation
Escalate to Embedded On Call.
9 action
If the
If the
If you are dealing with a Dock Offline event and you are still having issues, make sure that the Dock is assigned to a Bay at your location. Fleet Wrangler only displays Docks that are associated with Bays.
If the provisioning steps do not resolve the issue:
- If urgent, escalate to P2 Embedded Platform On Call and include the output above.
- If not urgent, post the output in
provisioning_agent service reads Failed to read identity package: OpenEepromFile(Os { code: 2, kind: NotFound, message: "No such file or directory" }), the Dock's identity was not properly written. Reach out to Laila Khan or Pin-Wen Wang to update the identity of the Dock.If the
provisioning_agent service reads Unable to read ca_cert file: Os { code: 2, kind: NotFound, message: "No such file or directory" }, the Dock does not have updated certificates. Complete one of the following actions:- If you are bringing up this Dock and can put the Dock onto a Dock smart-cart, run the
provision_dock test on it to get the latest certs.- If you are unable to update via a smart-cart, reach out to Pin-Wen Wang to add the requisite certs to the Dock.
If you are dealing with a Dock Offline event and you are still having issues, make sure that the Dock is assigned to a Bay at your location. Fleet Wrangler only displays Docks that are associated with Bays.
If the provisioning steps do not resolve the issue:
- If urgent, escalate to P2 Embedded Platform On Call and include the output above.
- If not urgent, post the output in
#ask-embedded-platform.Did this guide help you get unblocked?
Open suggestions
There are no open suggestions.