RTK Station Offline
Viewing this guide for
Background
Section titled “Background”This guide fits into the larger Asset Offline Flow described here: DRAFT Asset Offline Landing Page. Before making edits to this page, please review the Landing Page and ensure that the steps here fit into the overall flow.
This guide covers dock specific troubleshooting steps to determine why a dock is showing offline on Fleetwrangler.
Onsite Troubleshooting Steps
Section titled “Onsite Troubleshooting Steps”When there is a CORRECTIONS_REDUNDANCY_FAILURE across the fleet at one or more sites, it is possible that there is something wrong with the RTK base stations that provide RTK corrections to the Aircraft.
If sending MX to the site, ensure they have the following tools:
- PoE injector
- RJ45 tester
- Basic tools, such as cable strippers and screwdrivers
Corrections streams arrive via cellular connection to both computes from two separate providers:
- Cloud: A stream from Zipline’s cloud (corrections come from Zipline RTK Base Station)
- Polaris: A stream from a 3rd party vendor (corrections come from Point One Nav RTK Base Station)
- Polaris bridged: A copy of the other compute’s Polaris corrections stream (compute A bridges to compute B, and vice versa)
graph TD
check-noc-dashboard["<small><b>Step 1</b></small><br/>Action: Check the NOC Dashboardhttps://fle..."]
is-station-unhealthy{"<small><b>Step 2</b></small><br/>Question: Is the RTK station unhealthy based on..."}
report-dashboard-discrepancy{{"<small><b>Step 3</b></small><br/>Escalation: Escalate to ASW Delivery Support On C..."}}
choose-rtk-provider{"<small><b>Step 4</b></small><br/>Question: Which RTK provider is offline?"}
are-multiple-zip-sites-offline{"<small><b>Step 5</b></small><br/>Question: Are multiple Zip RTK sites offline?"}
do-sites-share-behavior{"<small><b>Step 6</b></small><br/>Question: Do all the sites share the same behav..."}
triage-sites-independently["<small><b>Step 7</b></small><br/>Action: Treat the sites as individual inciden..."]
escalate-multiple-site-networking{{"<small><b>Step 8</b></small><br/>Escalation: Tag IT Networking On Call to check if..."}}
what-did-it-networking-confirm{"<small><b>Step 9</b></small><br/>Question: What did IT Networking confirm?"}
resolve-missing-peplink-stations{{"<small><b>Step 10</b></small><br/>Escalation: If the RTK stations are not shown on ..."}}
return-after-networking-resolution["<small><b>Step 11</b></small><br/>Action: Once IT Networking resolves the issue..."]
check-cloud-metric-changes["<small><b>Step 12</b></small><br/>Action: Tag @utm-on-call to check whether a..."]
return-after-utm-resolution["<small><b>Step 13</b></small><br/>Action: The site may return to service once ..."]
check-rtcm3-messages["<small><b>Step 14</b></small><br/>Action: Check the **Rtcm3 Messages per second..."]
are-rtcm3-messages-present{"<small><b>Step 15</b></small><br/>Question: Are RTCM3 messages being reported?"}
are-corrections-stable{"<small><b>Step 16</b></small><br/>Question: Are you seeing stable Nominal Correct..."}
wait-and-confirm-corrections{{"<small><b>Step 17</b></small><br/>Escalation: Wait 30 minutes and confirm whether t..."}}
check-recent-site-work{"<small><b>Step 18</b></small><br/>Question: Check Zerp for work orders against th..."}
revert-site-change{"<small><b>Step 19</b></small><br/>Question: Tag metro specific GSSO-on-call. All ..."}
network-or-power-issue{{"<small><b>Step 20</b></small><br/>Escalation: There is likely an issue with network..."}}
are-all-pointone-sites-affected{"<small><b>Step 21</b></small><br/>Question: Are all PointOne RTK sites affected, ..."}
escalate-pointone-fleetwide{{"<small><b>Step 22</b></small><br/>Escalation: Escalate to P2 Nav On Call."}}
open-rtk-corrections-dashboard["<small><b>Step 23</b></small><br/>Action: Use the RTK Corrections Dashboardh..."]
is-polaris-base-station-up{"<small><b>Step 24</b></small><br/>Question: Is that specific site's **Polaris Bas..."}
is-polaris-uptime-above-85{"<small><b>Step 25</b></small><br/>Question: Is that specific site's **Polaris Upt..."}
are-client-stream-outages-below-20{"<small><b>Step 26</b></small><br/>Question: Is that specific site's **Polaris Cli..."}
polaris-base-station-working-properly["<small><b>Step 27</b></small><br/>Action: The POLARIS base station is currently..."]
pointone-network-or-power-issue["<small><b>Step 28</b></small><br/>Action: There is likely an issue with network..."]
is-pointone-station-poe-based{"<small><b>Step 29</b></small><br/>Question: Is this a PoE-based PointOne station?..."}
escalate-pointone-networking{{"<small><b>Step 30</b></small><br/>Escalation: Escalate to IT Networking On Call to ..."}}
escalate-pointone-cell-coverage{{"<small><b>Step 31</b></small><br/>Escalation: Tag P2 Nav On Call. They will reach o..."}}
replace-pointone-power-adapter["<small><b>Step 32</b></small><br/>Action: Wait for instructions from P2 Nav On ..."]
verify-networking-resolved{"<small><b>Step 33</b></small><br/>Question: Verify whether IT Networking On Call ..."}
request-onsite-gsso["<small><b>Step 34</b></small><br/>Action: Tag metro specific GSSO-on-call. All ..."]
do-rj45-tests-pass{"<small><b>Step 35</b></small><br/>Question: Inspect the cables for any water dama..."}
check-poe-coupler["<small><b>Step 36</b></small><br/>Action: Check the PoE coupler and replace it ..."]
open-junction-box["<small><b>Step 37</b></small><br/>Action: Open the junction box located at the ..."]
inspect-cables-with-rj45-tester{"<small><b>Step 38</b></small><br/>Question: Inspect the cables and verify connect..."}
reterminate-junction-cat6a["<small><b>Step 39</b></small><br/>Action: Re-terminate the CAT6A cable and re-t..."]
did-junction-retest-pass{"<small><b>Step 40</b></small><br/>Question: Did the RJ45 re-test pass?"}
repeat-junction-retermination["<small><b>Step 41</b></small><br/>Action: Re-terminate the CAT6A cable and re-t..."]
reterminate-first-cat6a["<small><b>Step 42</b></small><br/>Action: Re-terminate the CAT6A cable and re-t..."]
did-first-retest-pass{"<small><b>Step 43</b></small><br/>Question: Did the RJ45 re-test pass?"}
repeat-first-retermination["<small><b>Step 44</b></small><br/>Action: Re-terminate the CAT6A cable and re-t..."]
check-power-and-ethernet{"<small><b>Step 45</b></small><br/>Question: Check with IT Networking On Call whet..."}
is-onsite-issue-resolved{"<small><b>Step 46</b></small><br/>Question: Is the issue resolved?"}
request-hardware-guidance["<small><b>Step 47</b></small><br/>Action: Reach out to Site-Auxiliary-On-Call..."]
resume-flights["<small><b>Step 48</b></small><br/>Action: The issue is resolved and you can res..."]
site-may-return-to-service["<small><b>Step 49</b></small><br/>Action: The site may return to service."]
check-noc-dashboard --> is-station-unhealthy
is-station-unhealthy -->|Yes| choose-rtk-provider
is-station-unhealthy -->|No| report-dashboard-discrepancy
choose-rtk-provider -->|Zip RTK| are-multiple-zip-sites-offline
choose-rtk-provider -->|PointOne RTK| are-all-pointone-sites-affected
are-multiple-zip-sites-offline -->|Multiple sites| do-sites-share-behavior
are-multiple-zip-sites-offline -->|Single site| check-rtcm3-messages
do-sites-share-behavior -->|Yes| escalate-multiple-site-networking
do-sites-share-behavior -->|No| triage-sites-independently
triage-sites-independently --> check-rtcm3-messages
escalate-multiple-site-networking --> what-did-it-networking-confirm
what-did-it-networking-confirm -->|RTK details cannot be found...| resolve-missing-peplink-stations
what-did-it-networking-confirm -->|Not an IT issue| check-cloud-metric-changes
resolve-missing-peplink-stations --> return-after-networking-resolution
check-cloud-metric-changes --> return-after-utm-resolution
check-rtcm3-messages --> are-rtcm3-messages-present
are-rtcm3-messages-present -->| 7 RTCM3 messages| are-corrections-stable
are-rtcm3-messages-present -->| 7 RTCM3 messages| check-recent-site-work
are-corrections-stable -->|Yes| site-may-return-to-service
are-corrections-stable -->|No| wait-and-confirm-corrections
wait-and-confirm-corrections --> network-or-power-issue
check-recent-site-work -->|A change was made| revert-site-change
check-recent-site-work -->|No change was made| network-or-power-issue
revert-site-change -->|Yes| site-may-return-to-service
revert-site-change -->|No| network-or-power-issue
network-or-power-issue --> verify-networking-resolved
are-all-pointone-sites-affected -->|All sites are affected| escalate-pointone-fleetwide
are-all-pointone-sites-affected -->|Site-specific outage| open-rtk-corrections-dashboard
open-rtk-corrections-dashboard --> is-polaris-base-station-up
is-polaris-base-station-up -->|Yes| is-polaris-uptime-above-85
is-polaris-base-station-up -->|No| pointone-network-or-power-issue
is-polaris-uptime-above-85 -->|Yes| are-client-stream-outages-below-20
is-polaris-uptime-above-85 -->|No| pointone-network-or-power-issue
are-client-stream-outages-below-20 -->|Yes| polaris-base-station-working-properly
are-client-stream-outages-below-20 -->|No| pointone-network-or-power-issue
pointone-network-or-power-issue --> is-pointone-station-poe-based
is-pointone-station-poe-based -->|Yes| escalate-pointone-networking
is-pointone-station-poe-based -->|No| escalate-pointone-cell-coverage
escalate-pointone-networking --> verify-networking-resolved
escalate-pointone-cell-coverage --> replace-pointone-power-adapter
verify-networking-resolved -->|Yes| resume-flights
verify-networking-resolved -->|No| request-onsite-gsso
request-onsite-gsso --> do-rj45-tests-pass
do-rj45-tests-pass -->|Yes| reterminate-first-cat6a
do-rj45-tests-pass -->|No| check-poe-coupler
check-poe-coupler --> open-junction-box
open-junction-box --> inspect-cables-with-rj45-tester
inspect-cables-with-rj45-tester -->|Pass| is-onsite-issue-resolved
inspect-cables-with-rj45-tester -->|Fail| reterminate-junction-cat6a
reterminate-junction-cat6a --> did-junction-retest-pass
did-junction-retest-pass -->|Pass| is-onsite-issue-resolved
did-junction-retest-pass -->|Fail| repeat-junction-retermination
repeat-junction-retermination --> is-onsite-issue-resolved
reterminate-first-cat6a --> did-first-retest-pass
did-first-retest-pass -->|Pass| check-power-and-ethernet
did-first-retest-pass -->|Fail| repeat-first-retermination
repeat-first-retermination --> check-power-and-ethernet
check-power-and-ethernet -->|Yes| resume-flights
check-power-and-ethernet -->|No| request-hardware-guidance
is-onsite-issue-resolved -->|Yes| resume-flights
is-onsite-issue-resolved -->|No| request-hardware-guidance
style check-noc-dashboard fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-station-unhealthy fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style report-dashboard-discrepancy fill:#fef3c7,stroke:#f59e0b,color:#78350f
style choose-rtk-provider fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style are-multiple-zip-sites-offline fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style do-sites-share-behavior fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style triage-sites-independently fill:#dcfce7,stroke:#22c55e,color:#14532d
style escalate-multiple-site-networking fill:#fef3c7,stroke:#f59e0b,color:#78350f
style what-did-it-networking-confirm fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style resolve-missing-peplink-stations fill:#fef3c7,stroke:#f59e0b,color:#78350f
style return-after-networking-resolution fill:#dcfce7,stroke:#22c55e,color:#14532d
style check-cloud-metric-changes fill:#dcfce7,stroke:#22c55e,color:#14532d
style return-after-utm-resolution fill:#dcfce7,stroke:#22c55e,color:#14532d
style check-rtcm3-messages fill:#dcfce7,stroke:#22c55e,color:#14532d
style are-rtcm3-messages-present fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style are-corrections-stable fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style wait-and-confirm-corrections fill:#fef3c7,stroke:#f59e0b,color:#78350f
style check-recent-site-work fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style revert-site-change fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style network-or-power-issue fill:#fef3c7,stroke:#f59e0b,color:#78350f
style are-all-pointone-sites-affected fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style escalate-pointone-fleetwide fill:#fef3c7,stroke:#f59e0b,color:#78350f
style open-rtk-corrections-dashboard fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-polaris-base-station-up fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style is-polaris-uptime-above-85 fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style are-client-stream-outages-below-20 fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style polaris-base-station-working-properly fill:#dcfce7,stroke:#22c55e,color:#14532d
style pointone-network-or-power-issue fill:#dcfce7,stroke:#22c55e,color:#14532d
style is-pointone-station-poe-based fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style escalate-pointone-networking fill:#fef3c7,stroke:#f59e0b,color:#78350f
style escalate-pointone-cell-coverage fill:#fef3c7,stroke:#f59e0b,color:#78350f
style replace-pointone-power-adapter fill:#dcfce7,stroke:#22c55e,color:#14532d
style verify-networking-resolved fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style request-onsite-gsso fill:#dcfce7,stroke:#22c55e,color:#14532d
style do-rj45-tests-pass fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style check-poe-coupler fill:#dcfce7,stroke:#22c55e,color:#14532d
style open-junction-box fill:#dcfce7,stroke:#22c55e,color:#14532d
style inspect-cables-with-rj45-tester fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style reterminate-junction-cat6a fill:#dcfce7,stroke:#22c55e,color:#14532d
style did-junction-retest-pass fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style repeat-junction-retermination fill:#dcfce7,stroke:#22c55e,color:#14532d
style reterminate-first-cat6a fill:#dcfce7,stroke:#22c55e,color:#14532d
style did-first-retest-pass fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style repeat-first-retermination fill:#dcfce7,stroke:#22c55e,color:#14532d
style check-power-and-ethernet fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style is-onsite-issue-resolved fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style request-hardware-guidance fill:#dcfce7,stroke:#22c55e,color:#14532d
style resume-flights fill:#dcfce7,stroke:#22c55e,color:#14532d
style site-may-return-to-service fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
1 action
Check the NOC Dashboard when it shows that an RTK station is unhealthy.
2 question
Is the RTK station unhealthy based on the alarms shown on the Grafana dashboard? Check the PointOne graph and check the ZipRTK graph.
3 escalation
Escalate to ASW Delivery Support On Call and report that the NOC Dashboard differs from the Grafana dashboard.
4 question
Which RTK provider is offline?
5 question
Are multiple Zip RTK sites offline?
7 action
Treat the sites as individual incidents and triage each independently.
8 escalation
Tag
@it-networking-oncall to check if the stations are up and aren't reporting data to the cloud. 9 question
What did IT Networking confirm?
10 escalation
If the RTK stations are not shown on any of the Peplink dashboards, this could be a switch issue. Have IT Networking On Call resolve the issue.
11 action
Once IT Networking resolves the issue and the NOC Dashboard and Grafana dashboard show healthy RTK signals, the site may return to service.
12 action
Tag
@utm-on-call to check whether any changes were made that prevent the cloud from receiving metrics. 13 action
The site may return to service once
@utm-on-call resolves the issue and confirms site operations. 14 action
Check the Rtcm3 Messages per second graph for the site in the RTK Corrections Dashboard.
15 question
Are RTCM3 messages being reported?
16 question
Are you seeing stable Nominal Corrections for at least 30 minutes?
17 escalation
Wait 30 minutes and confirm whether the nominal corrections have been stable. Switch to the <7 RTCM3 messages path if the value drops during this 30-minute period and begin the triage process. If the RTK station is oscillating between 7 RTCM3 and <7 RTCM3 messages in a given 30-minute period, escalate to
@it-networking-oncall. There is likely an issue with network or power going to the base station; have them verify the status on the Peplink switch. 18 question
Check Zerp for work orders against this site or its towers, or against the RTK asset, within the last 24 hours. Check with MX teams and review work orders against the site or asset.
19 question
Tag metro specific GSSO-on-call. All metro specific GSSO-on-call can be found here. Have them determine what change was made onsite and revert the change. Is the issue resolved?
20 escalation
There is likely an issue with network or power going to the base station. Escalate to IT Networking On Call to check the status on the Peplink switch. Have them power cycle the port via the PepLink dashboard and have them check for any CRC errors.
21 question
Are all PointOne RTK sites affected, or is this a site-specific outage?
22 escalation
Escalate to P2 Nav On Call.
23 action
Use the RTK Corrections Dashboard to check whether PointOne RTK issues are resolved.
24 question
Is that specific site's Polaris Base Station status reported as Up?
25 question
Is that specific site's Polaris Uptime % greater than 85% within the last 10 minutes?
26 question
Is that specific site's Polaris Client Stream Detected Outages count less than 20 for the last 10 minutes?
27 action
The POLARIS base station is currently working properly and flights can resume. If you are still seeing RTK-related issues, escalate to P2 Nav On Call because there could be GNSS interference in the area.
28 action
There is likely an issue with network or power going to the base station.
29 question
Is this a PoE-based PointOne station? Find this information in the site handoff documents. All 12DS sites have PoE. All Walmart sites are DDS sites and the type of station can be found in the Handoff documents. All information related to 12DS sites can be found here.
30 escalation
Escalate to IT Networking On Call to check the status on the Peplink switch.
31 escalation
Tag
@p2-nav-on-call. They will reach out to Point One to confirm cell coverage and whether the frequency needs to be reduced. If there are no improvements once these changes are made, have the Point One team confirm if the power source has failed. If they have confirmed that this is indeed a Power Supply failure, issue a work order against this asset to get the AC adapter replaced. Tag metro specific GSSO-on-call. All metro specific GSSO-on-call can be found here. 32 action
Wait for instructions from
@p2-nav-on-call. If they confirm the station is up and running, then the issue is resolved and you can resume flights. 33 question
Verify whether IT Networking On Call resolved the issue. Check the Grafana dashboard and the NOC Dashboard to confirm the stations are up and healthy.
34 action
Tag metro specific GSSO-on-call. All metro specific GSSO-on-call can be found here. They will need to go to the site and verify physical connections.
35 question
Inspect the cables for any water damage and confirm if the RJ45 cable terminations look clean. Use a RJ45 tester and test the cables. Do they fail visual inspection or fail RJ45 tests?
36 action
Check the PoE coupler and replace it if necessary.
37 action
Open the junction box located at the base of the RTK tower or the 12DS site.
38 question
Inspect the cables and verify connection and continuity with an RJ45 tester.
39 action
Re-terminate the CAT6A cable and re-test the RJ45 connector.
40 question
Did the RJ45 re-test pass?
41 action
Re-terminate the CAT6A cable and re-test the RJ45 connector until the test passes.
42 action
Re-terminate the CAT6A cable and re-test the RJ45 connector.
43 question
Did the RJ45 re-test pass?
44 action
Re-terminate the CAT6A cable and re-test the RJ45 connector until the test passes.
45 question
Check with IT Networking On Call whether they can see power and ethernet on the port. Power-cycle the port if necessary. Check the Grafana dashboard and the NOC Dashboard to confirm the stations are up and healthy. Is IT Networking able to see power and ethernet on the port?
46 question
Is the issue resolved?
47 action
Reach out to
Site-Auxiliary-On-Call to check whether the hardware needs to be changed. 48 action
The issue is resolved and you can resume flights.
49 action
The site may return to service.
Resolution note
This plaintext block is ready to paste into the Resolution Note field on the ZERP issue you're working. The permalink below gives engineering review access to the full step-by-step timeline including any back-navigation; share it if the case needs a deeper look.
Permalink:
Session is too large to fit in a URL. Use Download session JSON to share the timeline out-of-band.
Did this guide help you get unblocked?
Open suggestions
There are no open suggestions.