Live Diagnose a Zip with Hawkeye
Viewing this guide for
Related Search Phrases
Section titled “Related Search Phrases”This issue may also be described as live IPC, Hawkeye live, bench testing, smartcart Hawkeye, Zip IPC visualization, real-time diagnostics, or Tailscale Hawkeye.
Background
Section titled “Background”Hawkeye is a native desktop visualization tool for inspecting ZipIPC messages in real time. It can connect directly to a Zip’s IPC server over TCP to display live telemetry, time-series plots, and 3D visualizations — useful for diagnosing communication issues, verifying sensor data flow, or debugging service behavior on the bench.
For a browser-based alternative that connects through a Smart Cart, see Using ZipViz for Live Diagnostics.
There are two connection methods:
- Method 1 — Tailscale (preferred): Enable Tailscale on the Zip so your laptop can reach its IPC server directly over VPN. You can SSH into the Zip however you like (directly, through a smart cart, etc.) to set this up.
- Method 2 — SSH Port Forwarding via Smart Cart: Forward the Zip’s IPC port through a smart cart to your laptop. Use this when Tailscale is not an option. Requires the Zip to be connected to a smart cart.
Important version requirement: Your local Hawkeye build (from develop) must include PR #45679 (“Fix Hawkeye metadata fetching”). Without this fix, Hawkeye will not receive topic metadata from Zips running fsw-2026.03 or later, due to the lazy metadata change in PR #42673.
Prerequisites:
- macOS dependencies installed:
brew install ffmpeg gdal opencv pkg-config - FlightSystems repo checked out on
develop(must include the metadata fix above) - Tailscale installed on your laptop (for Method 1)
- SSH access to the Zip (directly or through a smart cart/dock)
Troubleshooting Steps
Section titled “Troubleshooting Steps”graph TD
tailscale-available{"<small><b>Step 1</b></small><br/>Question: Is Tailscale available for use on the..."}
method-1-tailscale-setup["<small><b>Step 2</b></small><br/>Action: **Use Method 1: Tailscale** 1. **SSH ..."]
tailscale-ping-check{"<small><b>Step 3</b></small><br/>Question: Can you ping the Zip from your laptop?"}
verify-tailscale-before-method-2["<small><b>Step 4</b></small><br/>Action: Verify Tailscale is running on your l..."]
zip-ipc-service-check{"<small><b>Step 5</b></small><br/>Question: **Is the Zip IPC service running?** O..."}
start-zip-ipc-service["<small><b>Step 6</b></small><br/>Action: Start the service and flight stack: ..."]
run-hawkeye-tailscale{"<small><b>Step 7</b></small><br/>Question: **Build and run Hawkeye.** On your la..."}
tailscale-cleanup["<small><b>Step 8</b></small><br/>Action: Select topics relevant to your invest..."]
method-2-smart-cart-setup{"<small><b>Step 9</b></small><br/>Question: **Use Method 2: SSH Port Forwarding v..."}
smart-cart-tunnel-not-reaching-zip["<small><b>Step 10</b></small><br/>Action: The tunnel is not reaching the Zip. V..."]
smart-cart-laptop-tunnel{"<small><b>Step 11</b></small><br/>Question: 3. **Open a tunnel from your laptop t..."}
close-ssh-tunnels["<small><b>Step 12</b></small><br/>Action: Select topics and begin visualization..."]
metadata-timeout-check{"<small><b>Step 13</b></small><br/>Question: **No topics shown or metadata timeout..."}
metadata-timeout-fix["<small><b>Step 14</b></small><br/>Action: Your Hawkeye build may be missing the..."]
connection-refused-fix["<small><b>Step 15</b></small><br/>Action: If Hawkeye shows 'Connection refused'..."]
smart-cart-sync-only-check{"<small><b>Step 16</b></small><br/>Question: **Smart cart tunnel only shows ^sync..."}
smart-cart-sync-only-fix["<small><b>Step 17</b></small><br/>Action: You are connected to the smart cart's..."]
metadata-mismatch-fix["<small><b>Step 18</b></small><br/>Action: If you see identifiers but Hawkeye sh..."]
ssh-host-key-check{"<small><b>Step 19</b></small><br/>Question: **SSH host key changed** Did you rece..."}
ssh-host-key-fix["<small><b>Step 20</b></small><br/>Action: Zips are frequently reimaged, changin..."]
ssh-network-check["<small><b>Step 21</b></small><br/>Action: If SSH hangs or times out, verify net..."]
tailscale-available -->|Yes| method-1-tailscale-setup
tailscale-available -->|No| method-2-smart-cart-setup
method-1-tailscale-setup --> tailscale-ping-check
tailscale-ping-check -->|Yes| zip-ipc-service-check
tailscale-ping-check -->|No| verify-tailscale-before-method-2
verify-tailscale-before-method-2 --> method-2-smart-cart-setup
zip-ipc-service-check -->|Yes| run-hawkeye-tailscale
zip-ipc-service-check -->|No| start-zip-ipc-service
run-hawkeye-tailscale -->|Yes, topics loaded| tailscale-cleanup
run-hawkeye-tailscale -->|No topics shown or metadata...| metadata-timeout-check
run-hawkeye-tailscale -->|Smart cart tunnel only show...| smart-cart-sync-only-check
run-hawkeye-tailscale -->|SSH host key changed| ssh-host-key-check
method-2-smart-cart-setup -->|Yes| smart-cart-laptop-tunnel
method-2-smart-cart-setup -->|No| smart-cart-tunnel-not-reaching-zip
smart-cart-laptop-tunnel -->|Yes, topics loaded| close-ssh-tunnels
smart-cart-laptop-tunnel -->|No topics shown or metadata...| metadata-timeout-check
smart-cart-laptop-tunnel -->|Smart cart tunnel only show...| smart-cart-sync-only-check
smart-cart-laptop-tunnel -->|SSH host key changed| ssh-host-key-check
metadata-timeout-check -->|Yes| metadata-timeout-fix
metadata-timeout-check -->|No| connection-refused-fix
smart-cart-sync-only-check -->|Yes| smart-cart-sync-only-fix
smart-cart-sync-only-check -->|No| metadata-mismatch-fix
ssh-host-key-check -->|Yes| ssh-host-key-fix
ssh-host-key-check -->|No| ssh-network-check
style tailscale-available fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style method-1-tailscale-setup fill:#dcfce7,stroke:#22c55e,color:#14532d
style tailscale-ping-check fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style verify-tailscale-before-method-2 fill:#dcfce7,stroke:#22c55e,color:#14532d
style zip-ipc-service-check fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style start-zip-ipc-service fill:#dcfce7,stroke:#22c55e,color:#14532d
style run-hawkeye-tailscale fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style tailscale-cleanup fill:#dcfce7,stroke:#22c55e,color:#14532d
style method-2-smart-cart-setup fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style smart-cart-tunnel-not-reaching-zip fill:#dcfce7,stroke:#22c55e,color:#14532d
style smart-cart-laptop-tunnel fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style close-ssh-tunnels fill:#dcfce7,stroke:#22c55e,color:#14532d
style metadata-timeout-check fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style metadata-timeout-fix fill:#dcfce7,stroke:#22c55e,color:#14532d
style connection-refused-fix fill:#dcfce7,stroke:#22c55e,color:#14532d
style smart-cart-sync-only-check fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style smart-cart-sync-only-fix fill:#dcfce7,stroke:#22c55e,color:#14532d
style metadata-mismatch-fix fill:#dcfce7,stroke:#22c55e,color:#14532d
style ssh-host-key-check fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
style ssh-host-key-fix fill:#dcfce7,stroke:#22c55e,color:#14532d
style ssh-network-check fill:#dcfce7,stroke:#22c55e,color:#14532d Hover or focus a block to preview it. Select the block to keep the preview open.
1 question
Is Tailscale available for use on the Zip?
2 action
Use Method 1: Tailscale
1. SSH into the Zip and enable Tailscale:
a. Get a shell on the Zip. Most commonly this is through a smart cart:
b. Once on the Zip, start the Tailscale daemon and log in:
The
2. Get the Zip's Tailscale IP and verify connectivity:
a. Once authenticated, get the Zip's Tailscale IP:
b. Note the IPv4 address (e.g.,
1. SSH into the Zip and enable Tailscale:
a. Get a shell on the Zip. Most commonly this is through a smart cart:
# Example: SSH through a smart cart to the Zip
ssh zipline@<smart-cart-hostname>
ssh_compute_ab. Once on the Zip, start the Tailscale daemon and log in:
sudo systemctl start tailscaled.service
sudo tailscale loginThe
tailscale login command will output an authentication URL. Open it in your browser and authenticate.2. Get the Zip's Tailscale IP and verify connectivity:
a. Once authenticated, get the Zip's Tailscale IP:
tailscale ipb. Note the IPv4 address (e.g.,
100.70.241.107). From your laptop, verify you can reach the Zip over Tailscale:ping <tailscale-ip> 3 question
Can you ping the Zip from your laptop?
4 action
Verify Tailscale is running on your laptop. Check that the Zip appears in your Tailscale admin console. If the Zip cannot establish a Tailscale connection, click Continue to use Method 2 (port forwarding via smart cart) instead.
5 question
Is the Zip IPC service running?
On the Zip, run:
Is the service
On the Zip, run:
systemctl status zip_ipc.serviceIs the service
active (running) and listening on tcp://:::10000? 6 action
Start the service and flight stack:
Verify
sudo systemctl start base.target
sudo systemctl start sleep.targetVerify
zip_ipc.service is now active. If it still fails to start, escalate to the FlightSystems IPC team. 7 question
Build and run Hawkeye.
On your laptop, from the FlightSystems repo root:
Flags:
-
-
Did topics load in the Hawkeye topic selection modal?
The initial metadata fetch can take 15-30 seconds over Tailscale. Wait for the "waiting for IPC metadata" message to clear.
On your laptop, from the FlightSystems repo root:
cargo run --release -p hawkeye -- -t unix tcp://<tailscale-ip>:10000Flags:
-
-t unix - Use wall-clock timestamps (required for live connections)-
tcp://<tailscale-ip>:10000 - Direct TCP connection to the Zip's IPC serverDid topics load in the Hawkeye topic selection modal?
The initial metadata fetch can take 15-30 seconds over Tailscale. Wait for the "waiting for IPC metadata" message to clear.
8 action
Select topics relevant to your investigation and begin visualization. You can use templates (e.g., "Executive Status") for predefined topic sets.
When you are finished, you MUST complete the cleanup steps below.
CRITICAL: Cleanup After Tailscale Use
When you are finished with your Hawkeye session, you MUST perform the following steps on the Zip:
1. Stop the Tailscale daemon:
2. Verify Tailscale is no longer running:
Confirm output shows
3. Reboot the Zip to ensure it returns to a clean, known-good state. This guarantees that no residual Tailscale network configuration remains active:
Do NOT return the Zip to service without completing these steps.
When you are finished, you MUST complete the cleanup steps below.
CRITICAL: Cleanup After Tailscale Use
When you are finished with your Hawkeye session, you MUST perform the following steps on the Zip:
1. Stop the Tailscale daemon:
sudo tailscale down
sudo systemctl stop tailscaled.service2. Verify Tailscale is no longer running:
systemctl status tailscaled.serviceConfirm output shows
inactive (dead).3. Reboot the Zip to ensure it returns to a clean, known-good state. This guarantees that no residual Tailscale network configuration remains active:
sudo systemctl rebootDo NOT return the Zip to service without completing these steps.
9 question
Use Method 2: SSH Port Forwarding via Smart Cart
Use this method when Tailscale is not available on the Zip. The Zip must be connected to a smart cart so that you can forward the IPC port through SSH tunnels.
1. Open a tunnel from the smart cart to the Zip:
SSH into the smart cart, then create a tunnel to the Zip's IPC port. The Zip is typically reachable from the smart cart via its IPv6 link-local address:
Important: The smart cart runs its own IPC server (
2. Verify that the tunnel reaches the Zip:
In a second terminal on the smart cart:
Does
Use this method when Tailscale is not available on the Zip. The Zip must be connected to a smart cart so that you can forward the IPC port through SSH tunnels.
1. Open a tunnel from the smart cart to the Zip:
SSH into the smart cart, then create a tunnel to the Zip's IPC port. The Zip is typically reachable from the smart cart via its IPv6 link-local address:
ssh zipline@<smart-cart-hostname>
ssh -L 20001:localhost:10000 zipline@'fe80::50%enp3s0'Important: The smart cart runs its own IPC server (
dock_charging) on port 10000. Use a different local port (e.g., 20001) to avoid connecting to the smart cart's IPC instead of the Zip's.2. Verify that the tunnel reaches the Zip:
In a second terminal on the smart cart:
ss -tlnp | grep 20001
zip_ipc tcp://localhost:20001 listDoes
zip_ipc list show the Zip's full set of topic identifiers (not just ^sync.* entries)? 10 action
The tunnel is not reaching the Zip. Verify the Zip's IPv6 link-local address with
If you get SSH host key warnings, clear the old key:
Verify
ip -6 addr show enp3s0 on the smart cart.If you get SSH host key warnings, clear the old key:
ssh-keygen -f "$HOME/.ssh/known_hosts" -R 'fe80::50%enp3s0'Verify
zip_ipc.service is running on the Zip. 11 question
3. Open a tunnel from your laptop to the smart cart:
On your laptop:
This chains:
4. Run Hawkeye:
5. Did topics load?
On your laptop:
ssh -L 20001:localhost:20001 zipline@<smart-cart-hostname>This chains:
laptop:20001 -> smart-cart:20001 -> Zip:100004. Run Hawkeye:
cargo run --release -p hawkeye -- -t unix tcp://localhost:200015. Did topics load?
12 action
Select topics and begin visualization. When finished, close your SSH tunnels.
13 question
No topics shown or metadata timeout
Did Hawkeye display "timed out waiting for IPC metadata?"
Did Hawkeye display "timed out waiting for IPC metadata?"
14 action
Your Hawkeye build may be missing the
Also verify the flight stack is running on the Zip (
^metadata subscription fix. Pull the latest develop and rebuild:git pull origin develop
cargo run --release -p hawkeye -- -t unix tcp://<ip>:<port>Also verify the flight stack is running on the Zip (
systemctl status base.target should show active). 15 action
If Hawkeye shows "Connection refused" / os error 61, the Zip's IPC server is not running or not listening on TCP. SSH into the Zip and start the service:
sudo systemctl start zip_ipc.service
systemctl status zip_ipc.service 16 question
Smart cart tunnel only shows
Are you seeing only
^sync.* identifiersAre you seeing only
^sync.* and ^redirect topics with no actual data identifiers? 17 action
You are connected to the smart cart's own IPC server (
dock_charging) on port 10000, not the Zip. Verify your tunnel uses a different local port (e.g., 20001) and that the SSH tunnel to the Zip is active. Run zip_ipc tcp://localhost:20001 list on the smart cart to confirm you see Zip identifiers before connecting Hawkeye. 18 action
If you see identifiers but Hawkeye shows nothing, the metadata may have empty schemas. This indicates a version mismatch. Follow the metadata timeout troubleshooting path above.
19 question
SSH host key changed
Did you receive an SSH "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED" error?
Did you receive an SSH "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED" error?
20 action
Zips are frequently reimaged, changing their host keys. Remove the old key and reconnect:
ssh-keygen -f "$HOME/.ssh/known_hosts" -R '<zip-ip>' 21 action
If SSH hangs or times out, verify network connectivity (
ping <zip-ip>) and that the Zip is powered on.Resolution note
This plaintext block is ready to paste into the Resolution Note field on the ZERP issue you're working. The permalink below gives engineering review access to the full step-by-step timeline including any back-navigation; share it if the case needs a deeper look.
Permalink:
Session is too large to fit in a URL. Use Download session JSON to share the timeline out-of-band.
How Live IPC Works (Reference)
Section titled “How Live IPC Works (Reference)”- Hawkeye connects to the ZipIPC server as a TCP client named
hawkeye_listener. - The server sends control identifier metadata (
^sync.*,^start) on connection. - Hawkeye subscribes to
^metadata(opt-in since PR #42673). - The server dumps all registered topic schemas, followed by a
^metadataend marker. - Hawkeye parses the schemas and presents the topic selection UI.
- After topic selection, Hawkeye subscribes to the chosen topics and begins receiving live data.
Did this guide help you get unblocked?
Open suggestions
There are no open suggestions.
Open pull requests
There are no open pull requests.