SSH to device works temporarily, then after some time fails with "No route to host."

I have a Jetson Nano edge device that runs Ubuntu 18.04 I wish to ssh to. Its connected to my home network wirelessly through a USB WiFi dongle. On my desktop I can ssh -v <user>@ip to the device for a period of time after the device boots up, but around thirty minutes later when I attempt I receive ssh: connect to host <IP> port 22: No route to host errors. Here is the output:

$ ssh -v <user>@192.168.0.11 OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 21: Applying options for * debug1: Connecting to 192.168.0.11 [192.168.0.11] port 22. debug1: connect to address 192.168.0.11 port 22: No route to host ssh: connect to host 192.168.0.11 port 22: No route to host 

When in the "unconnectable" state, pinging looks like:

PING 192.168.0.11 (192.168.0.11) 56(84) bytes of data. From 192.168.0.3 icmp_seq=1 Destination Host Unreachable From 192.168.0.3 icmp_seq=2 Destination Host Unreachable From 192.168.0.3 icmp_seq=3 Destination Host Unreachable 

Which is odd. It seems like I can ping once only. And when in the "connectable" state, pinging works fine.

When in the "unconnectable" state, I have to reboot my edge device and all is well again and I can connect for a period of time.

This same behavior is found on other networks as well. So I do not believe its a network issue. This behavior also occurs on Windows and Mac machines when attempting to ssh to the edge device. So my gut then tells me the problem lies on the edge device. I've also tried removing entries to known_hosts in the .ssh folder without success. And finally when in the bad state, the device is still connected to the Internet. I can ping google and ping the desktop computer from the edge device.

Update: I just noticed more odd behavior. After I use the edge device to ping the machine that I want to ssh from, I can then ssh to the edge device.

Edit: Here is traceroute output when in "unconnectable" state:

traceroute to 192.168.0.11 (192.168.0.11), 30 hops max, 60 byte packets 1 tower (192.168.0.3) 3050.223 ms !H 3050.170 ms !H 3050.141 ms !H 

1 Answer

Sounds like an arp issue. Your router advertises the mac addresses to requesting devices on the network only after the device broadcasts it's existence to the router. This information is unicast to other nodes on the network and stored in the local system's arp cache when requested for a certain (typically 60s) period of time.

The device is either not properly broadcasting to the network, so your local arp cache loses the information after a certain period of time and no longer knows how to connect to it, OR the router is not sending the arp data when requested, and thus there's an issue on the router itself with the arp cache.

So to figure this out, you'll want to check a couple of things:

1. Make sure the device's broadcast address is correct vis-a-vis the network settings. Typically this is the last octet in the prefix range of the network (e.g. 192.168.0.255/24)
2. Check the device and system's local arp cache before and after the issue. (this varies, on ubuntu it's arp -a to display what's in the local cache). Adjust things accordingly. If multi-homed, make sure the ip expected is associated with the right mac address in the arp table.
3. Check your router. Most likely if the broadcast is setup properly and arp is active on the device, the router is having an issue either sending out to other nodes on the network when requested or the router's arp cache duration is too low.

This is why your system can connect to it after you ping from the device as it re-establishes the layer 2 mac information with your system directly and it then updates the local arp table on the system in question, bypassing the router communication, with the device's mac address.

I would check the device to make sure it's advertising itself properly.

If you can't figure it out, you can add static arp entries to your system's arp table via:

arp -s or arp -f <filename>

I found this reference when confirming your error, and my assumptions, which might have other solutions as well:

UPDATE: I just realized you are connecting over wifi/wireless. You might want to make sure the signal is good once the problem starts. While I still think it's arp-related, wifi can be finicky and doesn't always behave properly. Your traceroute response time would lead me to think there's an issue w/ interference, but I might just be misreading how you performed that. 3050ms is a very long network response time.