Initial and ongoing evaluation and documentation are two strategies at the core of effective troubleshooting. These can be applied to both your Local Area Network (LAN) and the interconnection of your LAN with the Internet through your Internet Service Provider (ISP) interconnect device(s). Here are some key techniques I use, which help me to rapidly move up the tiers of support when a problem is encountered.
Essential Network Troubleshooting Tools
In some cases, we connect computers and other electronic devices using serial communications protocols such as Universal Asynchronous Receive Transmit (UART) and Inter-Integrated Circuit (I2C). But today it is very common to instead or also use the Internet Protocol (IP), whether just within a Local Area Network (LAN), as part of a Community Network, or as part of the Internet. Below are some of the key investigative tools we can use to assess these IP network connections.
- LED light found on wired Ethernet cards and switches.
- When lit, indicates that the card and switch are on and working properly and the Ethernet cable is plugged in and working properly.
- Operating system utility that shows network configuration.
- An IP address of 0.0.0.0 or one that starts with 169.254 indicates you are connected to the media but aren’t getting a proper IP address from the address server (usually the gateway router).
- Run from the command line or terminal application to test if you can communicate with another node.
- ping IP_ADDR or ping IP_NAME (e.g. ping google.com)
- Times of 1 ms or less are good on a local network; 10-50 ms or less are good on the Internet.
- Hint: ping different nodes to track down faulty component.
- Run from the command line or terminal application to test the performance of each router between nodes.
- Windows: tracert IP_NAME
- Linux: traceroute IP_NAME
- macOS: traceroute IP_NAME
- Smartphone apps available
- Asterisks (*) indicate failure or firewall
- Hop 1 is usually your router; hop 2 is your ISP
Web-based Internet connection speed test:
To access the command line:
- Windows: Click on the Start button, choose run, type in cmd in the open bar and hit the enter key.
- macOS: Click on the spotlight, type in terminal and choose Terminal under Applications.
- Linux: Click on the Start button, choose Terminal under Accessories.
Previously, we saw how the “data beanbag” could be passed between just two nodes, your laptop and the Raspberry Pi. But even there, the server and client VNC (Virtual Network Computing) software are standing in the circle along with the hardware (computers). The Ethernet cable served to carry the data beanbag between these players.
For most of our work on computers, smartphones, and other networked devices, the beanbag needs to pass between many more interconnect devices to get between the starting and ending node. And when gathering information from something like a web server, the server itself may first act as a client, dynamically collecting data from a range of other servers, again using interconnect devices, before packaging this into a web page that it passes back to you.
Let’s use some essential network troubleshooting tools to follow the data beanbag as it’s passed around the circle. Along the way, let’s do a little bit of testing to make sure our networking is performing as expected, and if not, where it might be failing. This is a great opportunity to work collaboratively.
Exercise: Explore Your Network Configuration
All computers have some form of network management software that you should be able to access to see the network configuration. This varies considerably between computer operating systems, and even between versions of the same operating system. And sometimes you need administrative access to actually have significant access. All the same, try to track down how you might do this on your computer. Pair up with someone using the same type of computer as yours and see if you can track it down. Take some notes.
Another way to do this is from the command line or terminal window. On the Raspberry Pi or on a Mac you would type in:
In response, you should see something like the following:
In the above ifconfig, you can see the Raspberry Pi has been assigned an eth0 (wired Ethernet) IP address of 192.168.45.1 and a wlan0 (wireless Ethernet) IP address of 172.16.107.205.
On a Microsoft Windows command line or PowerShell terminal you would type in:
In response, you should see something like the following:
In the above ipconfig, you can see the Microsoft Windows 10 laptop has been assigned a wired Ethernet IP address of 10.0.0.33 and a wireless Ethernet IP address of 192.168.137.1.
In the Raspberry Pi terminal window, you can also type in:
$ hostname –I
The output from that command should look something like the following:
In the above example, we see the Raspberry Pi has the IP addresses we found using the ifconfig: 192.168.45.1 and 172.16.107.205. But in this case we cannot identify which are wired Ethernet and which are wireless Ethernet. We also cannot define using hostname the new listed IP address of 169.254.206.69 (this is actually part of the private IP address range used for Automatic Private IP Addressing, or APIPA, a failsafe used by some devices when no other IP address is assigned to a device).
Whether you run ifconfig on a Mac, or ipconfig on Windows, compare the outputs with those you see on your Raspberry Pi to develop a first picture of your LAN. Indeed, you can find a range of apps for your smartphone to run similar commands to identify the IP addresses these are also assigned by the LAN’s router or other Dynamic Host Configuration Protocol Server (DHCPS).
There are a couple of other helpful Linux commands to further track down information about your LAN, including the Local Area Network IP Address assigned to the router which then connects you to the your Internet Service Provider (ISP), including “netstat” and “ip route”:
$ netstat -nr
$ ip route | grep default
Exercise: Test the Network Node-to-Node Path
When your request for a web page is sent to a web server, and when the web server sends back that web page, the information is broken into a bunch of much smaller “data beanbags” called packets. While packets often take the same route along the circle between starting and ending nodes, each packet actually might change its course around the circle, depending on how the different interconnect devices it must pass through are performing at a given moment.
Let’s imagine I’m on an extended vacation, traveling by bicycle, and that I’m doing this without a physical or software-based map of any kind. I just know the name of my starting point and ending point, and I have a general sense of direction to get there.
To start, I might go to the local bike store/club and ask for suggestions. They may give me a perfect set of roads to get me from here to some intermediary destination that they’ve biked before. Great! Off I go. On arriving at that intermediary location, I’ll find the local bike store/club and get a new set of directions. They may mention that the usual route is preferred, but that there’s some construction and so it’s more of a rough ride. As a result, they recommend an alternate route. Great! Off I go again. As I near my destination town, I need to find one more stop-off site to get directions to reach my goal destination: Beer, Bed and Breakfast. With directions in hand, I complete the ride.
This is the way of internets. While some interconnect devices, for instance your home gateway router, have only one way in and one way out if you are to leave one network and join another, the larger routers that make up an Intranet or Internet can provide multiple different routes in case of decreased or failed performance of one route at the given moment.
There are three widely used pieces of software to do network testing. Two, ping and speed test, give you a general overview of the connection between two nodes. But the third, traceroute, will show you each of the router hops between two nodes.
Ping started as a text-based software installed by default on all major computers. There are now some graphical ping apps available, including ones for smartphones. First, if you’ve identified your gateway router, such as the gateway of 10.0.0.1, let’s use ping to test that connection. On your laptop, open a terminal (Mac) or cmd (Windows) window and type:
$ ping 172.16.96.1
The command ping sends a ping to a destination node. If all is good, the destination node returns back a ping. Well-functioning LANs should have times listed of less than 1 ms, and should show 0% loss.
On the Mac and Raspberry Pi computers, the default is to continue sending, and hopefully receiving, test pings until you hit CTRL-c. On Windows computers, the default is to run four test pings, and then to stop automatically.
Let’s test ping, this time from the Raspberry Pi. Open up a terminal on the Pi and do a ping to some of these sites (hit CTRL-c between each, or open up three terminal windows and run the three simultaneously):
$ ping illinois.edu $ ping adafruit.com $ ping raspberrypi.org
Ping sends the information from the node on which you are running it to a destination node, which then returns a ping. You will be told where the ping is returned from if all is well. It will also return a sequence number, and the time in milliseconds (ms) it took to go there and back again. In general, times under 10 ms occur on LANs and ultrafast WANs. Times above 10 ms but under 100 ms are typically found on fast WANs. Times over 100 ms reflect slower WANs, or networks of all types not working at expected speeds.
Sometimes the failure to get successful pings does not represent a failure in the network, but rather represents a security firewall that does not permit ping traffic to pass to and fro along a certain portion of the network path.
Speedtest is a general descriptor for websites such as the M-Lab Speed Test, Ookla Speedtest, and Speakeasy Speed Test, each of which are useful tools for testing a network’s speed and connectivity. When troubleshooting problems with your Internet Service Provider, they may ask you to employ such a tool. As a default, these sites don’t let you select a destination node to which you will test the network path. But by collecting an extended set of speedtests, it becomes possible to see if the upload and download speeds change at certain times or under certain conditions.
Speedtests generally provide the following key data points:
- Ping, which is identical to what we get from the command line ping application;
- Download, the speed in Mbps that describes the rate of data, measured as millions of bits (0 or 1) passed per second, traveling from a remote destination to your own location.
- Upload, the speed in Mbps of data traveling from you to the remote site.
The United States Federal Communications Committee (FCC) has a Lifeline program in which low-income households are assured ways to get low-cost Internet to the home. In 2020, the minimum download/upload speed has been set to either 25/3 or 100/10. But often promised IP speeds are much higher than the actual speeds at which networks are currently performing. Speedtests provide a very helpful measurement tool in which to collect baseline and current data when filing reports on real Internet access.
Traceroute is another text-based software installed by default on all major computers. Depending on operating system, the program will use one of several underlying network protocols. As a result, some succeed more than others at going through the full set of hops, as sometimes security firewalls are set up to stop one type of network protocol or another at some hop along the way. But while results vary a little, overall traceroute is an essential troubleshooting tool for networking.
One key point before doing the following activities using traceroute: While on Mac and Linux computers you type in
traceroute in the terminal window, on Windows computers you type in
tracert in the command line window.
To start, open two terminals on the Raspberry Pi. In one terminal, run ping 10 times and stop:
$ ping -c 10 illinois.edu
In the second terminal, run traceroute to the same location:
$ traceroute illinois.edu
Whereas ping does a general test from one node to another independent of the routers along the way, traceroute does a test of each hop between routers between the starting node and ending node. Typically, traceroute does three tests at each hop.
- The first hop indicates the time it took to pass from the starting node to the first, gateway router that moves us from the LAN to the next network between that node and the destination node.
- The second hop indicates the time it took between the gateway router and the second router in the path. For small to mid-sized LANs, this is usually the hop that takes us from the LAN to the ISP. For MANs and CANs, this is often the next hop in this community-based network.
- Subsequent hops move us further along in a MAN, CAN, or WAN until…
- The final hop brings us to the destination node.
When troubleshooting using traceroute, the first hop is a great indicator as to whether the problem is somewhere on our LAN and needs our attention or the attention of our internal technical support team. The second hop often means we need to contact either our ISP or our campus or neighborhood technical support team. Generally, the further we go along, the more the troubleshooting moves out of our hands.
To review, the first hop typically indicates an in-building issue, the second hop a problem with our direct network provider, and further hops a problem outside our realm of control. But if we do a traceroute test of just one destination node, it’s still questionable where on the path the problem is occurring.
To this end, traceroute really shines when we perform multiple traceroute tests. We can do this on one node, say a laptop, going to a multiple different sites, some more local and some more distant. We can take it a step further by repeating the range of traceroute tests on different nodes, say on two different laptops. We can take it a step further still by repeating the range of traceroute tests on a node connected to one network and then to another, say on a node connected to the LAN WiFi, then to your local cellular service. (That last test could be done using a smartphone that can connect to either, or by using the smartphone as a hotspot allowing a laptop to connect to the Internet through cellular.)
For this next test, open one terminal or cmd window on your laptop and two terminals on the Raspberry Pi. As a bonus, if you can, open a traceroute app or terminal that can run traceroute on your smartphone. Run the following traceroute tests on each device, and in the second terminal on the Raspberry Pi, a ping to those sites. Compare and contrast the results.
$ traceroute illinois.edu $ ping illinois.edu $ traceroute zoom.com $ ping zoom.com $ traceroute adafruit.com $ ping adafruit.com $ traceroute raspberrypi.org $ ping raspberrypi.org
Compare the different number of hops, the performance of hops, and even some of the possible router location identifiers you find. For instance, below are two traceroute results, one going from my home office to adafruit.com, the other to raspberrypi.org. In both cases, the first hop is the the home’s gateway router with IP address of 10.0.0.1. The second hop is the ISP’s router, 188.8.131.52. The next two hops go through routers with firewalls and so do not give performance reports (I know this because it is consistent over time and over destination).
In the adafruit.com example, the next three hops travel along the ntt.net backbone within the Chicagoland area before getting to a destination router with IP address 184.108.40.206. This means Adafruit Industries has a server themselves or a provider with a location in the Chicagoland area. All Internet tests are under or just above 10 ms, indicating a high performing Internet connection at every router interconnect device between my house and the site hosting the Adafruit.com server.
In the raspberrypi.org example, we see a journey of routers that moves from the Chicagoland area to somewhere else, possibly VA, that is, Virginia, in the United States, before going to The Netherlands, likely in Amsterdam, before ending up with Mythic Beasts networks, a privately owned hosting ISP. In this case, note that once travel moves from Chicago to the East Coast of the United States, travel speeds move from below 10 ms to 20-30 ms. Then as the route passes over the Atlantic Ocean, it jumps up again to the lower 100 ms performance level. This is much better performance than even a few years ago, and is an example of why it is possible to have more audio and video communications than ever before!
But how does this compare to your own Internet service?
Using ping, speedtest, and traceroute, you will build a strong portfolio of normal and problem network performance data for use when things go wrong on your network. Combining a continuous ping written to a file with regular timestamps, you can have a clear picture of when and at what level network performance is impacted. Speedtest provides a second tier of data, and is often the data ISPs want to see themselves.
Using traceroute, some tests will only have one hop as they stay on the local network. Other tests will have multiple hops as they leave one local area network and move to another, and perhaps yet more local area networks until they arrive at the destination local area network. Routers serve as the go-between, passing between local area networks. Intranets and the Internet are simply multiple interconnected local area networks.
Usually, hop one in a traceroute is from your node to the interconnect device (such as a gateway) in the building hosting your local area network. A decrease or failure in network performance in hop one generally rests on you to troubleshoot.
Hops two and three are more variable, and may indicate a problem on your Intranet or the hop between you and your Internet Service Provider (ISP). If the former, the troubleshooting onus may still be yours, or that of your IT support team. If the problem is at the level of connectivity between your network and the ISP, then it might be time to give them a call. Or, you might need to first collect data on performance variability over time, as ISPs can often shift the burden of testing back on you.
Regardless, the expected results are that you will increase your level of success at troubleshooting.
When information and communications data sent is sent from the node and onto the LAN and often to the WAN, it is not sent as a single file, such as that we store on a hard drive or flash drive. Rather, it is divided into pieces, called packets. Each packet contains a header and a payload. The header contains information including the source IP address, the destination IP address, and an identification tag to order the packets. The payload is the actual information that is being transmitted, such as web page data or an email. When a packet reaches its destination, the software application will piece all the packets together.