The drawbacks of local packet captures

Probably the most common way of capturing network data is not a decision between SPAN or TAP – it is Wireshark simply being installed on one of the computers that need to be analyzed. While this an easy way to capture network packets it is also an easy way to get “wrong” results, because there are a lot of side effects when capturing packets directly on a computer. I discussed a lot of these side effects in my Sharkfest 2013 talk “PA-14: Top 5 False Positives” already, but let’s go check them out again.

Sideeffect #1 – Changing the problem environment

There are a couple of reasons why someone would run Wireshark on a computer to see what packets are coming in and going out. Some are just interested in what kind of communication is going on and there is no need to get very exact results. Others are looking for a more or less very specific problem. In the latter case it is not a good idea to install Wireshark/tShark or any other capturing solution on the host, especially on Windows, but some of it also applies to *nix systems.

First of all, by installing WinPCAP you’re going to insert some deep level code into your network stack to pick up packets, and to capture the packets you’re going to set your network card into Promiscuous Mode. There are countless questions on the Wireshark Q&A site where someone is surprised that the system behaves differently when activating that mode, so that should tell you something about how the situation of the problem computer has changed by capturing packets on it. Sometimes the problem even goes away completely (which usually leads to the kinda funny question “is it a good idea to install Wireshark on all my clients to fix the problem for them, too?” – the answer is “Don’t even think about doing that!”), or the original problem gets worse.

Second, Windows systems often have a lot of “security” related software installed, e.g. VPN connectors, Personal Firewalls, HIDS/HIPS, and other funky stuff. These software often conflict with the packet capturing process, which may lead to results like seeing only incoming packets in the capture (documented in countless questions on Wireshark Q&A, again). As soon as you disable one or all those programs you often see both directions again.

Sideeffect #2 – Woah, BIG packets! And small ones, too!

One of the first questions I usually asked in my Wireshark classes was “what is the minimum size of an Ethernet frame? What is the maximum size?” – since it was a beginners class I got more or less interesting answers ranging from 10 to 255 for the minimum and up to several megabytes for the maximum. There were only a couple of classes where the students did get both right: 64 bytes minimum size, 1518 maximum size (there are some special cases with VLAN tags etc. boosting the maximum to 1522, but let’s not bother with those right now).
Anyway, take a look at this capture taken at a server:
LocalCaptureServerSample
As you can see there are lots of packets with a length much greater than 1518 bytes, and there are even greater packets later in the trace, going up to more than 65535 bytes in size. Now lets look at a capture taken at the same time, but on the client side:
LocalCaptureClientSample
No more huge packets – instead they’re now all 1514 bytes at most, which is the maximum size without the Ethernet FCS (which most analyzers do not keep). Instead, there are now packets less than 60 bytes, which would be the minimum without the FCS being shown.
If you look closely you can see a pattern:
  1. all outgoing packets ignore one or more rules about packet sizes
  2. all incoming packets seem to be okay
  3. if you enable CRC checks for the TCP layer you’ll also notice that there are lots of CRC errors, but once again, only for outgoing packets while incoming packets are okay:LocalCaptureChecksumError

To make matters worse, there are even situations where incoming packets appear to be oversized, so you can’t assume that only outgoing packets are affected. By the way, this is how the same capture looks if taken on the network via SPAN port, between client and server (no checksum errors anymore, and all frame sizes are fine):LocalCaptureSPANSession

Reasons for the side effects

You probably asked yourself by now: “why does all this happen? Why does Wireshark show outgoing (and sometimes incoming) packets differently than whats actually going on on the network?”

The answer is pretty simple: it’s because of system optimization techniques, like TCP Segmentation Offloading, CRC Calculation Offloading and others. Basically, the network card takes over tasks that used to be handled by the CPU: cutting payloads into segments which fit the packets, calculating the correct CRCs, padding small packets up to the minimum size (remember the 54 byte packet above?). When you capture on a local system all these optimizations happen after the packets were already recorded (e.g. with dumpcap, which Wireshark uses to capture packets in the background):

NIC Offloading Capture effect

NIC Offloading Capture effect

And that is not a problem of Wireshark, it happens for any capture tool, including TCPDump and all others. The capture process cannot grab the finalized packet because that is only happening right before it goes out onto the Wire. That is the reason why you can see the correct packets on the SPAN session, and on the receiving side.

But wait, wasn’t there a problem with capturing locally for receiving packets? Yes, because there are now features like the TCP chimney, where packets are handed up to the upper levels by the network card in larger chunks.

Solutions

Now you might be inclined to turn all those network optimizations off to avoid those side effects. While that will work you’re going to do two bad things when doing that:
  1. in a problem scenario, you’ve just drastically changed the network environment situation, so anything you diagnose may not be correct anymore
  2. the optimizations are useful because they relieve the CPU of additional stress, especially at Gigabit and higher speeds. So by turning the optimizations off you might get a situation that is even worse than normal

The real solution is this, and it is pretty simple: do not capture on local systems. Use SPAN ports and TAPs, if you have to. It’s the only way to get good readings. Unless you don’t care about frame sizes, timings, broken checksums and interference from other software you should not capture on a local system unless it’s the only way to get the data at all. Period.

 tl;dr: do not capture on a computer directly unless you understand the side effects and you can live with them.

Discussions — 9 Responses

  • Fredrik September 25, 2015 on 12:57 am

    This post helped me a lot. Thanks.

    Reply
  • Ernest November 22, 2016 on 8:24 am

    Thanks very much for posting, very helpful 🙂

    Reply

*