The drawbacks of local packet captures
Probably the most common way of capturing network data is not a decision between SPAN or TAP – it is Wireshark simply being installed on one of the computers that need to be analyzed. While this an easy way to capture network packets it is also an easy way to get “wrong” results, because there are a lot of side effects when capturing packets directly on a computer. I discussed a lot of these side effects in my Sharkfest 2013 talk “PA-14: Top 5 False Positives” already, but let’s go check them out again.
Sideeffect #1 – Changing the problem environment
There are a couple of reasons why someone would run Wireshark on a computer to see what packets are coming in and going out. Some are just interested in what kind of communication is going on and there is no need to get very exact results. Others are looking for a more or less very specific problem. In the latter case it is not a good idea to install Wireshark/tShark or any other capturing solution on the host, especially on Windows, but some of it also applies to *nix systems.
First of all, by installing WinPCAP you’re going to insert some deep level code into your network stack to pick up packets, and to capture the packets you’re going to set your network card into Promiscuous Mode. There are countless questions on the Wireshark Q&A site where someone is surprised that the system behaves differently when activating that mode, so that should tell you something about how the situation of the problem computer has changed by capturing packets on it. Sometimes the problem even goes away completely (which usually leads to the kinda funny question “is it a good idea to install Wireshark on all my clients to fix the problem for them, too?” – the answer is “Don’t even think about doing that!”), or the original problem gets worse.
Second, Windows systems often have a lot of “security” related software installed, e.g. VPN connectors, Personal Firewalls, HIDS/HIPS, and other funky stuff. These software often conflict with the packet capturing process, which may lead to results like seeing only incoming packets in the capture (documented in countless questions on Wireshark Q&A, again). As soon as you disable one or all those programs you often see both directions again.
Sideeffect #2 – Woah, BIG packets! And small ones, too!
- all outgoing packets ignore one or more rules about packet sizes
- all incoming packets seem to be okay
- if you enable CRC checks for the TCP layer you’ll also notice that there are lots of CRC errors, but once again, only for outgoing packets while incoming packets are okay:
To make matters worse, there are even situations where incoming packets appear to be oversized, so you can’t assume that only outgoing packets are affected. By the way, this is how the same capture looks if taken on the network via SPAN port, between client and server (no checksum errors anymore, and all frame sizes are fine):
Reasons for the side effects
You probably asked yourself by now: “why does all this happen? Why does Wireshark show outgoing (and sometimes incoming) packets differently than whats actually going on on the network?”
The answer is pretty simple: it’s because of system optimization techniques, like TCP Segmentation Offloading, CRC Calculation Offloading and others. Basically, the network card takes over tasks that used to be handled by the CPU: cutting payloads into segments which fit the packets, calculating the correct CRCs, padding small packets up to the minimum size (remember the 54 byte packet above?). When you capture on a local system all these optimizations happen after the packets were already recorded (e.g. with dumpcap, which Wireshark uses to capture packets in the background):
And that is not a problem of Wireshark, it happens for any capture tool, including TCPDump and all others. The capture process cannot grab the finalized packet because that is only happening right before it goes out onto the Wire. That is the reason why you can see the correct packets on the SPAN session, and on the receiving side.
But wait, wasn’t there a problem with capturing locally for receiving packets? Yes, because there are now features like the TCP chimney, where packets are handed up to the upper levels by the network card in larger chunks.
- in a problem scenario, you’ve just drastically changed the network environment situation, so anything you diagnose may not be correct anymore
- the optimizations are useful because they relieve the CPU of additional stress, especially at Gigabit and higher speeds. So by turning the optimizations off you might get a situation that is even worse than normal
The real solution is this, and it is pretty simple: do not capture on local systems, or any system that processes the packets on the way. Use SPAN ports or TAPs to send copies of the packets to a receive-only capture device instead. It’s the only way to get good readings. Unless you don’t care about frame sizes, timings, broken checksums and interference from other software you must not capture on a local system unless it’s the only way to get the data at all. Period.
tl;dr: do not capture on a computer, router, firewall or any network packet processing device directly unless you understand the side effects and you can live with them.
This post helped me a lot. Thanks.
Thanks very much for posting, very helpful 🙂
very useful information, thanks alot. keep writing.
thank you so much