The Network Capture Playbook Part 3 – Network cards
One of the most common answers that come to my mind when being asked questions during or after a talk at a conference is the famous phrase “it depends…”. This may sound unsatisfactory at first, but the problem with a lot of questions regarding network analysis (and packet capture) is that there are always so many things to consider. So when we’re talking about using a standard network card like they are built into most PCs and laptops these days, the answer to the question of “is it good enough to capture packets?” is – you probably guessed it already: “it depends”.
Capturing packets is one of those disciplines that are best described as “easy to learn, hard to master”. It’s relatively easy to capture packets with a PC or Mac using a built-in network card, but getting exactly what you need may be a bit more difficult or even turn into a real challenge. Let’s take a look at the basic things you need to consider when capturing packets from Ethernet (we’ll look at WiFi captures later):
- Access rights
- Destination MAC filters and “Promiscuous Mode”
- Passiveness
- Network card processing capabilities
- Packet mix and capture efficiency
- Pitfalls
Access Rights
This is something Windows users of Wireshark might wonder what the fuss is all about, while Linux users often encounter the frustrating situation of not being able to capture any packets unless they run Wireshark as root. The problem is that capturing packets is considered a security risk, as it may allow access to data the user shouldn’t have or isn’t allowed to have. Some companies even have a security rule that users aren’t allowed to run packet capture software on their laptops or PCs.
Windows
Wireshark (and most likely any other network analysis tool) on Windows solves the problem by creating a special capture service running with enough authorization to access available network cards. Wireshark (well, dumpcap, to be more specific) accesses the driver (called “NPF”), allowing packet captures even if Wireshark itself runs with normal user credentials that wouldn’t be good enough to do the same without NPF.
If you want to check the status of the NPF service, you won’t find it in the services list of Windows. Instead, you can check it using the “sc.exe” utility with the “query” command on a command prompt:
[C:\]sc query npf SERVICE_NAME: npf TYPE : 1 KERNEL_DRIVER STATE : 4 RUNNING (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0
The “sc” command is also used to “stop” or “start” the NPF service, but you need an administrative (“elevated”) command prompt to do that.
Linux
The situation on Linux is a bit different – instead of a service the user needs to be allowed to access the network card for capture tasks. How to do that may depend on the flavor of Linux that you’re running, but here are some links that may help to set up the required privileges:
https://wiki.wireshark.org/CaptureSetup/CapturePrivileges
https://blog.wireshark.org/2010/02/running-wireshark-as-you/
https://ask.wireshark.org/questions/7976/wireshark-setup-linux-for-nonroot-user
http://askubuntu.com/questions/74059/how-do-i-run-wireshark-with-root-privileges
And please remember: don’t run Wireshark (or any capture software) as root user if you can avoid it. The reason for this is simple: the packets you capture with a software running as root could exploit a weakness/bug if they are intentionally malformed by an attacker, and lead to a total system compromise. And since most network analysis tools are very complex beasts (take a look at the Wireshark source code if you don’t believe me), so it’s practically impossible for them to be bug free.
Destination MAC Filters and Promiscuous Mode
An Ethernet network card has a unique (well, at least in theory; let’s not go into this right here) hardware address, called “Media Access Control address”, or “MAC address”. It’s a 6 octet (=byte) value, and I won’t go into OIDs and other things here as well. The “problem” for capturing network packets is that an Ethernet network card in normal operation mode will only accept packets (well, “frames“, to use the correct layer 2 term) that have a destination address of the card’s MAC address, or are Broadcast/Multicast frames. This means that a network card will filter MAC addresses and completely ignore any frame that is sent between two device MACs that are different from it’s own, while the receiver network card will accept it:
This is a problem, since we want to avoid capturing on client or server at almost any cost – both of which would be able to accept the packet, of course. So instead we want to turn off that MAC address filter of the network card in the capture laptop. Doing that is called “enabling Promiscuous Mode” – the card now accepts any frame it receives, even if it is not the intended receiver:
So in a good capture setup, the capture device needs to have “Promiscuous Mode” enabled to be able to see frames other than it’s own, Broadcast and maybe Multicast. Wireshark by default enables it when you capture:
Passiveness
Capturing network packets requires a certain level of precision, otherwise the captured frames/packets aren’t going to be useful for any kind of performance or forensics analysis. We basically have two important requirements when capturing frames/packets:
- The capture is always taken with a device that isn’t part of the communication (dedicated capture device), unless you can afford to take a local capture and know all side effects, errors and problems that come with it. Remember, any device being part of the active communication (including firewalls, routers, load balancers, WAN accelerators, traffic shapers, proxies etc.) are off limits and considered “local” if you capture on them.
- The dedicated capture device must be strictly set to listen only. It must not, ever, send frames/packets into the network itself, messing up capture results.
Passiveness is a key feature on professional capture cards, to avoid injecting any data (“noise”) into the captured data. “Normal” network cards like those being part of laptop and computer mainboards are made for send and receive operations, so they tend to try to communicate themselves during a capture (e.g. DHCP address requests etc.). To “silence” the network card you can remove all protocol bindings so that the card can’t be used by the TCP/IP stack.
Windows
On Windows, you can simply open the card properties and disable (uncheck) all protocol bindings except the “Npcap Packet Driver (NPCAP)” (because this is required for capturing packets on the network card), e.g. like this:
Wireshark (and other capture tools) will still be able to use the card to capture packets, but it can’t send anything anymore.
Linux
For Linux I usually use the same settings the SecurityOnion distribution uses to put the card into promiscuous mode and disable all other card activities in the /etc/network/interfaces config file, e.g for eth1 (the main trick is setting a manual IP address without specifying one, enabling promiscuous mode and then using ethtool to turn off card features in a loop):
auto eth1 iface eth1 inet manual up ip link set $IFACE promisc on arp off up down ip link set $IFACE promisc off down post-up ethtool -G $IFACE rx 512; for i in rx tx sg tso ufo gso gro lro; do ethtool -K $IFACE $i off; done post-up echo 1> /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6
Network card processing capabilities
You should think of a network card as a computer within your computer. Sure, it’s a very restricted computer in it’s ability to receive, process and send network packets/frames. Processing performed by a network card may include:
- Calculating checksums (IP, TCP, UDP)
- splitting outgoing large data chunks into the size required for the physical network
- merging incoming packets into large data chunks to be transferred to the application layer
But you should also think of the network card as a gate keeper: the card has the power to decide which packets/frame to pass to the computer, and which ones to discard. With “discard” meaning that the computer never knows about the packets/frames ever arriving at the network card.
We have already seen that the card will discard any frame with the destination not matching the MAC address unless Promiscuous Mode is enabled. But what you also need to keep in mind is that a normal network card will discard all frames with a broken checksum or those that are too big (“oversize”), too small (“runt”), or otherwise damaged – and running in promiscuous mode won’t change that fact. Which means that you can’t troubleshoot layer 2 errors without professional capture cards, because only those may offer features to forward damaged frames to the computer itself. But then again layer 2 troubleshooting isn’t something you do with packet analysis anymore – it’s usually the task of cable testers or administrators looking at switch/router port statistics.
Another thing you need to keep in mind that the capture card may recording incoming packets incorrectly if it performs “Large Receive Offload” – this means that the card will recombine multiple TCP (or other protocol) packets into a big single packet before sending it to the capture process. That way it may look like you have oversized packets on the network, while it is just the card. To prevent that, turn of those optimizations on your capture card. The Linux script above does that, and Windows with all protocol bindings disabled should, too, otherwise disable it in the advanced settings of the card.
Packet mix and capture efficiency
Okay, so far so good. But now for one of the most common questions many readers may want to ask: “how many MBit/s can I capture with a normal Gigabit network card?” And the answer is – you probably guessed it: “it depends”. Usually, you’d expect a Gigabit network card to be able to capture network frames/packets at a maximum speed of one Gigabit. Right? Well… sometimes.
The problem with standard network cards is that they’re not designed as capture cards, but as send/receive data cards. So they do not expect to be bombarded with frames or packets sent between two or more other devices. And that can be a problem, because in normal operation mode, a stack using the network card for communication can throttle the conversation in various ways if it gets into trouble. It could send packets with lower frequency, or wait longer to put more bytes into the packets it sends. But in a capture situation, the card is on the receiving end of a potential frame/packet tsunami with no way of slowing things down. So it can sometimes feel like a drink from the firehose for the NIC. There are a couple of conference presentations where capture speeds of normal network cards are tested, for example Chris Greer’s great Sharkfest talk “At what point do laptops start dropping packets?“.
As a rule of thumb I do not trust standard 1Gbps network cards to capture more than 200 to 300 MBit/s unless I’m lucky. And there’s a surprising fact for many people: the most important aspect of “how much can I capture without drops?” is not the amount of MBit/s arriving at the card. It’s the amount of packets: you can often capture almost up to the full GBit/s if you receive few but large packets (usually 1518 bytes including FCS). But this doesn’t happen that often during a capture situation. Instead, you usually get a ton of very small packets (as low as 64 bytes). And if the Gbit/s link is saturated with many small packets, the effective capture performance drops to a few hundred MBit/s. It’s not what the card was designed for: in a situation like that capturing with a standard network card may feel like trying to eat a bowl of soup with a teaspoon – it’ll work, but it’s not very efficient. But if you don’t have a tablespoon, it gets the job done, more or less.
So, what can be done? There’s two options for choosing a capture card:
- use a standard card and test if it’s good enough for the capture situation, and use it if it’s okay. What’s “okay” is defined by what your capture goal is (see the 2nd part of this series, regarding drop ratio)
- if the drop ratio is too high to get acceptable results, consider buying a professional network capture card. We’ll look at those cards in a later part of this blog series
Pitfalls
There are a couple of things you need to keep in mind when you plan to capture packets with a standard network card that you may not expect to have an effect, or which may even make the option of using such a card not an option:
VLAN tags: some cards do capture VLAN tagged frames without a problem, but others may do something you do not expect – which can either be
- ignoring VLAN tagged frame completely, so they will not end up in the capture at all
- remove the VLAN tag and only deliver the untagged frame to the capture process
So you should check how your card behaves when it sees VLAN tagged frames before going into a capture situation where you need them in your file.
Frame sizes: standard cards will ignore oversized frames larger than 1518 (or maybe 1522, with VLAN tags) and not deliver them to the capture process (meaning, they’ll be missing in your files). This can be a problem if you are capturing on a link that has oversized frames, either because there are Jumbo frames or a crypto solution adding a couple of bytes to the unencrypted frame. You might want to make sure that your capture card accepts Jumbo frames in the advanced settings.
Receive Offloading: modern network cards have acceleration features that are used to offload certain tasks from the CPU, e.g. calculating checksums or cutting large payloads into multiple frames. What’s relatively unknown is that some cards also do this when they receive frames, and deliver them recombined to the CPU (which means: the capture process). So you may end up with huge frames that were never really seen on the network. For that, you might want to turn off offloading features on your capture card as well.
Out-of-Order frames: if you’re capturing on more than one interface at the same time (e.g. if you have a card with two or four ports you), you’ll end up with frames that are out-of-order. This means that if the frames arrived at the card in the port order of 1-2-1-2 you might see 2-1-1-2 or something similar. This is a huge problem in some capture scenarios, because it can put your analysis results in jeopardy if you’re unlucky. This is how it looks like (an excellent indicator are the negative delta times):
Now there’s good news and bad news: you may be able to recover from out-of-order frames by using reordercap to save them to a new file in correct order. But there can also be out-of-order frames that cannot be reordered because they don’t have a negative delta time, and in that case you’re not going to be able to fix your capture files.
Final words
Capturing packets with a standard (“COTS”- Consumer Off The Shelf) network card is okay if
- it doesn’t inject packets into the capture itself (DHCP, ARP, IPv6 Multicast,…)
- you can live without being able to capture broken frames
- you’re able to enable Promiscuous Mode
- it’s fast enough to capture all the packets you need (meaning: the drop rate is still acceptable)
If not, you’ll need a professional capture card. To be honest – all captures I have taken in the last couple of years that weren’t at a user’s desktop where the packet flood is low (meaning I had to walk into a data center instead) I took with professional capture hardware. It’s simply no fun to call a customer to tell him “well… the capture wasn’t good enough, so we need to do it again…”, especially when the preparation for the capture takes time and effort.
Other parts of this series
Part 1: Ethernet Basics
Part 2: Speed, Duplex and Drops
Part 4: SPAN Port In-Depth
Part 5: TAP Basics
Part 6: Planning Network Captures
This is one of your best articles.
Thanks, Christian, I’ll try to keep writing helpful stuff 🙂
Totally agree with Christian. Enjoyed reading the article. Every peace of information is on it’s place.
Especially liked the part regarding efficiency, because these details were not described in Chrees Greer’s video with such depth.
I performed tests with Iperf and can confirm every word. Just an ordinary onboard Gigabit NIC can actually capture 900Mbit/s with zero dropped packets, but only with bulk TCP transfer (I’ve reached 890Mbit/s). But with UDP flows and TCP small packets I’ve had really hard times with only 100-200+ Mbit/s maximum.
Thanks, Vladimir, it’s really helpful you’re sharing your observations!
Hi Jasper
Another great post very informative, I particularly liked the info on disabling all bound protocols on the Windows computer. Not sometime that would occur to most people, but when you think about it you are capturing frames, and as long as you can grab the frame you can grab what rides on it IP, TCP, SMB etc..
Your posts are very informative (as are Kary videos) and help immensely
Ernie
Hi Jasper,
Thank you for putting together these helpful tutorials.
One question I have is: Can you give any examples of a professional network capture card that one could buy to best capture network traffic?
Thank you,
Brett
Hi Brett,
thanks! I am still going to write a post about those capture cards, but haven’t had the time yet. Some examples for professional capture card vendors are: Endace, Napatech, Accolade Technology, Fiberblaze. In general you should make sure that they will even want to sell their cards to you – Napatech had very strange opinions about that, and I’m not sure if that got any better in the meantime, while Endace is much friendlier in my experience.
Cheers,
Jasper
Hey Jasper,
Regarding your comment:
“Remember, any device being part of the active communication (including firewalls, routers, load balancers, WAN accelerators, traffic shapers, proxies etc.) are off limits and considered “local” if you capture on them.”
There are a couple folks at work who are pushing towards using the newer network gear’s local capture options. Having not yet done a deep dive as to why this is a bad idea I would love to hear your thoughts on the topic.
Thanks,
–Antacus
Hi Antacus, I totally seemed to have forgotten to answer this one, sorry!
Capturing on network gear using a “local capture” option is fine in some situations, e.g. when you want to see if certain traffic passes the device (when checking firewall rules), or when doing baselining, or just application level troubleshooting.
But the moment there is connection trouble (slow, session aborts, other weird behavior) with network communication running through that device the local capture capabilities are not an option anymore. Because that would be putting the fox in the henhouse – any device that may be part of the problem cannot be trusted to provide proof for anything. Ever.
Hi Jasper,
habe gerade ein kleine typo gespottet/short type spotted:
/etc/network/interfaces config file
where it reads
`up up link set $IFACE promisc on arp off up`
IMHO it should be
`up ip link set $IFACE promisc on arp off up`
Correct?
As always, keep up the great work!
Grüße/Greetings
Joao
Hi Joao, yes, that’s correct and fixed, thx! 🙂