The PCAPng file format

Starting with Wireshark 1.8, the old PCAP format was replaced by PCAPng as the new default file format for packet captures. I have to admit that I may be one of the people to blame for this – at the end of Sharkfest 2011 we had a panel discussion with Gerald and some other guys when the topic of PCAPng support came up. Gerald said he wasn’t sure if the move to PCAPng as a default format would be such a good idea, since many existing tools only supported PCAP back then (and unfortunately many still do) and can’t read PCAPng. My comment to that was that in my opinion Wireshark should take take the first step of improving the popularity of PCAPng by using it as its default file format since it is the most popular network analysis tool – and if Wireshark wouldn’t start using PCAPng, nobody would. I’m not saying it was my statement that finally made it happen, but the next stable version of Wireshark did in fact use PCAPng instead of PCAP. But in case you want to blame someone that Wireshark doesn’t use PCAP anymore, blame me, I can take it 🙂

Multiple capture interfaces

One of the big advantages of PCAPng is that it supports storing packets for multiple capture interfaces, even if they have different link types. You can capture packets like that by selecting more than one interface before starting the capture:

You can even set a different capture filter for each interface if you like, or one general filter for all of them. Hint: as of the time of writing this post there is a small bug, though: when changing a capture filter for an interface, all capture filters will be cleared. The bug report for this can be found here.

If you look at a trace that was captured on multiple interfaces you’ll probably not notice that there was more than one interface used for the capture. To find out, you can open the summary statistics which will give you a list, including the capture filters that were used, and how many dropped packets it had (something which PCAP can’t save, either. hah!):
You can also just take a look at the frame header of each frame to see the interface index of the interface it was captured on:

And finally (which is what I do), you can add a custom column to display the value of “frame.interface_id”, which looks like this:

Unfortunately, you can’t add the name of the interface as a column at the time of writing this post, only the index. The index of the interface starts at zero, so the first interface in the file has an index of 0, the second an index of 1, and so on. In the example above you can see that I captured on 4 interfaces at the same time (Indexes 0-3).

The trouble with Tribbles Out-of-Orders

We all know out-of-order packets, when talking about TCP analysis, but they are not the ones I am talking about here. Take a look at this capture example:

This is a complete TCP conversation that does look a little odd. For example, it starts with a SYN/ACK flag in packet 93, instead of a SYN. But the SYN is not lost, it just isn’t where it should be: you can see it in frame 145. So what happened?
If you take a closer look at the delta time column you can see that the SYN packet has a negative delta time of about 59 milliseconds. If you want you can use a calculator but I can tell you right here that the absolute time stamp of that SYN packet puts it 7.5 milliseconds before the SYN/ACK.

What happened here is what almost always happens when capturing on multiple interfaces with standard PC hardware: some network cards deliver their captured frames faster than others, and often in small batches instead of the correct interleaved sequence (for the exact reasons I’m the wrong guy; ask Jörg Mayer or Guy Harris if you want over-the-top in-depth details 🙂 BTW, Jörg had a nice talk at Sharkfest 2014 about the capture process). Fortunately, the frames are (usually) time stamped on arrival on the card, so at least you can see what their original timing was, but the capture process (dumpcap, in case of Wireshark/tshark) writes them to the capture file in the order they arrive.

The problem is now that e.g. the TCP expert of Wireshark doesn’t care about out-of-order sequences of frames – it will incorrectly diagnose symptoms because it strictly follows the order of packets as it finds them in the file. This is why you can see messages like “TCP ACKed unseen segment” in frame 215 in the screen shot above. I can quite easily prove that I’m right – take a look at the same capture again, but this time with the frames being correctly ordered by absolute time stamp:

As I said – no problem at all, and the TCP expert has no objection this time 🙂

How did I fix the file? Easy, there’s two options I know of right now – you can either use TraceWrangler running an “Edit” task, or you can use reordercap. Reordercap is a command line utility that is part of the Wireshark installation, so if you have Wireshark, you also have reordercap (unless your Wireshark installation is really old, but then you’re most likely not going to have trouble with multiple interfaces anyway). What I did was run reordercap like this:

[D:\Traces]reordercap "HTTP Sample 4.pcapng" "HTTP Sample 4 reordered.pcapng"

which will read the input file, sort all frames by timestamp, and write them back out in order to the output file. Easy 😉

More tribbles interfaces, More trouble

There is another issue where capturing on multiple interfaces complicates things: reading a trace, and writing only some of the frames back to file. Or, on a similar level, merging capture files. Maybe you have seen something like this happen:

[D:\Traces]mergecap -a -w mergedfile.pcapng "Test1.pcapng" "Test2.pcapng"
mergecap: Error writing to outfile: Internal error

This happens when you try to use mergecap (again, part of the Wireshark installation) to merge PCAPng files that contain more than one capture interface. The reason is probably that mergecap has trouble determining how to merge the interface entries for the output file.

Merging interfaces
Assume you have two capture files with two interfaces each:

File 1: Interface 0, Interface 1, with names “eth0” and “eth1”
File 2: Interface 0, Interface 1, with names “eth0” and “eth1”

Can you assume that Interface 0 in file 1 is the same as Interface 0 in File 2? Not really, because there is no unique identifier in all cases, e.g. you could have captured the files on totally different systems. How would mergecap know that it should not assume that the interface pairs are the same? It can’t, unless the link type is different – but unfortunately it is going to be Ethernet in most cases, so it doesn’t really help to look at it.

Funny enough, on Windows the story is a little different, because on Windows each interface has a GUID as part of the interface name. You can see that if you run dumpcap on a Windows box with the “-D” parameter, like on my system:

[D:\Traces]dumpcap -D
1. \Device\NPF_{0E372DAC-4F9E-4A19-B021-F947073BA9B4} (VMware Network Adapter VMnet8)
2. \Device\NPF_{1F92B6D8-3450-4CD3-9438-5D3C0567EEB5} (Capture SPAN)
3. \Device\NPF_{1F3417DD-876A-4E61-BE17-2DC1ADE14DCD} (Capture2)
4. \Device\NPF_{3F324CFC-62E7-4599-ABB7-7A1B97F67F9B} (Bluetooth Network Connection)
5. \Device\NPF_{B4A61A68-2E27-46FA-BB69-098FFFFCC2C2} (Main)
6. \Device\NPF_{F9C8C1F5-F80B-411A-B794-0EEBB26FFE21} (Capture1)
7. \Device\NPF_{A9BC8B16-BADD-4C26-8041-166AF4612E65} (VMware Network Adapter VMnet1)
8. \Device\NPF_{098DA058-4B9E-495F-95F3-4848849E2E32} (VPN)
9. \Device\NPF_{FA642256-AFE6-46C6-8263-42FAC83F923F} (VMware)

So for Windows traces it is easier to merge interfaces because it is highly unlikely that two different NICs will have the same GUID as part of their name. It is even consistent across multiple capture jobs because it is assigned by the OS and will not change unless the actual NIC is replaced.

In TraceWrangler, there is a “Merge” task since version 0.3.1 which I released at Sharkfest 2014. I had to put a lot of thought (and work) into figuring out how to make it easy to merge files. You can select to

1. automatically merge interfaces, by link type and name (if possible)
2. create a single interface that all frames are mapped to (basically “flatten” the interface list to a single entry)
3. map each interface found in the source files to a unique new interface

After the initial mapping, you can fine tune the interface settings like this:

The bottom list holds the list of unique interfaces that will appear in the merged file, and the upper list shows what source interface is mapped to which target interface. You can change the mapping, and add additional interfaces with a popup menu if you need.

Extracting conversations from multiple files

One of the things that are really useful when working with a large set of capture files is the process of extracting single conversations from them, no matter where they start and where they end. The problem is that a conversation may use only one or two interfaces, but if a capture has 3 or more in the interface list you need to find out which ones you need. Of course it would be possible to write all interfaces to the target file regardless of them being used by the packets, but that could lead to huge interface tables that are good for nothing. Again, if your capture consists of multiple files you need to merge interface entries like for the merge process we already looked at.

To make things even more complicated you need to know what interface entries you need before you start writing packets, because the PCAPng file format requires the Interface Description Block (IDB) to be written before the first packet block.So you can’t just read packets and add interfaces as you need them – you need to determine the final list before you start reading the packets, and map them correctly to a list that may be totally different than the source list. Yep, it can give you a bad headache 🙂

This is what I am currently working on for TraceWrangler: allowing the extraction of one or more conversations to a new file with a correct interface list, and it requires a lot of work in the code base. Basically all frames need to be scanned before processing them, because it’s the only way to know what interfaces they use.

tl;dr; PCAPng is a much better file format than PCAP, but it is also much more complex to read and write (which is probably why some people refuse to use/code for it – they’re too lazy I guess 🙂 ). Captures performed on more than one interfaces are hard to merge and extract parts from.