Working with multi-point captures

Every now and then most analysts run into a troubleshooting situations where they need to capture the same packets at different locations in the network. Some reasons for such multi-point captures include

  • having to determine if packets get delayed at some point in the network (this would be one of the few cases where “it IS the network”)
  • checking if there is packet loss, checksum errors (which is basically leading to packet loss, too) or any unwanted packet modification “on the way”
  • determining if all conversations look the same on both (or more) capture locations


One of the problems is to select the correct capture locations, but let’s assume we got that covered for the sake of the length of this post. And sometimes it’s really easy, e.g. when you capture at a client and at a server (and have enough TAPs and capture devices, of course).

MultiPoint-ClientServerSetup

Quick reminder: situations like the three mentioned above will most likely require the use of network TAPs to exclude packet loss/timing issues caused by SPAN ports. Using SPAN is really only good for application behavior tracking, and when you really do not care about network/packet transmission characteristics. BTW, yes, that couch is a server. Old school ftw! πŸ˜‰

Comparing captures

After recording the packets the task at hand is to compare the capture files. Depending on the problem situation this usually means either to find matching packets and conversations at each location, or determining where the differences are. It’s not possible to say which of both is easier to achieve since both can get pretty complex. Finding and matching UDP packets is often more difficult than doing the same for TCP packets, simply because there is no “conversation” going on. So if you look for a UDP packet you’ve found at one location it may be difficult to find the same packet at the other location – because if it’s not there its often hard to say why, or if it should have been there at all. With TCP, you get at least a couple of packets for each conversation, and that makes it more likely to find them.

Relative and absolute TCP conversation filters

There are basically two filters that you can use to isolate TCP conversations in Wireshark : one of them is a relative filter, the other is absolute.

The relative filter is “tcp.stream” which is used whenever a user selects the “follow TCP stream” popup menu option on the packet list (which, as far as I can tell, is the one thing each and every user does first when starting to work with Wireshark πŸ™‚ ). “tcp.stream” filters on the stream index, which is assigned to each TCP conversation by Wireshark when loading the trace. The index of zero is assigned to the first conversation found in the file, the second has the index of 1, and so on. This makes it real easy to extract all TCP conversations if you want – just write a small script that runs tshark and filters on increasing stream indexes. By the way, in old versions of Wireshark (1.6 and earlier) the stream index was also counted up for UDP packets, so there were often gaps in the index count when using “tcp.stream“.

The absolute filter uses the TCP socket pair to filter for the conversation, meaning both IP addresses as well as both TCP ports. A typical example would be “ip.addr==192.168.0.1 and tcp.port==49152 and ip.addr==10.0.0.1 and tcp.port==80“. Since the client will use a different port for each new connection there will never be two identical socket pairs at the exact same time.

Extracting TCP conversations

When looking for the same conversation in different capture files, the only useful filter for matching TCP conversations is the absolute filter, using the socket pairs. This is true for multi-point captures, but also for multi-file captures at the same location, e.g. when doing a long time capture with a fixed file size and multiple capture files. Imagine a capture where a specific TCP conversation starts in the first file and ends in the third file. You’d have something like this, with the packets of the conversation stretched over all three files:

MultiPoint-MultiFileExampleYou obviously cannot use the “tcp.stream” filter, because the same TCP conversation will have a different stream index in each file – depending on what else is stored in the file. Instead you have to use the absolute filter, because it will work no matter where the packets are stored within each file.

Finding and matching packets

When you have managed to extract the same conversation on both (or more) location, the best case scenario is that all conversations start and end with the same SYN and FIN or RST packets, meaning that they’re available end to end. In this case the rest is more or less routine: go through the packets in both captures and compare timings, content, packet loss etc., depending on what you wanted to do. Keep in mind that it is often not necessary to have the exact same time on all the capture devices, because most timing problems can be solved by comparing relative timings.

This is how two captures taken at different locations (client and proxy server) could look like – you can tell by the timings that they’re different (and if you could see the Ethernet addresses, you’d notice they’re different, too):

MultiPointCaptures-ClientCaptureSample

packets at client location

packets at proxy server location

packets at proxy server location

But sometimes you end up with two (or more) captures that aren’t complete on all locations, which is often caused by starting one of the captures too late. Unfortunately, it’s not always easy to do another take of the captures, e.g. when you can only look at the files later, or when someone else captured them for you. Sometimes, even a partial capture can be used to diagnose the problem, especially when it’s a long conversation where the problem is known to occur at least once during the duration of minimum packets available in all captures.

If you’re trying to match incomplete conversations the process works like this:

  1. Toggle Wireshark to use absolute TCP sequence numbers (in the TCP preferences) – relative numbers won’t work, because you have various starting sequence numbers, messing with the relative numbers
    MultiPointCaptures-RelativeSequenceNumbers
  2. Determine the lowest sequence number in each capture file for the conversation you want to analyze
  3. Compare the sequence numbers you determined
  4. Use the highest of all the sequence numbers to find the packet with that sequence number in each capture, by filtering for “tcp.seq==highseq” (replacing “highseq” with the sequence number).
  5. Do the analysis

Final words

Analyzing multi-point captures isn’t something you have to do often, but when you do, these are my recommendations:

  1. Filter by absolute conversation filter (socket pairs)
  2. Disable relative TCP sequence numbers, and verify that you really found the same packets
  3. If you don’t have them already, get at least two large TFT monitors to have two Wireshark instances running side by side
  4. Comparing UDP packets is much harder as there is no sequence to follow, so you need to find a criterion can you hope to match packets with

Also keep in mind that matching packets in multi-point captures gets much harder (and sometimes close to impossible) if there are NAT devices or load balancers involved, not to mention proxies/reverse proxies, encryption devices and other nasty surprises.

Oh, and in case you’re wondering: there are commercial analyzers that can help with correlating packets, e.g. Wildpackets Omnipeek or Riverbed’s “Pilot” (now “SteelCentral Packet Analyzer”), of which the output for the two captures show above looks something like this:

MultiPoint-PilotMultiSegmentAnalysis

 

 

Discussions — 6 Responses

  • Ioannis February 15, 2015 on 1:26 pm

    Thanks for sharing ! Always clear the way you write.

    Reply
    • Jasper Bongertz Ioannis February 15, 2015 on 8:08 pm

      Thank you, you’re welcome!

      Reply
  • tony fortunato February 16, 2015 on 1:35 pm

    great job Jasper
    Your articles are always a delight to read

    Reply
    • Jasper Bongertz tony fortunato February 16, 2015 on 5:58 pm

      Thanks, Tony, I’ll try to keep it that way πŸ˜‰

      Reply
  • uhei March 11, 2015 on 4:27 pm

    Regarding multi-point captures and UDP traffic:

    I’ve made good experience to spot packet loss by using the IP ID.
    One way to do it, is running tshark with a display filter for one direction and only output the IP ID for each capture file.
    Afterwards diff the list of IP IDs to spot the packet loss…

    Reply
    • Jasper Bongertz uhei March 11, 2015 on 7:17 pm

      Yes, that is a good way to track UDP packet loss. Unfortunately, it doesn’t always work, e.g. when a TCP stack like the one in OpenBSD sets the IP ID to zero except when fragments occur. Also, IP ID will not be available in IPv6 anymore, because it’s moved to the optional Fragmentation Extension Header. Fortunately for network analysts, both cases are pretty rare πŸ™‚

      Reply

*