Deep Dive: Frame Timestamps

Last week Uwe, one of the instructors of the Wireshark class I created for FastLane, gave me a call in the evening. He was teaching a 5 day class in Hamburg at the time, and had had a student ask about a peculiar problem with frame/packet timestamps. I remembered that I had read something about this issue before, so I told him I’d investigate. And in the end, it looked like a good topic for a blog post, so here it is. It also means that I can point Uwe at this post instead of writing a lengthy email. Hm, wait… so now I write a blog post that is even longer?! Nevermind. Let’s go.

Zombie Packet Mayhem!

My previous post was about one of multiple false positives a network analyst needs to keep an eye out for to avoid writing down findings in a report that weren’t really there. So when I looked at my Sharkfest traces to see what other topic I could write this post about I realized that I have already “burned” two of the other false positives in earlier posts. Well, the good thing is you don’t have to wait and can read about them right away (if you haven’t already).

TCP Analysis and the Five-Tuple

The TCP expert of Wireshark is doing a pretty good job at pinpointing problems, helping analysts to find the packets where things go wrong. Unfortunately, there are some things that can throw the expert off pretty badly, which can fool inexperienced analysts in believing that there are big problems on the network. I did a talk about some of those problems at Sharkfest 2013 called “Top 5 False Positives”, and this post will be about on of them: Duplicate packets.

Working with multi-point captures

Every now and then most analysts run into a troubleshooting situations where they need to capture the same packets at different locations in the network. Some reasons for such multi-point captures include

  • having to determine if packets get delayed at some point in the network (this would be one of the few cases where “it IS the network”)
  • checking if there is packet loss, checksum errors (which is basically leading to packet loss, too) or any unwanted packet modification “on the way”
  • determining if all conversations look the same on both (or more) capture locations

Determining network protocols

If you spent enough time using Wireshark or any other network analysis tool, you’ll sooner or later be able to even read bare hex dumps of packets, at least partially (it’s a little bit like Neo seeing the Matrix). So maybe you run across a text dump of a packet like this one:

0000  00 0d b9 21 95 18 c8 60 00 16 7c cc 08 00 45 00   ...!...`..|...E.
0010  00 34 6b 8a 40 00 80 06 00 00 c0 a8 7c 64 51 d1   .4k.@.......|dQ.
0020  b3 45 c4 60 00 50 19 00 52 e7 00 00 00 00 80 02   .E.`.P..R.......
0030  20 00 42 4a 00 00 02 04 05 b4 01 03 03 02 01 01    .BJ............
0040  04 02

Diagnosing intermittent “network” problems

There’s that one thing that customers usually ask, and that question is if I would be able to help diagnosing a problem on the network. My answer has two parts:

  1. If we can capture the problem situation in packets, I will find it
  2. When I find it, I’ll tell you if it’s a network problem (which, in my experience of over 10 years, is the case in only about 20%)

The trouble is: there are some problems where it’s not easy to capture packets, and that’s when you don’t know the correct capture location (usually meaning “there’s too many possible locations”) or the exact time for the packets with the symptoms to be recorded – or both.

A look at a portable USB3 network TAP

A while ago I wrote a post for LoveMyTool about how I managed to power my Garland Gigabit TAP with a USB cable, which got me into a discussion about the ProfiTap USB3 device on Linkedin. I had used 100Mbit USB2 ProfiTap devices before and had some issues with it on Linux, so I was a bit skeptical towards the new ProfiShark 1G as well. In the end, the nice people at Comcraft offered to send me a sample to see how it performed, and I am always happy to get my hands on interesting capture solutions to see how they perform.

Determining frame forwarding latency

In some situations the question arises how much a frame was delayed by a device it has to pass through, e.g. firewalls, loadbalancers and sometimes even routers and switches. Usually, novice network analysts think that for that you need to synchronize the clocks of the capture PCs down to microseconds or even better, but that is not necessary for this kind of reading. It is possible to capture the packets with completely different time settings on the capture PC left and right of the device you need to determine the delay for.

How millisecond delays may kill database performance

Mike, an old buddy of mine is one of the best database application development consultants I have ever met. We worked together for the same company for a couple of years before I got into network analysis and he started his own company. A couple of months ago I found out that there was going to be a conference in my home town where Mike was on the organization team. After a friendly banter on Twitter about him having to come to my city (Düsseldorf; which guys from Cologne like Mike don’t like ;-)) he told me that I should turn in a proposal for a talk. I said I could do that, but not on any database development topic – but maybe a generic network application performance talk might be interesting for those guys attending. So I did, and it got refused, despite Mike advocating for me. Darn.

Well, it’s a nice topic for a blog post nonetheless. So here we go.