TCP Expert Updates in Wireshark 1.12
Wireshark 1.12 has just arrived, and of course the first thing to do is to download and install the new version. The second thing to do should be to read the release notes.Nobody seems to do it, but everybody should. Okay, before I get to the TCP expert thing, let’s see why release notes are important.
Read the release notes!
The reason for reading the release notes is simple: sometimes, things that you were used to behave in a certain way may have been changed by the developers.
It happened to me during a Wireshark class a couple of years ago (using the then brand new version 0.99.6) that suddenly my delta time column showed strange values when I applied a filter. They were slightly different from what I expected, and I knew they were just wrong. I had used the trace for so many trainings that I was sure what Wireshark should have displayed, but it didn’t.
So I was about to open a bug report, annoyed that the new version would introduce a bug like that. When I open bug reports I always try to check if there is already a report for my issue, and in this case I even googled for the problem. It turned out that the unexpected behavior wasn’t a bug – the Google search results guided me to the release notes of Wireshark 0.99.6. In there it was mentioned that the “delta time” value was no longer showing the time between displayed packets, but between captured packets. Which means that if you apply a filter, my delta time column would show delta times to now invisible packets. To allow displaying delta times between packets visible after filtering, a new value was added: “delta time displayed”.
I changed my column setup, done. Since that day I always read the release notes. By the way, if you ever have the chance to join us at Sharkfest, you might find yourself in a session where the “Read the Release Notes” inside joke comes up again – usually when Hansang and myself are in the room 🙂
TCP Expert changes
Unfortunately, there are changes in Wireshark 1.12 regarding the TCP expert that are not listed in the release notes. Let’s take a look at a trace, first in Wireshark 1.10.9:
As you can see, Wireshark flags packet 570 as a TCP Retransmission. Now lets open the same trace in Wireshark 1.12.0:
This time, the same packet is flagged as “TCP Out-of-Order”. Everything else is the same – apart from the (annoying) transport name resolution feature now being disabled by default (yay!) – so what did happen between 1.10 and 1.12?
Retransmission vs. Out-of-Order
The difference between a packet being a retransmission or an out-of-order arrival is not really that big. Both are packets that arrive later at the receiving node than they were expected to. The out-of-order packet is the original packet, while the retransmission is a replacement for the original which had been lost somewhere somehow. If you have a capture where the original and the replacement are both included you can easily tell that a late arrival is a retransmission (assuming the trace is free of duplicates).
If you have a trace where you don’t have original and retransmission you need to look at packet timings. Let’s look at the 1.10 trace again, this time filtered for the TCP conversation and with a time reference set to the “segment lost” message:
The retransmission in packet 570 arrives about 3.04 milliseconds after the “segment not captured” message. By looking at TCP sequence numbers it can be confirmed that packet 570 is in fact the missing segment (let’s just say we know it is – if you’re unsure how TCP sequence number tracking is done, take a look at my Sharkfest slides). Now, how can we tell if packet 570 is a retransmission, and not an out-of-order?
Basically, we need to ask one simple question: assuming a packet was really lost, could the sender have known it was lost and have sent the retransmission? So it’s all about timing, and for that it is important to know what the Round Trip Time of a connection is. Because the message indicating packet loss needs to travel to the sender, and the retransmission needs to travel to the recipient. This means that a real retransmission cannot arrive faster at the receiver than RTT, best case. So, if a packet arrives in significantly less time, it is out-of-order, not a retransmission.
So, how do we tell what the RTT is? It’s pretty simple, as long as we have the TCP three way handshake in the trace. Wireshark does Initial RTT calculation for you – as a new feature in 1.12, and it is displayed in the SEQ/ACK Analysis section of every TCP packet:
So now we know that the Initial RTT (iRTT) is about 170ms. And according to what we said earlier, this makes the retransmission in packet 470 an out-of-order instead. Question is – why did Wireshark mark it as Retransmission in earlier versions? Well, look at this code comment taken from packet-tcp.c of Wireshark 1.10, and keep in mind that iRTT wasn’t calculated:
       /* If the segment came <3ms since the segment with the highest         * seen sequence number and it doesn't look like a retransmission         * then it is an OUT-OF-ORDER segment.         *  (3ms is an arbitrary number)         */
As you can see, up to version 1.12, Wireshark had a static limit of 3 milliseconds. It was chosen because in LANs it is a good number, and most troubleshooting situations should involve LAN issues. 3ms doesn’t work for WAN connections, but as long as you don’t know iRTT you can’t do anything other than selecting a static number. So any packet arriving later than 3ms was flagged a retransmission, regardless of the RTT – in case of the example trace the delay way barely above the 3ms, by 4 microseconds. Boom, “Retransmission” 😉
I had a discussion about this with behavior with Jörg Mayer at Sharkfest 2014. A little later Evan and Jörg changed that code in the latest development builds. For that, Evan added another piece of code to determine Inital RTT if there is a full three way handshake in the trace. By the way, if Wireshark 1.12 cannot determine iRTT, the old 3ms threshold is used.
Another good job of the Wireshark core developers!
I thought you were awesome before, now I know you rock the house. Not having to go back up to the top of the conversation to check the rtt of the handshake when I’m looking at a retransmission. It’s the little things that make life wonderful.
*blush* Thanks, Betty 🙂
I have to add that, as far as I know, Robert Bullen had some impact on this issue as well, since he brought up the importance of knowing RTT in his awesome Sharkfest Jeopardy session, too. That’s probably the reason why it was implemented rater quick without me having to nag too much 😉
I mean come on…who reads them… really! 🙂 Another link I’ll be using when teaching Wireshark and Protocol Analysis. Dumpcap one and this one. Must read once again! Great job, Jasper. Now I’m even more inclined to collaborate like we talked about at Sharkfest 😉
Thanks Hansang, my pleasure… I’ll try to do more posts worth your link list 😉 Only problem is that those take time, but it’s like gathering interesting things for later, so it’s time well spent 🙂
Hi. I just downloaded & installed the new version of wireshark. Stumbled upon your blog when I was looking for some info on packet analysis. I am going to bookmark you blog & will be visiting it often.
Thank you.
Cheers!!
Saurabh
Thank you. You’re welcome!
Great info Jasper as always. I’ll try not to commit the “Just Read the Release Notes” sin again.
Great note on the iRTT! Awesome stuff.
Thanks, Chris!
I’m using Wireshark version 1.12.4 but it’s still dispalying Retrasmission instead of out of order. It looks like the feature rolled back.
Well, it depends on iRTT being available or not, and what the values are. As far as I have checked, the code of the feature is still active.