TCP Server slamming the door
After doing a lot of analysis sessions on TCP connections there are some patterns that you see again in a trace every once in a while. And often it comes in handy to remember what the situation was and what the circumstances were that led to the trace showing what it did.
One of the interesting connection problems can be seen in the following snippet:
If you look closely you can see that the client (IP address 192.168.1.1) does a successful Three Way Handshake with the FTP server (IP address 10.0.0.1), but before it has a chance to send any useful commands the server tears the session down again in packet 4. So the question is: why does the server agree to a three way handshake, only to never allow the client to do anything with it? Lets take a look at the possible reasons:
- The clients ACK of the Three Way Handshake was somehow faulty, forcing a connection abort
- The FTP server had already too many connections open and refused a new one
- The FTP application on the server crashed right after the connection was accepted
- The server doesn’t like the IP address the client is coming from
So let’s take a closer look at each one.
The ACK of the Three Way Handshake may have been faulty
This theory sounds interesting, but it’s not very likely. First of all a TCP Three Way Handshake rarely fails at all, and I have never seen it fail this way. Calculating sequence and acknowledge numbers is no rocket science, at least for a computer (in my Wireshark classes it often seemed to be amazingly difficult for humans sometimes), so we can assume that there was no problem with that. In fact the sequence numbers in the trace shown above are all perfectly fine. And if they weren’t, the server would not send a nice “FIN” flag to tear down the session, but a fatal “RST” flag. So this is not it.
FTP connections limit reached
Maybe the server had a limit configured, and that limit of concurrent connections was already reached when the client connected? It’s a possible scenario, but usually servers send a FTP error code 412 “too many connections” when that happens. Much more polite, and the user even knows what is happening, and why the connection didn’t work. It will also keep him from trying again and again like a madman. So no, it’s not the connection limit, at least in this case. Next.
FTP application Crash
Could it be that right after the connection was established the FTP server application simply crashed? Well, no. First of all because other connection were still working at the same time instead of being disconnected the same way, which is additional info not seen in the trace but which you should always try to get in cases like this. And, more importantly, when an application crashes it releases (or is forced to release) the sockets it has in the operating system. And when that happens we will see a RST coming from the TCP stack, not a FIN. So no crash, either.
The Server doesn’t like the client IP
Maybe the server just doesn’t like the IP the client is coming from. But if that would be the case, why does it go through the whole TCP Three Way Handshake instead of simply refusing the SYN packet with a RST packet?
Well, the answer is that in this case the server really refused the client IP, but it wasn’t the TCP stack itself that decided to drop the connection. It was the FTP server application. Remember the OSI modell? 😉
The FTP server application gets notified of the new connection only after the TCP stack has finalized the Three Way Handshake. So first there needs to be that handshake, and then the application learns that a new client is connected. Since the FTP server application in this case has a list of acceptable IP addresses that are allowed to connect it does a compare between the client IP and the whitelist. It realizes that the client is not on the list, so it terminates the connection as soon as possible. And that is the FIN that we see coming from the server – the socket is not abruptly closed like in an application crash (which would result in an RST packet) but “gently” with a FIN flag.
So, if you ever see a TCP connection behaving like this one you should find the administrator responsible for the application and tell him that he needs to find the configuration setting where the blacklist or whitelist is configured.
Happy new year, and happy packet reading!
Jasper
Hi, nice article and thanks for the explanation. BTW, how do you edit the arrow -> icon ? It looks more meaningful to me 🙂
thanks in advance.
Thanks, Macha. The new arrow icon is something that seemed to have been introduced with the latest developer builds. If you’re using the latest stable version (1.10.5 at this moment of writing this) you get the old “>” sign instead. BTW the new arrow character does not work on all fonts, because some fonts do not have the glyph for it.
If you dare, try downloading the developer build from the automated build section, e.g. for Windows x64 at http://www.wireshark.org/download/automated/win64/
Keep in mind that those builds are “bleeding edge” and may have strange side effects when working with it. I usually recommend to keep using the latest stable builds instead unless you are capable of spotting problems caused by Wireshark instead of the captured packets 😉
This article is close to a problem I’m having. Do you have any ideas on this one?
3 way handshake succeeds, go to http.
Client sends http request (GET /)
Server sends response (starts sending page)
Client seems to simply stop listening. No RST, no FIN, no nada.
I’m stumped.
Do you have a trace you can show me? Either on http://www.cloudshark.org, or via email? The email is “jasper” at this domain (“packet-foo.com”).
Many thanks Jasper, greatly Appreciated !
Capture is here if you want to look at it. Issue is between 10.20.30.42 and 10.20.30.45
https://www.dropbox.com/s/whb0h1r55mn15cr/Server_Terminating_TCP_Session.pcapng?dl=0
Hi Very nice article, we have similar pattern on one of the connection issue we are facing. But in our case it works all the time but sometimes it fails and we see such FIN,ACK packets whenever we see issue, Could this be because of somekind of blacklist?
Thanks
Could be a blacklist, or sometimes software is licensed or otherwise configured to a certain amount of concurrent sessions. So if that limit is reached, new sessions are refused politely with FIN/ACK until other connections are closed, allowing new sessions to be accepted.
HEI JAPSPER CAN YOU TAKE A LOOK AT MINE, I AM REALLY CONFUSED ABOUT IT. WHAT IS GOING ON WITH IT AND WHAT ARE THE SOLUTION TO IT IF YOU HAVE IDEA ABOUT IT. i tried sending it to your mail but is not working. can you resend the email
Send it to jasper [ät] packet-foo.com and I’ll take a look.
HEI JASPER, THANK FOR YOUR RESPONE, I HAVE SENT IT TO jasper@ packet-foo.com. HOPE TO HEAR FROM YOU.
Thanks for sharing this great information.