System Error 58 – Wireshark to the rescue

The other day I was called to investigate a problem where a user could no longer mount a share. The client was running Windows 7. The user got the somewhat obscure message “System error 58 occurred”.

Figure 01 – System Error 58

The usual search engines would give silly recommendations like “turn off your anti-virus”. Using Wireshark and knowing a bit about the SMB protocol would quickly deliver a much more reasonable solution.

Refresher: SMB connection set up

Client and server will exchange several messages when a network drive is mapped:

Figure 02 – SMB Connection Setup

Please note the following events, each coming as a request/response pair.:

  • Negotiate Protocol
  • Session Setup
  • Tree Connect

The Negotiate Protocol phase is used to identify the highest common SMB version shared by client and server. The Session Setup phase is used to authenticate the client. The trace shows an NTLM authentication, which takes 4 packets. In packet 39 the server presents a challenge which the client uses in packet 40 to formulate a response. The Tree Connect would specify the share name.

The faulty connection

A trace file taken from the failing connection shows that client and server will not get beyond the Negotiate Protocol phase. The client aborts the TCP connection with a RESET.

Figure 03 – Connection Reset

SMB Dialects

For a root cause analysis, we need to find out if something is wrong with the client or with the server. There are only two packets to look at. Let’s start with the Negotiate Protocol Request:

Figure 04 – SMB Dialects requested

That’s very odd: The workstation is offering the SMB dialect “NotSmb”. The usual protocol mix offered by Windows 7 would be 6 SMB v1 dialects plus SMB 2:

Figure 05 – Usual SMB dialect List

For the failed connection we don’t get an error code. Still, we see what’s happening by looking at the answer in packet 12:

Figure 06 – SMB Dialect selected

Clearly, the server did not like the SMB dialects offered by the client. The operation fails because client and server don’t support a common SMB dialect.

Digging into the client

A look at the registry confirmed my suspicions: Someone had turned off SMB v1 on the client. This is done by disabling the driver MrxSmb10 and adjusting the dependencies for the LanmanWorkstation Service. The whole process is described in great detail in an excellent article on Microsoft’s Website:

https://support.microsoft.com/en-us/help/2696547/how-to-detect-enable-and-disable-smbv1-smbv2-and-smbv3-in-windows-and

Figure 07 – Registry Editor

Please note that the configuration for the MrxSmb10 driver and the LanmanWorkstation service can be found under the registry key HKLM\SYSTEM\CurrentControlSet\Services.

Clearly, the workstation administrator was paying attention to Microsoft’s recommendation and had turned off SMB v1. Unfortunately, the server would only support SMB v1, which causes the connection to fail.

Resolution

On first impulse one might condemn the workstation administrator for disabling a widely deployed protocol. Watch the wording: “Widely deployed”. Hopefully not widely used: SMB v1 is only required to communicate with Windows XP / Server 2003 and older systems. Or Samba v3 and older, if you use Linux. These are legacy systems and should be migrated to a newer OS or turned off as soon as possible. SMBv1 is a security risk and should be disabled everywhere.

In this case the server was running a more recent version of Samba. Instead of going back SMB v1 we tweaked the server to enable SMB v2. Case closed.

Discussions — 4 Responses

  • Thomas April 13, 2018 on 2:43 pm

    Nice post, thanks Eddi!
    I tried to watch “Negotiate Protocol” on an already mounted share, so I unmounted it and mount it again, but could only watch “Session Setup & Tree Connect”

    Thought about caching issue, waited more than 30 minutes, still no “Negotiate Protocol”.
    Haven’t tried to reboot client though.

    Any idea of caching issue of “Negotiate Protocol” phase?

    Reply
  • Eddi April 13, 2018 on 9:25 pm

    Interesting situation. It is possible that the client still has open SMB connections or at least a file handle.

    I assume that you checked TCP and SMB Sessions?

    – netstat -ano | findstr :445
    – net use
    – Event Logs (if they are configured to this level of detail)

    Can you share a trace file?

    Reply
    • Thomas Eddi April 16, 2018 on 2:38 pm

      As always: no trace file…
      The only thing I’m sure the client was Windows 10, maybe you can try to reproduce the issue…

      Reply
  • Vladimir April 16, 2018 on 2:59 pm

    We’ve had very similar issue on one of our site. 20 IP cameras ” suddenly stopped recording” a video archive onto SMB server displaying meaningless error like “mount failed with unknown error…” or like that.
    Analysing a capture we’ve found the customer at one moment just limited SMB dialect on a server to the minimal version of 2.0 whereas IP camera’s firmware were not able to use it. So the customer needed to turn SMB 1.0 on again.

    Reply

*