In the first post I described how to capture packets in VMware vSphere environments when dealing with standard vSwitches. While that works fine, some larger installations have an even better way of doing network captures of virtual machine traffic, provided by the so-called Distributed vSwitches. Unfortunately, those special vSwitches require a Enterprise Plus license, so all installations that run on Enterprise or less do not get these and have to stick with standard vSwitches.
The main reason to use Distributed vSwitches is that you can create single logical vSwitch instances that span across multiple ESXi nodes instead of having to create identical vSwitches on each hypervisor. Also, VMware seems to always put the cool new stuff into the distributed vSwitches, so there are quite a few reason to use them if you can: they provide e.g. NetFlow options, active LACP capabilities, private VLANs and – what helps with troubleshooting a lot: Port Mirroring.
So let’s take a look at an sample setup of a Distributed vSwitch running in a vSphere 5.0 environment, which isn’t the latest version – but I’ll get to a newer setup in a post in the future, because it gets much more complicated. This test dvSwitch is only used by one hypervisor, but that is enough for a capture test setup.
Let’s assume that we want to capture the traffic of Server1. To do that we have to create a monitor session on the Distributed vSwitch by moving to the “Networking” view of the inventory.
Important: do not enable promiscuous mode on the portgroup you are using the monitor session on or you’ll end up with duplicate frames. Either use promiscous mode or a monitor session, but not both at the same time!
The first thing we need to do is to write down the ports the VM we want to capture traffic of and the port of the capture VM are connected to. In this case the Server1 VM is at port 101, and the capture VM at port 102, as you can see in the next screen shot:
Now, edit the settings of the Distributed vSwitch. Make sure that you do not right click on a port group or any other object in the inventory tree or you’ll not get to the right settings dialog:
Don’t get distracted by all the tabs that offer other interesting settings and select the “Port Mirroring” tab:
As you can see there is no port mirroring configured at the moment, since all details fields are empty. Click on the “Add…” button to create a new monitor session. A dialog will open where you can enter a name and a description. It is usually good practice to use descriptive names as well as putting in a good description, because you may need to decide later if that session is still necessary or good for removal. A name like “Span Session 1” will most certainly lead to it being removed by the next administrator that wonders what it is good for (well, mine is temporary anyway, so it doesn’t matter):
The interesting things are the checkboxes in the “Port Mirroring Session Details” pane. Here’s what they do:
- Allow normal IO on destination ports: the traffic you’re going to monitor has to be copied from the source port to a destination port. At that destination port the capture VM will record any frame that is copied. By default, that destination port will not allow the capture VM to send any data back into the switch, which makes perfect sense – you don’t want your recording device to inject traffic into the link. It should only listen and record data.Sometimes, you want to be able to send data on the capture link, e.g. for allowing the vSphere client to configure the settings. I usually recommend adding a second management network card to do that and keep the capture card as “receive only” card. Anyway, if you know what you’re doing you can enable normal I/O operations with this checkbox.
- Encapsulation VLAN: if checked, you can encapsulate the frames on the monitor port into a VLAN of your choice. The sniffer running on the monitor port will see Ethernet frames with VLAN tags, and if the original frame was already tagged you can choose to keep it as well. In that case you’ll end up with double VLAN tagged frames.
I have say that I tried this feature on the vSphere 5.0 setup and it didn’t work. At all. There was no VLAN tag added to the copied frames, no matter what I tried. I’ll have to recheck this behavior with a 5.1 setup again. I even tried a couple of different capture cards (virtual as well as physical) because some cards discard VLAN layers, but I used at least 2 that are proven to handle VLAN tags correctly, so it’s not the capture equipment that has a problem.
- Mirrored packet length: this setting allows you to cut copied frames at a specified offset, e.g. if you set this to 60, all frames that are copied will be 60 bytes long with anything else discarded at the dvSwitch level. This may seem familiar with the Wireshark capture options setting (“Limit each packet to… bytes“) where you can do the same, but it isn’t: other than the Wireshark setting you will record a frame that is 60 bytes long and lose the information about how long it really was! The Wireshark setting will record 60 bytes and store the information about the real length in the packet headers instead. While that may sound like a minor advantage it is a lot more than that, because when Wireshark doesn’t know that the original frame was longer than what it sees it will annoy you with tons of checksum error messages, TCP lost segment warnings etc. that can throw you off. Rule of thumb: don’t limit the packet length on the mirror session! Do it in the capture software if you have to.
Next step is to add a source port. There are a couple of things you need to do here:
- find out the port ID number where the VM you want to capture traffic of is connected to. You need to do that at the port group level looking at the “Port” tab as we did at the start.
- There is no list of ports you can select from. You have to enter them as text in the “Port IDs (e.g. 1-4, 5, 10-21)” box in the middle of the dialog and press the “>>” button to add them to the list on the right.
- Usually, selecting the “Traffic direction” of Ingress and Egress is correct, but sometimes you may only want one of them.
Now we got the source covered, so the next step is the destination port. There are two possibilities here: you can either mirror the traffic to another VM, or (and that is pretty cool) you can mirror the traffic to an uplink port. That means that you need a free physical network card in your ESXi where you can connect a physical capture device to record the traffic of a VM, so it is sort of an outbreak for virtual packets. In our case, we’ll do a virtual capture, so we add the port of the capture VM:
The last step is to verify the setup and enable the monitor session by activating the check mark before pressing the “Finish” button:
Summary: distributed vSwitches are very helpful when you need to capture traffic of a VM since you can do it without having to do the duplicate port group trick I described in the first post. Also, you can “exfiltrate” traffic via a physical NIC and record the data on a physical capture device, which takes away the problem of the I/O generated on the hypervisor/storage by writing packets to disk.