Bob Plankers, Technical Marketing Architect for vSphere at VMware, has a very simple but important message for all of us, and it isn’t really limited to VMware itself: “Patch! Patch! Patch! Did I say… patch?”. That was his starting message when I visited VMware during Security Field Day 2, and there was more Bob had in store for us.

Patching is the one most effective security measure you can take, reducing the attack surface by fixing known security problems. And if we’re honest: that’s the most common way of getting breached these days. Not zero days. Not sophisticated custom code attack tools. Just exploiting known bugs, when administrators forget or neglect patching their systems. VMware has various hardening guides that helps further reducing the risk for your vSphere installation here:

https://www.vmware.com/security/hardening-guides.html

And guess what? The “ESXi.apply-patches” item is the first in the list 🙂

Don’t get me wrong: I know that patching isn’t always simple. There’s a lot of reasons why you might not be able to patch whenever you like, mostly because:

1. “systems need to stay up”
2. Certified devices cannot be patched without having to re-certify them

Well, both are valid points, but the downside is that attackers can exploit well known bugs easily if they manage to get to the system somehow. And patching VMware ESXi is really simple if you have DRS in auto mode because it will vMotion your VMs away, patch the ESXi automatically and continue with the next one until they’re all done. So if you can you should patch. If you can’t, get the risk signed off on 😉

Hardware Vulnerabilities

The vulnerabilities that were discovered in modern CPUs are more or less problematic if you’re running your own personal computer at home, but they get a lot more critical in a virtualization environment. It all depends on if you can trust the code in the VMs or not. For a hosted environment with multiple customers having access to their respective VMs on the same ESXi as other customers this can become a problem: using side channel attacks like “Spectre”, “Meltdown” or all the others with a funky name and a logo (keep in mind: it’s not a real vulnerability unless it has those!) a malicious user can potentially access data he/she shouldn’t be able to.

To deal with these kinds of attacks VMware created new CPU schedulers to prevent leaking secrets between Programs and VMs, and not just between ESXi (which is what the default scheduler provides):

For ESXi on AWS it is interesting to know that those won’t be shared between customers, so if you get an ESXi instance in the Amazon cloud it’s yours alone.

To change the scheduler, you need to select your ESXi in your inventory and go to the “Advanced System Settings” in the “Configure” tab. The knowledge base article for the various configuration elements can be found here:

https://kb.vmware.com/s/article/55806

Of course, changing the scheduler and increasing isolation between programs and VMs comes with a performance penalty. VMware has some numbers in a whitepaper about that here (even though Bob warned us that the MDS attacks will void all prior measurements):

https://www.vmware.com/techpapers/2018/scheduler-options-vsphere67u2-perf.html

Another interesting info Bob mentioned was that VMware also patches the CPU microcode whenever you update your ESXi, so that’s another reason to patch. But keep in mind that you need to power-cycle your VM to get that microcode into the running VMs as well.

vSphere Platinum

I have been a certified VMware instructor, but lost that certification when Airbus wouldn’t let me teach the 30 days of training per year I needed to keep it active. Don’t get me wrong – the VMware trainer certification was the hardest test I ever had to get through, so it wasn’t easy to let it go. But I know that it really wasn’t the fault of Airbus because I moved from being a consultant and instructor at Fast Lane to a quite different role at Airbus CyberSecurity. Anyway, long story short: I knew there were various license levels VMware offered, but I hadn’t heard about vSphere Platinum before. It is a new license that adds a lot of security features and controls on top of Enterprise Plus licensing.

One of the feature vSphere Platinum offers is Secure Boot, both for the ESXi as well as for the VMs:

Since I don’t have access to that kind of license I couldn’t play around with it yet, but the general idea is to make it harder to manipulate both ESXi and the guest operating systems. There is a feature called attestation which checks that the whole ESXi software stack hasn’t been manipulated, which can be checked in security tab of the monitoring section of  a vSphere cluster.

Final Words

VMware takes security seriously, which is good to see. Better yet, they try to make it easy to configure, hiding some pretty complex mechanisms behind simple check boxes. Of course in some cases you need to perform a couple of complex tasks before you can use those check marks, e.g. when you try to encrypt a VM to protect it against being stolen from a cluster (e.g. a domain controller with all those juice secrets inside) – because for that you need a Key Management Server first. But in general VMware tries to help you make your setup as secure as possible, and I liked to see the effort they’ve put into their products.

The Tech Field Day video channel with the Security Field Day 2 playlist can be found here:
https://www.youtube.com/channel/UCSnTTyp4q7jMhwECxXzMAXQ

Disclosure

This post is a part of my Tech Field Day post series. I was invited to this event by Gestalt IT. Gestalt IT covered travel, accommodation and food during the event duration. I did not receive any compensation for participation in this event, and I am also not obliged to blog or produce any kind of content.

Any tweets, blog articles or any other form of content I may produce are the exclusive product of my interest in technology and my will to share information with my industry peers. I will commit to share only my own point of view and analysis of the products and technologies I will be seeing/listening about during this event.