vSphere ESXi SSL mystery solved

For quite a while I've been trying to get SSL certificates uploaded to an ESXi 4.0 host which were issued by our internal Microsoft CA. Unfortunately I ran into issues, the last being that adding an ESXi 4.0 host to vCenter 4.0 with the certificate would die at 80%.

After additional testing, I now have a procedure which seems to work perfectly for ESXi 4.0 and 4.0 Update 1 hosts. But you must follow the steps exactly as written, or it may not work. It even works with a certificate from a Windows Server 2008 R2 CA using the new sha512ECDSA (elliptic curve digital signature althorithm with secure hash algorithm 512) NSA Suite-B certificates.

1. Download the Windows OpenSSL binaries, either 32-bit or 64-bit. Remember to install the Visual C++ binaries on prior to OpenSSL.

2. I create a directory called Certs in c:\OpenSSL just to keep certificates separate.

3. Cd c:\openssl\certs

4. c:\openssl\bin\openssl genrsa 2048 > rui.key

5. c:\openssl\bin\openssl req -new -key rui.key > rui.csr

6. At this point OpenSSL will prompt you for various parameters. Enter any information you wish, but make sure the Common Name is the FQDN of your ESX server (.e.g. Q100ESX01.contoso.net). Do not set a password.

7. Use NotePad and copy the contents of rui.csr to the clipboard.

8. Navigate to your Microsoft CA and select the option called something like "Submit a certificate request by using a base-64-encoded CMC...."

9. On the Saved Request screen paste the contents of the clipboard, and change the certificate template to Web Server.

10. Submit the request, then download the Base-64 encoded certificate (not the certificate chain). I saved the file as rui.cer into the c:\OpenSSL\Certs diretory.

11. Optional: Perform verification of the certificates per my blog post here.

12. c:\openssl\bin\openssl x509 -in rui.cer -out rui.crt

13. Open a VMware vSPhere CLI command prompt (if you don't have RemoteCLI installed, download it here.

14. vifs.pl --server ESXhostname --put c:\openssl\certs\rui.key /host/ssl_key

15. vifs.pl --server ESXhostname --put c:\openssl\certs\rui.crt /host/ssl_cert

16. Reboot the ESXi host and wait five minutes after the ESXi console appears. Use a web browser and navigate to your ESXi host. In the address bar of your browser open the properties of the SSL certificate and verify it was issued by your CA and is not the self-signed certificate.

17. Add your ESXi host to vCenter, and it should NOT get stuck at 80% and fail.

If you run into problems, make sure on the ESXi console that the hostname is configured with a FQDN. From the ESXi console you can also view the management agent logs and look for any SSL related errors.

Next up is changing the vCenter server SSL certificates, as well as VUM. This was broken in 4.0, so hopefully Update 1 has solved these problems. Expect a blog update on this and a procedure, if I find one that works.