VMware vCenter 5.1 Installation: Part 1 (SSO Service)

UPDATE 3/6/2013: There has been some discussion on what version of OpenSSL to use for vSphere 5.1. VMware states one should use 0.9.8. In OpenSSL 1.0 they changed the hashing algorithm, so you need to use a switch to produce the "legacy" hash of 0.9.8. -subject_hash_old is the switch you must use in OpenSSL 1.x. I get the same hashes with 0.9.8y, 1.0.1c, and 1.0.1e using that switch for the 1.x tree. In Part 3 under Update Trusted Certificate Store I show both command lines so you can produce the correct hashes.

UPDATE 2/17/2013: Added Part 15, replacing ESXi host SSL certificate. I also updated the Microsoft CA template creation instructions found here.

UPDATE 2/15/2013: Updated SQL service pack/patch level information, that may address SQL SSL problems that users are still reporting. Basically you need SQL 2008 R2 SP1 CU6 or later. I've updated the post below as well. You can find the MS KB here.

UPDATE 1/30/2013: Updated the JDBC instructions with a possible workaround for SSO/Inventory service install problems when trying to use SQL SSL. Double back-slash the path!

UPDATE 1/29/2013: I've now completed the refresh on all 15 blog posts. I was able to finally build a vCenter 5.1.0b system without undocumented script hacking or running into bizarre errors. Hopefully the revised blogs can help out others with a smoother, if not quicker, install. It's still way harder than it should be, though.

UPDATE 1/27/2013: Now that I'm back from vacation, I'm going to revise the vCenter 5.1 install series to cover 5.1.0b, plus reader feedback, and updates to VMware KB articles. I will also do a little clean up as well, to hopefully make things even easier for people.

UPDATE 12/20/2012: VMware has released vCenter 5.1.0b, which addresses yet more installation bugs. I would advise skipping 5.1.0a, and go straight to 5.1.0b. You can find the 5.1.0b release notes here. I currently do not have time to update all the posts with 5.1.0b information.

UPDATE 10/25/12: VMware has released vCenter 5.1.0A, which addresses many major installation bugs/issues. You can find out more details by reading my article here. I would strongly urge you to use 5.1.0A, and NOT the GA release. You may save yourself some major headaches. I'm in the process of updating each blog post as appropriate to reflect any changes needed for the 5.1.0A release.
---

This is the first post in a series of blog articles about configuring vSphere 5.1.0b. Links to the other parts are below:

VMware vCenter 5.1 Installation: Part 2 (Create vCenter SSL Certificate)
VMware vCenter 5.1 Installation: Part 3 (Install SSO Service SSL Certificate)
VMware vCenter 5.1 Installation: Part 4 (Install Inventory Service)
VMware vCenter 5.1 Installation: Part 5 (Install Inventory Service SSL Certificate)
VMware vCenter 5.1 Installation: Part 6 (Create vCenter and VUM Databases)
VMware vCenter 5.1 Installation: Part 7 (Install vCenter Server)
VMware vCenter 5.1 Installation: Part 8 (Install Web Client)
VMware vCenter 5.1 Installation: Part 9 (Optional SSO Configuration)
VMware vCenter 5.1 Installation: Part 10 (Create VUM DSN)
VMware vCenter 5.1 Installation: Part 11 (Install VUM)
VMware vCenter 5.1 Installation: Part 12 (VUM SSL Configuration)
VMware vCenter 5.1 Installation: Part 13 (VUM Configuration)
VMware vCenter 5.1 Installation: Part 14 (Web Client and Log Browser SSL)
VMware vCenter 5.1 Installation: Part 15 (ESXi Host SSL Certificate)

Related Posts:
VMware vCenter 5.1 Installation: SSL Certificate Pre-Staging Script
Create VMware Microsoft CA certificate template

VMware vSphere 5.1.0b was released on December 20th 2012, while the 5.1 GA version was released in mid-September. Given all of the new components and architecture of vCenter, I wanted to walk through the entire configuration process from scratch assuming a greenfield environment. The original 5.1 release had a lot of, shall we say, bugs and issues related to the SSO service and SSL problems.

In addition, due to various configuration issues and bugs, save yourself headaches and install all the vCenter 5.1 services (except VUM for medium/large environments) on the same VM. Hopefully in vSphere 6.0 VMware will fix the plethora of SSO and SSL issues to be more friendly to distributing the services among two or more servers.

This series covers:

• vCenter 5.1.0b Installation on Windows Server 2008 R2
• SSO service, Inventory Service, vCenter Server, vSphere client, vSphere Web client, and VUM
• Configure SSL certificates for all vSphere 5.1 services
• Configuring SSL encrypted connection to the SQL 2008 R2 server

If you work in a very security conscious organization, you may be using SQL SSL encryption. If so, the instructions below have been updated to provide a method of fully implementing a trusted SSL SQL connection with the JDBC driver for the SSO service. You can check if you are forcing SQL SSL encryption by looking in the SQL Server Configuration Manager and looking at the properties of the Protocols for MSSQLSERVER. As shown below, if Force Encryption is set to Yes you will need to modify the JDBC connection string to use SSL. Even if encryption is not enforced, as long as the SQL server is configured to use SSL, vCenter can take advantage of that added security.

On another security note the SQL server MUST be configured to allow both Windows integrated authentication AND SQL authentication. SQL authentication is very weak, which makes the use of SSL for the database connection that much more imperative. Should the SQL server only allow Windows integrated authentication you will likely get the following error:

Error 29115.Cannot authenticate to DB.

Use SQL studio to login to your SQL server, open the server properties then use the less secure option of SQL Server and Windows Authentication mode. Restart the SQL services.

 

 

These instructions reflect using vSphere 5.1.0b, which has a different database setup than the 5.1 GA release. Do not use the 5.1 GA version, as it was in no way ready for production usage.  

Provision vCenter VM

1. Provision one or more VMs for the vCenter 5.1 install. In this blog series I'm assuming an all-in-one server to make things easier. You can certainly split up the services, which would be recommended in medium to large environments.

I provisioned a Windows Server 2008 R2 SP1 VM, with 2 HDs, and all of the latest Windows updates. 6GB of RAM and even 8GB of RAM for an all-in-one server is recommended, otherwise vCenter and SSO will run verrrrrrrry slowly. The 5.1 release has a high memory utilization.

2. Create a domain-based service account (e.g. SVC-VCTR02-001) which the vCenter services will use. Add that account to the local Administrator's group on what will become the vCenter 5.1 server.

You need to ensure the service account also has the "Act as part of the operating system" user right on the vCenter server. If the Administrators group has the right then you are covered. If not, explicitly add the service account to the user right as shown below. Should you need to add the account, reboot the server after so that it takes effect.

3. Open the Server Manager and add the .NET Framework 3.5.1 feature and wait for the install to complete.

Configure SQL Database

1. The SSO service requires a database, as do other vCenter services. In this example we are using SQL Server 2008 R2. SQL Server 2012 is NOT officially supported.  There are some hard coded restrictions in the SSO service which limit your ability to use customized names for all of the fields. In particular the DB name must only include letters, numbers, underscore (_), the at symbol (@) and the hash (#). No periods and no spaces. As of the 5.1.0b release, hyphens are now allowed though. As a reader has pointed out, you should be using SQL Server 2008 R2 SP1 and CU6 or later, which addresses a JDBC SSL issue. You can read the MS KB here. Personally I used SQL 2008 R2 SP2.

I created a custom database creation script (see below), based on the canned VMware script mentioned in the wizard above. In my case I called the database "D001_VMware_SSO". Run this script in SQL studio, modified to your liking. Note that you CAN NOT change "RSA_DATA" or "RSA_INDEX" as the SSO service is hard coded to use them and the install WILL fail if they are not present.

Note that SQL dynamic ports are NOT supported. Make sure your SQL server won't change the port it's listening on, as that will break the SSO service. I've also seen reports of successful installations with SQL clusters, as long as SQL server is listening on a static port. I personally haven't tried a clustered SQL instance. Users have reported that specifying a SQL instance in the JDBC string may not work for all vCenter products, even if the SSO service will initially install fine.

----
USE MASTER
GO
CREATE DATABASE D001_VMware_SSO ON PRIMARY(
    NAME='RSA_DATA',
    FILENAME='K:\Microsoft SQL Server\MSSQL\Data\D001_VMware_SSO_Data.mdf',
    SIZE=10MB,
    MAXSIZE=UNLIMITED,
    FILEGROWTH=10%),
FILEGROUP RSA_INDEX(
    NAME='RSA_INDEX',
    FILENAME='K:\Microsoft SQL Server\MSSQL\Data\D001_VMware_SSO_Index.mdf',
    SIZE=10MB,
    MAXSIZE=UNLIMITED,
    FILEGROWTH=10%)
LOG ON(
    NAME='translog',
    FILENAME='L:\Microsoft SQL Server\MSSQL\Data\Logs\D001_VMware_SSO_Log.ldf',
    SIZE=10MB,
    MAXSIZE=UNLIMITED,
    FILEGROWTH=10% )
GO

EXEC SP_DBOPTION 'D001_VMware_SSO', 'trunc. log on chkpt.', true
GO
CHECKPOINT
GO
-----

 

2. Unlike the 5.1 GA version which automatically created two SQL logins to access the database, in 5.1.0a and later you need to do additional account configuration. In SQL Studio we need to create two logins for the SSO service. These logins need to be created under the main security container in SQL studio, then added to your SSO database.

Create a new SQL user using SQL Server authentication, such as SSO_User, and modify the password options as needed. I would suggest not expiring the password, and certainly can't select the option to change at next login. The password must be 32 characters or less, and cannot contain semicolon (;), double quotation mark ("), single quotation mark ('), circumflex (^), and backslash (\).

 



3. Repeat the user creation process for a second user, SSO_DBA.

4. In SQL Studio for your SSO database you now need to add your two newly created users so they have proper permissions in the database. Open the Users node under your SSO database and select New User. For the SSO_DBA user I granted it db_owner permissions for both the schemas and database. For the SSO_User I only granted db_owner for the database.

Properly configuring a trusted SQL SSL connection requires that we first start the SSO Installation so that the proper Java keystore binaries are present on the system for us to create a trusted keystore with.

SSO Installation

1. Login as the newly created vCenter service account and launch the vSphere installer from the ISO image and you are presented with the following screen.

At this point VMware gives you the option of a "Simple Install" or install each component separately. Since we want to replace SSL certificates in an orderly fashion and in the easiest possible manner, do NOT select Simple Install. We want to deliberately install each service and perform configuration steps along the way.

2. Click on vCenter Single Sign On, then click on Install. Select the appropriate language and wait for the wizard to open. After clicking through the licensing agreements and carefully reading all of the patents, you are presented with a screen with several options.

VMware gives you the option to install multiple instances of the SSO service for high availability. So on the screen below you have the option of creating a new primary node instance, or join an existing SSO instance. Since this is a new deployment, we want to create a primary node.

Even if you don't want multiple SSO instances now, you may want them in the future. You don't need to configure additional ones from the outset, so there's no harm in leaving the door open for future expansion. Thus I selected the second option, as shown below. .

3. Next the installer will prompt you for the password to the default SSO Administrator account. Yes, this is a local account not tied to AD or the Windows host. After SSO is installed, you can configure it for one or more LDAP/AD server and other identity sources, so don't fret too much about this application password but DO remember it.

The password must have at least eight characters, at least one lowercase character, one uppercase character, one number, and one special character. Maximum password length is 32 characters. Passwords longer than 32 characters will be truncated and cause authentication problems.

Note: Do NOT use the following characters, or trailing spaces:

^ (circumflex)
* (asterisk)
$ (dollar)
; (semicolon)
" (double quote)
' (single quote)
) (right parenthesis)
< (less than)
> (greater than)
& (ampersand)
| (pipe)
\ (backslash)

These may cause a "Error 29133.Administrator login error." further on in the installation process. VMware has a KB article regarding these special characters here.

4. At this point you are presented with a dialog asking what kind of database you want to use. I would never use SQL Express in a lab or production environment, so select the second option.

 

 

5. Enter the database information in the window below, using the same details that you configured the SQL server with. If you do NOT want to use SQL SSL encryption you can click Next and proceed to step 6.

If you are concerned about security and want to use an SSL connection to the SQL server you have two implementation options. Both options assume your SQL server is configured for SSL encryption. I wrote a blog article here that you can check out for enabling SQL SSL.

The first, but less secure and easier option, is to amend the JDBC connection string and ignoring any trust issues with the SQL SSL certificate. Not ideal since the secure channel could easily be compromised by a man-in-the-middle attack, but probably better than clear text communications to the SQL server.

The second option requires the configuration of a Java keystore with the public key of your SQL SSL root CA so the JDBC driver can validate the certificate. This is much more secure and the preferred option, but requires more work.

Option 1 (less secure): Force trusting the SQL SSL certificate:

 

On the vCenter Single Sign On screen above, click on Next, then click on Back. You should now see the JDBC URL populated but greyed out. Check the box next to "I will enter the JDBC URL myself". At the end of the connection string add the following two options:



 

;encrypt=true;trustservercertificate=true

 

For example, my JDBC URL is (single line):

 

jdbc:sqlserver://;serverName=D001SQL01.contoso.net;port=1433;

databaseName=D001_VMware_SSO;encrypt=true;trustservercertificate=true

 

Your window should now look like the screenshot below. Click on Next, and if everything is validated, no errors will appear. Go to step 6.

Option 2 (most secure) : Configure fully trusted SQL SSL:

A. Further in SSL replacement process we will also need the public root certificate(s) file in Base-64 encoded format.

A. From the vCenter server go to the Microsoft CA home page that was used to create the SQL server certificate and click on Download a CA certificate, certificate chain or CRL. Change the encoding method to DER and click Download CA certificate chain. Change the file name to cachain.p7b.

B. Double click on the downloaded certificate file, then locate the certificate in the console. Right click on the certificate and select All Tasks -> Export. Select DER encoded and save the certificate with a filename of Root.cer.
C. Open an elevated command prompt then change directory to where the SSO installer files extracted themselves and find the jre\bin directory. In my case this was:

C:\Users\SVC-VCTR02-001\AppData\Local\Temp\{DEC4C346-414B-4814-9BF3-CAC14154B55A}\jre\bin

D. Create a Java keystore by issuing the following command (using the path of your SSO service account). Enter a keystore password (write it down). When prompted to trust the certificate type Yes.

keytool -import  -alias root -file c:\root.cer -keystore c:\users\SVC-VCTR02-001\keystore.jks

 

E. With the keystore now created, it is time to update the JDBC connection string back in the SSO installer. Add the 'truststore' and 'truststorepassword' connection options with the appropriate path and password.

Update: If you get any errors, you may need to double back-slash the trust store path (e.g. C:\\Users\\SVC-VCTR02-001\\keystore.jks).

Add these connection options:

;encrypt=true;truststore=C:\Users\SVC-VCTR02-001\keystore.jks;truststorepassword=testpassword

Full connection string example:

jdbc:sqlserver://;serverName=D001SQL01.contoso.net;port=1433; databaseName=D001_VMware_SSO;integratedSecurity=true; encrypt=true;truststore=C:\Users\SVC-VCTR02-001\keystore.jks;truststorepassword=testpassword

6. With your database details now properly configured, and maybe even using SSL to your SQL server, we can proceed with the SSO installer. If you were using a hardware load balancer, you would enter the FQDN of the VIP. Since I'm just installing one SSO instance, I'll stick with the FQDN of the vCenter server.

 

7. At this point input the vCenter service account details. Note that if you input the wrong password you will get an error "Could not find the specified user on provided domain." which is not entirely correct. The user exists but you just fat fingered the password.

 

 

 

8. For the installation path I left the default, as the installer has had problems in the path with custom paths or "unusual" characters in the path.





9. On the next screen I left the HTTPS port the default, then sent the installer off on its merry way.

At this point the vCenter Single Sign On service should have successfully installed. If you have more SSO installation problems than I've covered here, check out their 5.1.0b release notes for a laundry list of gotchas for the SSO installer. Next up is creating all of the SSL certificates that the vCenter services require. You can check out Part 2 here.