UPDATE 10/27/2012:I've updated the procedures below to reflect vCenter 5.1.0a and the official VMware procedures. You can find the full VMware KB article here.
UPDATE 11/20/2012: VMware has updated the KB article here with additional information if you are subordinate CAs. Since my lab consists of only a single CA, I don't cover the subordinate CA steps below. Refer to the KB article for more details.
--
In Part 13 we covered basic VUM configuration tasks. To complete the vCenter SSL configuration we now need to update the certificates for the Web Client and Log Browser service. This installation process assumes you are using vSphere 5.1.0b.
1. Stop the VMware vSphere Web Client service and the VMware Log Browser service.
2. From your D:\Certs\WebClient directory copy rui.crt, rui.key and rui.pfx to the directory below.
C:\ProgramData\VMware\vSphere web client\ssl
3. From your D:\Certs\LogBrowser directory copy rui.crt, rui.key and rui.pfx to the directory below.
C:\Program Files\VMware\Infrastructure\vSphereWebClient \logbrowser\conf
4. To un-register the Web Client with the SSO service, open an elevated command prompt and type:
set JAVA_HOME=c:\Program Files\VMware\Infrastructure\JRE
cd /d C:\Program Files\VMware\Infrastructure\vSphereWebClient\SsoRegTool\
regTool.cmd unregisterService -si "C:\Program Files\VMware\Infrastructure\vSphereWebClient\serviceId" -d https://SSOServer.domain:7444/lookupservice/sdk -u admin@System-Domain -p YourPassword
5. To Register the service type the command below. Note: The directory paths appear to be case sensitive, so make sure they match exactly what your system has. In particular, the "ssl" directory may need to be in all lower case.
regTool.cmd registerService --cert "C:\ProgramData\VMware\vSphere Web Client\ssl" --ls-url https://SSOServer.domain:7444/lookupservice/sdk --username admin@System-Domain --password YourPassword --dir "C:\Program Files\VMware\Infrastructure\vSphereWebClient\SsoRegTool\sso_conf" --ip "*.*" --serviceId-file "C:\Program Files\VMware\Infrastructure\vSphereWebClient\serviceId"
If successful you should see something like the following:
6. Navigate to C:\Program Files\VMware\Infrastructure\vSphereWebClient and open the ServiceID file in Notepad. Remove all entries except for the two IDs which were listed from the previous registration step (In my case they were :29 and :30, which I highlighted in yellow).
7. Restart the VMware vSphere Web Client and VMware Log Browser services. Wait a good five minutes for them to initialize.
8. Login to the Web Client (using a vCenter admin account, not the SSO admin account) and at this point you should be able to open the Log Browser without any errors and look at the available logs:
You are almost there....proceed to Part 15 to learn how to replace your ESXi host SSL certificates.
If anyone has missed the step to save the password from step 1 I have sent Derek a quick SQL script to pull the CA cert out of the SSO database and I am sure he will update with the needed steps.
ReplyDeleteInteresting. On the VCSA, it appears that the command to update the certificates also updates root-ca cert in the SSO root-trust.jks keystore.
ReplyDeleteI was able to pull a root-ca cert out of the root-ca.jks file, but it didn't fix the log browser problem.
Also, on the VCSA, the keytool command requires the '-rfc' parameter to output a plain-text PEM certificate. Otherwise it comes out in binary DER format.
I'm working on a newly deployed VCSA, to see if the cert extracted from the root-trust.jks keystore is different...
Ok, this fix works for the VCSA as well. However, I couldn't find the appropriate self-signed cert anywhere on the system after a successful certificate change. It is very easy to save a copy of the right cert before replacing them with CA-signed certs, though.
ReplyDelete---
cp /etc/ssl/certs/Embedded-SSO-Server-Root-CA.pem /etc/ssl/certs/SSO-STS-Root.pem
---
Then after replacing the certs, run these commands to update the logbrowser.properties file and restart the service.
---
sed -i.bak 's/.*sso-certs=.*/sso-certs=\/etc\/ssl\/certs\/SSO-STS-Root.pem/' /usr/lib/vmware-logbrowser/conf/logbrowser.properties
service vmware-logbrowser restart
---
I'll be updating the script I published on the VMware community discussion to handle this. http://communities.vmware.com/message/2124455#2124455
You done a wonderful job, I could not install Vcenter without your blog.
ReplyDeleteWe created 3 databases, but we used only 2, one with SSO installtion, the 2nd one for VUM but waht about database for Vcenter (you created 2 databases with a script).
Please explain?
Part 7 covers the installation of vCenter, which uses the vCenter database.
DeleteDerek, Epic Work! - VMware grow up, and start releasing some decent TESTED code!
ReplyDeleteHi,
ReplyDeleteThanks for all this work!
Unfortunately, I am not able to get the LogBrowser working...even after having loaded the new chain in the STS certificate, I still have the error...
Is the Logbrowser really working for you?
Hey, When i try add rui.jks file in step 10 i have error:
ReplyDeleteThe last operation failed for the entity with the following error message.
An error ocurred while updating server configuration.
Can you help on this? thanks
@Luk: I had that happen with the 5.1 GA release. But with 5.1.0A and using the updated instructions from VMware, which I've reflected in my post, it worked for me.
ReplyDeleteDerek - Why can't we prepopulate the logbrowser certs? What is the reason for that?
ReplyDelete/Jorgen
everything works except logbrowser!
ReplyDeletei have SSO on separate VM and everything else on another VM.
i have 3 certs in the chain (Cert+Intermediate+Root).
1 weeks i'm trying every keytool combinations possible found on google:
the 2 CAs never show in WebClient STS Certificate update!!!
any clue?
Hi Derek, great article !!!
ReplyDeleteUnfortunately I get the same error as @Luk, even with the 5.1.0A release.
"...When i try add rui.jks file in step 10 i have error:
The last operation failed for the entity with the following error message.
An error ocurred while updating server configuration..."
I have replaced all certificates succesfully, only this last one for the LogBrowser fails....
I would open a case with VMware. I had that error with 5.1, but with VMware's instructions and 5.1.0A, it worked for me.
DeleteThanks Derek, just opened a case with VMware, will keep you updated.
DeleteWe have a root and subordinate CA, and I think this got confusing somewhere. VMware new KB articles are not everywhere clear for this configuration.
Cheers,
Harold
I have this same problem. I'm watching vsphere_client_virgo.log and it generates the error 'keystore was tampered with or password was incorrect'.
DeleteChecked for any typos in the cmd and tried another password but still the same error. Still investigating.
Cheers
Kevin
After a few days working with VMware Support we have a workaround.
DeleteTake Root64.cer from top level CA-auth > copy contents into new file.
Take Root64.cer from subordinate CA-auth > copy contents into bottom of the same file with no extra lines or spaces.
Save file as myca.cer
create rui.pfx pointing to myca.cer (not Root64.cer)
openssl pkcs12 -export -in c:\certs\sso\rui.crt -inkey c:\certs\sso\rui.key -certfile c:\certs\myca.cer -name "rui" -passout pass:testpassword -out c:\certs\sso\rui.pfx
Then run from step 7 onwards.
Hope this helps.
Cheers
Kevin
That worked for me! Had the same issue and no info on vmware articles about this when you have a subordinated ca.
DeleteThanks!
Cheers
Ola
Would just like to add that the VMware kb has now been updated with this additional info if you have a subordinate ca.
DeleteStep 20.
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2037432
Cheers
Kevin
Thanks Kevin, your instructions helped get me past being stuck because I have a subordinate CA!
DeleteArrrgh, so frustrating. I am running 5.0.1a and I don't have a subordinate CA. I followed VMware's instructions in KB:2035010, Derek's steps and Kevin's steps a few comments above with no luck. I'm still getting the "An error occurred while updating server configuration" error. I got all of the previous 350 steps to work with no problem replacing all the certs. This is my only remaining stumbling block. What am I missing?
DeleteKevin Thanks for your help!
Delete@Kevin
DeleteCan you supply a detailed step by step guide of your post from "November 13, 2012 at 9:03 AM" please?
Thanks.
Best regards
Alex
Hi Derek, I'd like to add my thanks for the article.
ReplyDeleteStep 4, the file not found error is because the SSL folder is in upper case. This is set as upper case when you perform the pre-population of certs.
The cmd you run here it's in lower case.
Just make sure they match and there will be no error.
Cheers
Kevin
@Kevin: That didn't solve the error.
DeleteIt was VMware Support that asked me to change the SSL folder to lower case and it resolved the error for me. I'm stuck now at the same point as Harold. Ticket is still open with VMware.
DeleteCheers
@Anonymous: Thanks, I've updated step #4, and the pre-staging script. I still am having problems making it work, but maybe that change will help out others.
DeleteHi Derek,
ReplyDeleteHope you are well. Thank you for you tutorials, I have a my test lab setup with my own certs. as I am new to this, it's fantastic that you have simple instructions for people like me. I have the same issue as Harold, but using the original 5.1 version.
I am downloading the 5.1.0a iso at the moment, how do I upgrade? Just install on top of existing or do I need to start fresh again with a new install?
Regards
H
@Limbada: Given the fragility of the installation, personally I would start fresh. A pain, but has the highest chance of success.
DeleteThis comment has been removed by the author.
ReplyDeleteHi Derek,
ReplyDeleteAgain everything seems to go fine but after step 10 when clicking ok we get the following message:
The last operation failed for the entity with the following error message. An error occured while updating server configuration.
The STS Certificates screen shows a first line with the correct CN. The second line keeps showing CN=RSA identity as before updating....
Any clues?
@Derek: The post from Kevin fixed it!
ReplyDeleteDerek, thank you very much! Just a little remark to add: In part 14 to error "Invalid file location" and "Return code is: CertificateStoringFailed" while registering the Web Client Service is because of small "copy & paste" error. You got a space between " C:\ProgramData\VMware\...". Change "regTool.cmd registerService --cert " C:\ProgramData\VMware\vSphere Web Client\ssl" ...." to "regTool.cmd registerService --cert "C:\ProgramData\VMware\vSphere Web Client\ssl" ....". After that, the return code will be:"Success" ;-)
ReplyDelete@Thomas: Thanks..that worked! Updated the screenshot and post to reflect success!
ReplyDeleteOh My God! - what an epic journey!
ReplyDeleteThank you so much for your diligence, persistence, patience and perseverance.
Dear Derek,
ReplyDeleteFirt off, what a brilliant tutorial you made. I was able to upgrade complete Vcenter and without any big issue.
Unfortunately I do run into a problem no at step 3, with the message 'cannot find the path specified'
I checked if the files were there, copied the path directly from the explorer, but nothing seems to work...
Am I missing something obvious?
Many thanks Johan
Hi Derek, Perfect job, I have done every step en the new vCenter is running like a charm.
ReplyDeleteI only have trouble with the logbrowser (the unregister part at step 4) for some strange reason it keeps saying:
The system cannot find the path specified.
3
I tried everything, even editing the command file, with no result. Filename is correct, capital letters are correct, but nothing works.
Have you got any idea?
I have a strange behaviour too... faultCode:Server.Processing faultString:'javax.servlet.ServeletException : java.lang.Exception: https://server.domain.com:port/vmwb/logbrowser: Unauthorized access 'faultDetail:'null' shows when I try to open LogBrowser as user different than admin@System-Domain. After logging once as admin@SD login as other administator works too till next restart.
ReplyDeleteHi Derek, thanks for your great work also thanks to all others for their helpful posts. I have a problem that i canot solve. After i perform part 13 i get at log browser the following error: faultCode:Server.Processing faultString:'javax.servlet.ServletException : java.lang.Exception: https://SSOServer.domain:12443/vmwb/logbrowser: Unauthorized access
Delete' faultDetail:'null'
I have the same error before and after a i perform part 14.
additional information:
- existing subordinate CA
- installed release: 5.1.0b
Anyone can help?
Best regards
Alex
I have the same problem. When I try to open LogBrowser I have this in log file (C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\log\com.vmware.vide.ws.product-2012-12.0):
ReplyDelete2012-12-04 09:13:24,857 INFO [com.vmware.vim.sso.client.impl.SamlTokenImpl] [qtp454296260-39] SAML token cannot be constructed: Signature validation failed
2012-12-04 09:13:24,864 ERROR [com.vmware.vide.ws.sso.SsoRequestHandler] [qtp454296260-39] Unable to create token
com.vmware.vim.sso.client.exception.InvalidTokenException: Signature validation failed
Regards!
@Maslanka Can you check C:\ProgramData\VMware\SSL\ca_certificates.crt to make sure there is only one certificate in the file. You may also see an additonal hash file (xxxxxx.1). I have seen instances of this file being appended with additional certs. If it does then copy over the original Root64.cer and and rename to ca_certificates.crt. The other hash file can be removed. Restart log browser service.
DeleteGot the same error, please help to resolve
DeleteI see Auto Deploy uses certs as well. Can I use the same steps to get those self signed?
ReplyDeleteRegards
@Anonymous: I have not deployed AutoDeploy, so I can't speak to that certificate replacement process.
DeleteThis comment has been removed by the author.
ReplyDeleteI got a problem with the following log output: [com.vmware.vide.ws.sso.SsoRequestHandler] [qtp434033865-38] Using certificate file C:\ProgramData/VMware/SSL/ca_certificates.crt
ReplyDelete2012-12-17 16:33:59,256 ERROR [com.vmware.vide.ws.sso.SsoRequestHandler] [qtp434033865-38] Unable to load certificates
java.security.cert.CertificateParsingException: invalid DER-encoded certificate data
Every other Service is up and running and the ca_certifiates.crt is base64 but Logbrowser always got me the unauthorized error.
I changed the Path in logbrowser.properties (%programfiles%\VMware\Infrastructure\vSphereWebClient\logbrowser\conf) ca_certificates1.crt witch i exportet as DER format and now its working :)
Hi, we had the same problem with LogBrowser. Our problem was, that the content in the C:\ProgramData\VMware\SSL\ca_certificates.crt had a double instance, that means the CA Certificate existed twice. We removed one entry, restarted the logbrowser service and it is working.
DeleteI was able to install the web client by renaming the SSL to ssl in C:\ProgramData\VMware then i turned off all vmware services except the sso service, run install again. At one point it asked to overight the certificate and i accepted.
ReplyDeleteDerek, you are ranked among the highest in nerdom. Many thanks, I have spent over 50 hours trying to upgrade and it would have been impossible. My biggest hurdle was ca_certificates being saved as a cer instead of crt (spent 14 hours on that). I did notice a difference on pages between my work computer and laptop, even during a refresh (the page with the hash instructions could not be found on my laptop. I would recommend you put something at the beginning of the guide to read every page in and EVERY comment. I ran into most of the issues that commenters were able to fix. Kudos to you and a big FU to VMware.
ReplyDeleteThanks,
Zac
@Zac: Thanks! Yes I was actively updating pages this weekend, and moved the hashing instructions. They are now back, but in a different place to better match VMware KB articles. I still have posts 8-14 to refresh.
DeleteExcellent Job!
ReplyDeleteone question, I have added the JKS from the web client, after restart SSo servce, I couldn't login anymore, error message is "signature validation error"
Any idea? Thanks a lot.
@Anonymous: I would use the keystore tools to validate that your keystores contains valid information. Are you using an internal MS CA? vCenter supports both SHA1 and SHA256 certificates. I used SHA256 on my last install without issues. I'd also make sure your Windows certificate store trusts your root CA as well.
DeleteWhat exactly does "Windows certificate store trusts your root CA" mean?
DeleteI'm new to MS CA and having same problem.
If there are some settings for MS CA, would you show them too?
Hello,
ReplyDeletei had also trouble with intermediate ca and i have find a solution for my problem.
problem:
If you try to open the Log-Browser in vSphere Web Client you get the following error:
faultCode:Server.Processing faultString:'javax.servlet.ServletException : java.lang.Exception: https://BLPKVSPHERE01.BLPK2000.blpk.ch:12443/vmwb/logbrowser: Unauthorized access
' faultDetail:'null'
have a look at the logs from Log-Browser under the following directory:
C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\log
in my case the following error occurred:
2013-02-06 16:40:22,468 INFO [com.vmware.vide.ws.sso.SsoRequestHandler] [WrapperStartStopAppMain] Using certificate file C:\ProgramData/VMware/SSL/ca_certificates.crt
2013-02-06 16:40:22,478 ERROR [com.vmware.vide.ws.sso.SsoRequestHandler] [WrapperStartStopAppMain] Unable to load certificates
java.security.cert.CertificateException: java.io.IOException: DerInputStream.getLength(): lengthTag=127, too big.
solution:
- be sure the steps 1 to 8 of Part 14 are well done
- stop VMware Log Browser service
- rename the certificate of your intermediate ca to ca_certificates_2.crt
- copy ca_certificates_2.crt to C:\ProgramData\VMware\SSL
- copy logbrowser.properties from C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\ to desktop or an other place and edit it
ORIGINAL File:
admin-port=12221
secure-storage=../conf/storage
https-port=12443
https-maxidle=3000
https-keystore=../conf/rui.pfx
https-keystore-password=testpassword
https-keystore-type=pkcs12
sso-certs=%PROGRAMDATA%/VMware/SSL/ca_certificates.crt
NEW File:
admin-port=12221
secure-storage=../conf/storage
https-port=12443
https-maxidle=3000
https-keystore=../conf/rui.pfx
https-keystore-password=testpassword
https-keystore-type=pkcs12
sso-certs=%PROGRAMDATA%/VMware/SSL/ca_certificates_2.crt
- copy the new logbrowser.properties to C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\ and replace the original file
- start VMware Log Browser service
- wait a couple of minutes
-> and now, it works...
PS. Derek, really great job! Thanks!
Greets Alex
Hi,
DeleteIn my case just the IntermediateCA cert didn't do the job, I had to build the leaf-to-root chain, simply by:
copy IntermediateCA.cer+RootCA.cer %PROGRAMDATA%\VMware\SSL\ca_certificates_lb.crt
and editing the logbrowser.properties file accordingly.
However, just as Michal Grabowski wrote on December 6th 2012, the admin@System-Domain has to logon - logout. Afterwards the administrators can select objects and retrieve logs till the next vCenter restart.
Thanks Derek and all you others for a great help.
Zbynek
Alex's fix (just adding the intermediate CA and editing the properties file) worked for me.
DeleteThanks Alex.
And again, a BIG THANKS to Derek for putting all of this together.
- A slightly less frustrated Admin from San Antonio, TX
Derek Seaman is my new hero. Thank You Thank You Thank You.
ReplyDeleteHi Derek,
ReplyDeleteI have managed to resolve the issue; I came across this blog http://jackstromberg.com/tag/vcenter/ and followed the steps as below and that has resolved the problem:
1. Stop the VMware Log Browser service
2. Navigate to C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf via Windows Explorer
3. Backup existing rui.crt, rui.key, rui.pfx.
4. Replace the rui.crt, rui.key, rui.pfx files with the ones from C:\certs\LogBrowser
5. Next, create a new java keystore with the chain trust for the certificate
1. Open up an elevated command prompt
2. Change directories to the VMware JRE
cd /d C:\Program Files\VMware\Infrastructure\jre\bin
3. Generate the new keystore (Do not change the testpassword or changeit password) (Change the Paths to point to your SSO pfx certificate e.g. C:\certs\SSO\rui.pfx and the destination path to output the java Keystore e.g. C:\certs\SSO\JKS\rui.jks)
keytool -v -importkeystore -srckeystore C:\certs\SSO\rui.pfx -srcstoretype pkcs12 -srcstorepass testpassword -srcalias rui -destkeystore C:\certs\SSO\JKS\rui.jks -deststoretype JKS -deststorepass changeit -destkeypass changeit
4. Copy the rui.jks from C:\certs\SSO\JKS\rui.jks to:
C:\Program Files\VMware\Infrastructure\SSOServer\Security\
5. Login to your vSphere Web Client with the admin@System-domain account
6. Navigate to Administration > Sign-On and Discovery > Configuration
7. Click on the STS Certificate tab and the click Edit
8. Select the rui.jks file from C:\Program Files\VMware\Infrastructure\SSOServer\Security\
9. When prompted for a keystore password enter changeit
10. Click on the rui line to highlight it, then click OK
11. Enter changeit again for the password
12. Acknowledge the dialog box that says you need to restart the server in order for the changes to take effect.
13. Reboot your server
6. Log back into your vSphere Web Client
7. Click on the Log Browser link and verify the error has been resolved
I am not sure what I did wrong to get the error in the first place.(?)
AS