Derek Seaman's Blog

VCP5 Upgrade Deadline Looms

2012-01-29T19:13:00.000-08:00

For those of you that have a VMware VCP4 and want to take the VCP5 exam without additional classroom training requirements, your time is short! February 29th, 2012 is the last day you can take upgrade exam. For some good study material, I recommend the vSphere Resource Kit site. It has a great interactive VCP5 practice exam. Is it worth the nominal fee? Yes!

VMware does have a 7-day waiting period for re-taking the exam. So I would encourage you to book your exam ASAP, so if you do fail you can retake before the end of the month. Slots may fill up very quick, so check with your nearest exam provider and book your test today.

In the interest of full disclosure I know the primary author, Chris McCain, and he graciously listed me as an author. But I get no kick back whatsoever. I just think it's an excellent tool for your journey to a VCP5.

Microsoft Security Compliance Manager 2.5 Beta hits the streets

2012-01-25T20:45:00.000-08:00

Microsoft SCM can be a great tool for configuring and maintaining security baselines for various Microsoft products such as Windows operating systems, Exchange, IE and Office. In the past I've used it to establish golden OS security baselines that then get exported and baked into our VM templates and physical image discs.

Two major issues I've had with past releases was no easy way to import existing GPO state data from a "model" computer. So you had to either start from scratch with SCM and define your baseline or use a MS baseline and modify as needed. Neither way was very time efficient. Configuring standalone machines was easier, as they included a "localgpo" tool. But the process could be easier.

SCM 2.5 beta addresses these issues, and adds other enhancements as well. The release notes mention the following new features:

Integration with the System Center 2012 IT GRC Process Pack for Service Manager-Beta:Product baseline configurations are integrated into the IT GRC Process Pack to provide oversight and reporting of your compliance activities.
Gold master support: Import and take advantage of your existing Group Policy or create a snapshot of a reference machine to kick-start your project.
Configure stand-alone machines: Deploy your configurations to non-domain joined computers using the new GPO Pack feature.
Updated security guidance: Take advantage of the deep security expertise and best practices in the updated security guides, and the attack surface reference workbooks to help reduce the security risks that you consider to be the most important.
Compare against industry best practices: Analyze your configurations against prebuilt baselines for the latest Windows client and server operating systems.
NEW baselines include:

Exchange Server 2007 SP3 Security Baseline
Exchange Server 2010 SP2 Security Baseline

Updated client product baselines include:

Windows 7 SP1 Security Compliance Baseline
Windows Vista SP2 Security Compliance Baseline
Windows XP SP3 Security Compliance Baseline
Office 2010 SP1 Security Baseline
Internet Explorer 8 Security Compliance Baseline

I've found previous versions of SCM to be a valuable tool, and these enhancements make it all the better. You can find the beta on Microsoft connect here. If you haven't used it before, I would encourage you to try it out, if you value standizing your security baseline for a variety of MS products.

VMware Workstation 8.02 Released

2012-01-25T20:30:00.000-08:00

VMware just released Workstation 8.0.2, up from 8.0.1. This is not a major release, but has a few enhancements:

Ubuntu 11.10 is supported as a host.
Fedora 16 is supported as a guest.

Fixed bugs include:

The ACLs on the PID files needed improvement. VMware thanks Inode0 for bringing this to our attention.
Releasing input from the guest to the host, for example, moving the cursor from the virtual machine window to the host screen, failed with an unrecoverable error.
Copying and pasting from a guest to an Ubuntu 11.10 host failed.
Shared folders did not work in Fedora 16 and OpenSuse 12.1 guests.
In Ubuntu 11.10 hosts, key repeat was disabled after ungrabbing or quitting VMware Workstation.
On a Windows host, a virtual machine configured to use a physical disk or partition failed to power on if the host had a volume backed by more than one physical disk, for example, a RAID system.
On a machine with Microsoft Visual Studio 2010 SP1, vix-perl installation failed with dynamic link errors.

This is a free update for all Workstation 8.0.2 owners. Workstation 8.0.1 included dramatic graphics acceleration with IE9, so if you are still on 8.0.0 and using IE9 in guests, you really should upgrade.

You can download Workstation 8.0.2 from here and view the release notes here.

Get your free 2 socket Veeam Backup 6.0 License key

2011-12-16T17:09:00.000-08:00

Veeam is once again running a Christmas special where you can get a free NFR (not for resale) Veeam Backup 6.0 license key for 2 sockets which is good for 1 year. You can select VMware, Hyper-V, or both. All you need to do is fill out this form and wait for the email with the license key.

For home labs or just trying out Veeam without their more limited timed trial versions, this is a great opportunity. Even if you don't think you will use the key, I'd grab one anyway since you never know what may come up over the next year where it could come in handy.

Veeam is targeting the offer at certified VMware professionals such as VCP, but they don't require any identifying information.

Schedule VMware UMDS Downloads with Windows Task Scheduler

2011-11-13T11:15:00.001-08:00

VMware UMDS (Update Manager Download Service) is a product which ships with vCenter that allows you to download patches for VUM, then use them in air-gapped networks where VUM can't directly download updates from the internet. UMDS 5.0 has some nice enhancements from 4.x, which helps limit the amount of unnecessary patches it downloads.

If you work in an environment where you need to utilize VMware UMDS you probably want the download process to be as automated as possible. One of the missing features in UMDS is a scheduler, so that's where you can leverage the Windows Task Scheduler to make life easier. I'm assuming you are using Windows Server 2008 R2 and have already installed UMDS on a server. Creating the task is pretty quick and simple.

1. Launch the Task Scheduler. To keep the tasks more organized, I created a folder at the root level under Microsoft called VMware.

2. Right click on VMware and select Create Task. On the General tab you configure the basic task information. I recommend you change the task to use a privileged account, such as SYSTEM, and check the box to run the task with highest privileges. You could configure service account with only the required rights and use that instead, if running a task with SYSTEM rights doesn't sit well with you.

3. Click on the Triggers tab and add a new trigger. I like to run the task once a day, at 4AM.

4. On the Actions tab create a new action and configure it as shown below. Change the path as needed to the location of the vmware-umds.exe file. Be sure to add the argument of -D or nothing will happen with the task runs.

5. On the Settings tab I changed the parameter that limits how long the task can run, but this is optional, and you can use any value you want.

6. At this point the task is configured and you can click OK. To test it out you can right click on the task and Run it. Monitor the download directory you configured and make sure it is being populated. For ESXi 4.1.0 and ESXi 5.0 patches, the UMDS repository at this time of this article was about 1.5GB.

Once the download task completes, you then need to "export" the repository into a directory, copy to removable media, and upload to the air-gapped instance of VUM. You can find all of the gory details in the Installing and Administering VMware vSphere Update Manager 5.0 Guide.

XenDesktop 5.5 Resource Calculator

2011-10-20T20:34:00.000-07:00

Of course after a weekend of creating my own spreadsheet to calculate my storage requirements for XenDesktop Andre Leibovici created a XenDesktop version of his View calculator. This is a great resource for sizing your storage, calculating IOPS, number of datastores, and other details. A sample of the fields is below.

If you are using XenDesktop MCS, this is a must-use calculator. He says a PVS version is coming as well, so if you aren't a MCS user then check back with his site for an update.

Tips for measuring Windows 7 VDI IO Requirements

2011-10-15T15:02:00.000-07:00

When sizing your storage subsystem for a VDI implementation, it's extremely critical to understand how your VMs will behave and the resulting IO load. Miscalculate and you will suffer poor performance and angry users. Oversize your array and you will waste money. However, measuring your VM performance may not be as straight forward as you think.

A few months ago I posted a script here that let you dump basic IO performance stats for a VM on vSphere 4 and 5. But as you will see, the applications you load into your image and when you measure the performance has a significant impact on the collected metrics.

For my first round of tests I wanted to focus on the boot performance of Windows 7 64-bit. Booting can be one of the most taxing events (aside from full virus scans) on your VDI storage subsystem. Even if you stagger your VM boots over a few hours as a normal practice, what if you have a power outage or significant hardware failure and you need to rapidly power on hundreds of VMs? Will your storage array melt under the load? Will Windows boot so slowly that it will blue screen (hint: Windows 7 VMs should boot under 5 minutes to avoid problems.) SLAs play an important role here and you need to be mindful of them and verify they can be met.

The test environment is pretty basic and includes vSphere ESXi 5.0, XenDesktop 5.5, and Windows 7 64-bit. The IO measurements were performed over a five minute period after powering on the VM, and metrics were collected via my script. The measurements are only for boot IOs, as no user logged into the VM during the collection process. Tests were performed four times for each scenario and the results averaged. Five scenarios were tested:

Base Image: Windows 7 64-bit, Office 2010, VMware tools, joined to a domain
VDA Only: XenDesktop 5.5 Virtual Desktop Agent
VDA/Symantec: Citrix VDA and Symantec End Point Protection 12.1
Optimized: Quest vWorkspace Desktop Optimizer applied with all settings enabled except 15, 26, 27, 30; most VMware Windows 7 optimizations applied.
XenDesktop VM: VM created with XenDesktop 5.5 MCS from the optimized template

Drum roll for the results please!

As you can see in the table above, the base Win7 image required an average of nearly 15,000 IOs to boot. 15% of those IOs were writes, while the remainder were reads. Simply installing the XenDesktop VDA decreased the number of write IOs, but increased overall IOs by 17% over the base image. Next up is installing Symantec 12.1, and wow look at those numbers jump! 212% increase in IOs over the base image. Using the Quest and VMware recommended optimizations IOs dropped a bit, but nothing substantial.

What I found to be very interesting is what happened to the IOs when the optimized VM template was cloned by XenDesktop MCS and booted as part of a desktop pool. Zero changes were made to the VM, so the only difference is how the VM behaves when under the control of the Citrix Desktop Studio. Approximately 8000 more IOs are required during the boot process, and a lot more writes are taking place. I would not have guess that large of a delta, so this is an interesting find. The read/write ratio also drops to approximately 80/20.

So what does all of this mean? First, every environment is very unique and you should not use my results, or anyone elses, to estimate the IO load for your environment. Second, take your metrics from a provisioned VDI VM (VMware View, XenDesktop, etc.) and don't just take measurements from your VM template. Third, booting a VM is very IO intensive and if you only size your storage for steady-state IOPS, then boot storms will cause you major headaches.

Depending on the script/method you use to gather the VM IOPS stats, VMware may not always return the read/write stats in the same fashion resulting in the same order, so you may see inverted data. From my observation this happens on a per-VM basis, even through reboots and power on/off cycles. So if your data looks odd, question it, don't assume everything is legit.

Unattended vSphere Utility Installs

2011-09-10T17:17:00.000-07:00

Sometimes you may want to install the various vSphere utilities (PowerCLI, vSphere CLI, vSphere Client, and VUM PowerCLI) to non-default directories, or use a silent/unattended installation to automate the process.

Below are four batch files that you can run which will install the respective tool to the custom installation directory specified. What's cool about the batch file is you can double click on the batch file from any path and it will CD to the location of the installer and run it. If you are using Windows Server 2008/R2 with UAC, you will be prompted to elevate to do the installation, but otherwise there is no interaction required.

The VUM PowerCLI extensions can't be configured for a custom installation directory, so it will just silently install to the default location. You could of course also combine all of the commands and install all of the tools with a single click, silently.

I also included a silent installation of OpenSSL, which can be handy for creating ESXi, vCenter and VUM certificates.
----
cd /d %0\..
start /wait VMware-PowerCLI-5.0.0-435426.exe /q /s /w /L1033 /V" /qr INSTALLDIR=\"D:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\"
----
cd /d %0\..
start /wait VMware-viclient-all-5.0.0-455964.exe /q /s /w /L1033 /v" /qr INSTALLDIR=\"D:\Program Files (x86)\VMware\Infrastructure\"
----
cd /d %0\..
start /wait VMware-VSphere-CLI-5.0.0-422456.exe /s /v"/qb INSTALLDIR=\"D:\Program Files (x86)\VMware\Vmware vSphere CLI\\""
----
cd /d %0\..
start /wait VMware-UpdateManager-Pscli-5.0.0-432001 /q /s /w /L1033 /V" /qr
----
cd /d %0\..
Vcredist_x64.exe /q /norestart
Win64OpenSSL-1_0_0d.exe /verysilent /sp-

vSphere 5.0 Documentation Links

2011-09-04T19:32:00.000-07:00

For those of you that want easy access to vSphere 5.0 documentation, I stumbled upon a location that has well organized PDF and e-book resources. No more need to search all over VMware's site for a specific piece of documentation. You can check out the link here.

VSP3116: Resource Management Deep Dive

2011-09-01T15:22:00.000-07:00

I finally managed to get into a session by one of the VMware rockstars, Frank Denneman, who has co-authored several books that I highly recommend. Frank stated this topic could be a four day class alone, and this was just an hour, so it went quite quickly and just scratched the surface of the topic at hand. But nonetheless it was informative.

Highlights:

Resource entitlement

Dynamic: CPU and memory
Static: Shares, reservations, limits

Short term contention

Load correlation - Where two servers ramp up/down together (e.g. web and SQL)
Load synchronicity - All servers hammered at once (user logon storm at 8am)
Brown outs - System wide virus scanning at the same time

Long term contention

Ultra high consolidation ratios
Hardware limits exceeded
Massive overcommitment

VM-Level shares

Low (1), Normal (2), High (4)

VM CPU Reservation

Guarantees resources
Influences admission control
CPU does not use resources when VM doesn't need processing time (fully refundable)
CPU reservation does not equate to priority

VM Memory Reservation

Guarantees a level of resources
Influences admission control
Non-refundable. Once allocated it remains allocated.
Will reduce consolidation ratios

VM Limits

Applies even when there are enough resources
Often more harmful than helpful (don't use them often unless you like a hole in your foot)
Can very likely lead to negative impacts since the guest OS is not aware of the limits
Any extra memory the guest OS wants comes from swap (after TPS, memory compression), which is very slow.
De-schedules the CPU even if their are resources available and the VM wants them

DRS treats a cluster as one large host
Resource pools - Do not place VMs at the same level in the vCenter hierarchy as a resource pool. Always put VMs inside the appropriate resource pool.
Simple method to estimate resource pool shares

Step 1: Match defined SLA to pool (.e.g. 70 to production, 20 to test, 10 to dev)
Step 2: Make up shares per VM (.e.g. 70/Prod, 20/test, 10/dev).
Step 3: Based on the number of vCPUs per pool multiply shares per VM * vCPUs

E.g. 10 vCPUs for Prod = 700 shares; 5 vCPUs for test = 100 shares; 20 vCPUs for dev = 200 shares.

Schedule at task to do these calculations and set the shares per pool on a nightly basis. As you add VMs and change vCPUs new calculations are needed. Check out Frank's blog for an example script.

When you configure pool limits remember that each VM has overhead, which is between 5-10% of the total memory. Less VM overhead in ESXi 5.0 than previous versions.
Use resource pool limits with care as they can do more harm than good.
DRS affinity rules

Must run on - Cannot violate under any circumstances. You cannot even power on the VM if it's on the wrong host. Always honored, even though HA events like host failures.
Should run on - Can be violated as needed, such as during HA events.
NOTE: You must disable Must Run On or Should Run On rules BEFORE you disable DRS, as those settings are honored even when DRS is disabled and you can't change the rules when DRS is disabled.

Distributed Power Manager (DPM)

Frank did a poll of the room and hardly anyone is using this feature.
vCenter looks at the last 40 minutes and the host must be completely idle to be suspended.
If vCenter senses a ramp up in resource requirements in the last 5 minutes it will take the server out of stand by.
DPM will NOT degrade system performance to save power

Resource pools, shares, limits and reservations can be quite complicated. I strongly recommend checking out Frank's books for a lot more details.

VSP3111: Nexus 1000v Architecture, Deployment, Management

2011-09-01T14:36:00.000-07:00

This session focused on the Cisco distributed virtual switch, the Nexus 1000v. The speaker was very knowledgeable and a great presenter. Lots of great details, but as fast as he was going I didn't get all of the details. You can check out the his blog at jasonnash.com.

Highlights:

The VSM is a virtual supervisor module, which acts as the brains of the switch just like a physical switch.
The VEM is a virtual ethernet module, which is in essence, a virtual line card that resides on each ESXi host.
VSM to VEM communications are critical and you have various deployment options

Layer 2 only: Uses two to three VLANs and is the default option, and the most commonly deployed architecture.
Layer 3: Utilizes UDP communications over port 4785, so it can be routed

When in layer 2 mode you need to configure the control, management and packet networks

Management: End point that you SSH into to manage the VSM and maintains contact to vCenter. Needs to be routable.
Control: VSM to VEM communications (This is where most problems occur.)
Packet: Used for CDP and ERSPAN traffic

Nexus 1000v deployment best practices

Locate each VSM on different datastores
You CAN run vCenter on a host that utilizes the N1K DVS
ALWAYS, ALWAYS run the very latest code. Latest as of Sept 1, 2011 is 1.4a, which does work with vSphere 5.0.
Don't clone or snapshot the VSM, but DO use regular Cisco config backup commands
Always, always deploy VSMs in pairs (no extra licensing cost, so you are dumb not to do it).

Port profile types

Ethernet profile: Used for physical NICs and are used as uplinks out of the server. These use uplink profiles.
vEthernet profile: Exposed as port groups in vCenter and is the most common type of administrative change made in the VSM.

Uplink teaming

N1Kv supports LACP, but the physical switch must support it as well.
vPC-HM - Requires hardware support from the switch and more complex to troubleshoot
vPC-HM w/ MAC pinning - Most common configuration and easy to setup/troubleshoot.

On Cisco switches enable BDPU filter and BDPU guard on physical switch ports that connect to N1K uplinks.
Configure VSM management, control, packet, Fault Tolerance, vMotion as "system" VLANs in the N1K so they are available at ESXi host boot time and don't wait on the VSM to come up.
For excellent troubleshooting information check out Cisco DOC 26204.
You can also check out the N1KV v1.4a troubleshooting guide here.
The network team may prefer to use the Nexus 1010, which is a hardware appliance that runs the VSMs. This removes the VSM from the ESXi hosts, and could be better for availability, plus the network guys can use a serial cable into the 1010. You would deploy 1010s in pairs, and they have bundles that really bring down the price.
You can deploy multiple VSMs on the same VLANs, but just be sure to assign each VSM pair a different "DOMAIN" ID.

Not mentioned in this session are additional Cisco products that layer on top of the 1000v, such as the forthcoming Virtual ASA (firewall), a virtual NAM, and the virtual secure gateway. The ASA is used for edge protection while the VSG would be used for internal VM protection.

Did you know about Cisco UCS Express?

2011-08-31T21:27:00.000-07:00

Today while I was walking around the vendor expo at VMworld 2011, I saw a very interesting product from Cisco. I was familiar with the datacenter UCS product, but a mini version caught my attention. Called UCS Express this is a micro ESXi server that slides directly into their ISR G2 chassis, which is a branch office router.

This is a micro server that currently supports dual cores and upto 8GB of RAM with two 1TB HDs. With RAID 1 you get about 500GB of usable space. It will run ESXi, so you could put very lightweight services like AD/DNS, DHCP or print server out at your branch office without deploying a full rack mount server. List price is around $4K for just the mini server, which isn't bad. On the road map is a double wide server which will support more cores and up to 48GB of RAM. That should be coming in 2012.

They will also be working on a centralized management console, so if you have a lot of these micro servers on your network you have a single pane of glass to manage them through.

If your business has remote offices with limited space, and you only need very minimal Windows services, then this could be a great option for you. I don't think this product gets much press, as I had never seen it before.

BCO1946: Making vCenter highly available

2011-08-31T21:02:00.000-07:00

vCenter is a business critical service that when it goes down can cause substantial chaos, although VMs will happily keep running while it is down. Using VDI? Forget spinning up new VMs, or rebooting VMs. Using vCD? Forget doing anything while it's down. HA? Yes that will keep working (for one failure in 4.x, and indefinitely in 5.0). So this session focused on various means to make vCenter highly available since it has no built-in means to be HA, so a little help is needed.

Linked mode does NOTHING for HA. A little data is replicated between the various instances, but if one of the vCenters goes offline you can't manage the hosts it was servicing.
You really need to establish RTO and RPOs for vCenter so you know what to design for.
Other infrastructure like AD, DNS and SQL are critical and must be available. Also remember that network connectivity must be maintained.
The main options the speaker offered up as HA solutions are:

Traditional backup and restore: Does your backup solution need vCenter to do restores? This is a manual recovery process and doesn't help with planned downtime like OS patching. You need a DR plan in place. See VMware KB 1023985 for some tips.
Cold Standby: Easy if SQL DB is local, RTO can be shorter, but a manual recovery process. Harder to do if physical.
Windows Clustering: Not supported by VMware as vCenter is not cluster aware.
VMware HA and APIs: Neverfail and Symantec offer clustering/HA products. These are incomplete as their process monitoring is very basic and does not cover all scenarios. Better than nothing, but far from complete. Fairly automated, compliments HA, and fairly easy.
VMware vCenter heartbeat:

Not the cheapest solution, but it is the most comprehensive
Active/passive configuration, share nothing model
Protects against OS, HW, application and network failures
Can be triggered on performance degradation of vCenter
Protects against planned and unplanned downtime
Protects vCenter plug-ins, SQL databases (even if on separate server) and VUM
Works across the LAN or WAN
Limited to a 1:1 topology

The bottom line is if you want an automated and comprehensive vCenter protection mechanism you are really left with one option, vCenter Heartbeat. I did a quick evaluation of it a couple of years ago before it supported 64-bit operating systems and the GUI/installation had a lot of room for improvement. I haven't tried newer releases, so I hope it feels more like an integrated product than the Neverfail engine bolted on to some VMware customizations.

VSP1999: Advanced esxtop usage

2011-08-31T20:27:00.000-07:00

This session was quite advanced and had a lot of troubleshooting examples which are hard to adequately capture without the slides, so I'll just touch on some of the counters he used during some troubleshooting examples. Maybe in future posts I'll focus on one subject like storage stats and recreate a couple of the presented examples since I thought he did a very good job of showing you want to look for when problems rear their ugly heads.

There are a variety of management tools for ESXi

vCenter Alarms
vCenter Operations
vCenter Charts
esxtop - Live stats
esxplot (free utility)

Interpreting esxtop stats manual here
New counters in ESXi 5.0

CPU screen now shows the number of VMs and total vCPUs
%VMWAIT stat
Power: CPU pStates in Mhz (BIOS must be in OS controlled power mode)
Failed disk I/Os
VAAI block delete
Low latency swap - LLSWR, LLSWW (broke in 5.0, look at stats in vCenter GUI)
NHN - Wide NUMA indication

CPU counters are misunderstood

RDY - VM wants to run but can't due to scheduling issues (bad)
CSTP - Co-stopped state. Co-scheduling overhead due to multiple vCPUs
Run - VM is using processor time
(Note: At this point the speaker went into great depth about CPU utilization and even more states, so the stats above just scratch the surface.)

Storage stats

DAVG - Most important disk stat to monitor
QAVG - Should be nearly, if not, zero all the time
DQLEN - Driver queue length

So there you go..some very low-level counters that you can look at to start troubleshooting performance problems. If you use the vMA you can use resxtop to monitor real-time stats from an ESXi host, so you don't need to SSH in and do it locally. Better for security, and easier to grab stats from multiple ESXi hosts at once. The esxplot utility is great for analyzing a lot of captured data and easily graph/search it for what you want.

VSP3864: Best practices for virtualized networking

2011-08-31T15:24:00.000-07:00

This session was a bit more high level and basic than I had hoped for, but here are the highlights:

Virtual Port ID load balancing is the default option and the least complicated option.
IP hasing is more advanced and requires Etherchannel to be configured on the switch
VTS (virtual switch tagging) is the most common vSwitch configuration
Private VLANs provide for L2 isolation. Really good for DMZs.
If you use IP hashing on the Cisco switch side you must configure Etherchannel for IP-SRC-DST, which is a global policy on the switch. The default mode on older IOS versions was MAC hashing which is not compatible.
If you use beacon probing (not recommended) it really needs three or more NICs to work properly.
Enable portfast on and use BPDUGuard to ensure STP boundaries
The VMware dVS has smarter load balancing
General tips:

How to change the VM MAC: KB 1008473
Using MS NLB Multicast? KB 1006525
Enabling CDP KB 1007069
Beacon probing and IP hashing do not mix KB 1017612 and 1012819
Check drivers and firmware against the HCL (very important)
Use VLAN 4095 on the switches for promiscuous mode
In ESXi you can use tcpdump-uw for packet captures. KB 1031186

Nothing earth shattering, but a few good tidbits of information.

CIM1264: Private VMware vCloud Architecture Deep Dive

2011-08-31T13:50:00.000-07:00

This was a pretty advanced session about the vCloud Director, which is a complex product. The speakers were very, very good, but given the advanced nature of the discussion it will be hard to recap the session in full and some of the concepts need more explanation than I can provide here. But that being said, here are some of the highlights:

Value of chargeback for an organization: accurate TCO/ROI analysis, accountability
vCloud architecture

Horizontal scaling
Multi-tenancy
Limit single points of failure in the architecture
Leverages load balancers
You must make the database highly available or the whole vCD management subsystem goes offline, although existing VMs will continue to run.
A vCD architecture is pretty complex and hard to wrap your head around

Typically you setup a dedicated management cluster that runs the vCD infrastructure like vCD, AD/DNS, vCenter, SQL, etc.
Resource groups are compute resources
A virtual datacenter is typically divided into a provider vDC which has a single type of compute and storage resource (single tier of storage).
An organization vDC is an allocation from the PvDC
vCD has various allocation models which cannot be changed once you instantiate it

Pay as you go - Dynamic, unpredictable
Allocation Pool - % of resources, can burst, but pretty predictable. Most common type.
Reservation pool - Hard caps, not dynamic, cannot burst

Networking has three layers

External - Internet access, IP storage, backup servers, etc.
Organization - Allows vApps to communicate with each other
vApp - Private network for communications within the vApp

You can define network pools of various types

Portgroup backed - Manually create with vCenter
VLAN backed - Uses the vDS and you give it a range of VLANs to use. v1.5 supports the N1K
vCloud Network isolation (VCD-NI) - Creates networks on the fly and uses MAC on MAC encapsulation. Need to increase your MTU to 1524. VMware's secret sauce for multi-tenant isolation.

New features in vCD 1.5 include:

Microsoft SQL server (no more Oracle requirements!!)
vSphere 5.0 support
Custom guest properties
Much faster VM provisioning

The take away from this session is that vCD is very powerful, but also very complex. Today most use cases are test/dev and NOT production. The speakers said possibly next year they will see more production usage. vCD is the replacement for lab manager, which was discontinued last year.

VSP2884: vSphere 5.0 Performance enhancements

2011-08-30T22:02:00.000-07:00

This session covered some of the major performance enhancements in vSphere 5.0. The presenter flew through the slides at 100 MPH and didn't spend much time on the bullets so I wasn't able to capture all of the highlights. But here's what I did capture:

32-way vCPUs with 92-97% of native performance
CPU scheduler improvements for up to 30% performance increase
vNUMA for NUMA aware applications (mostly for HPC). Turned off for < 8way VMs, turned on for 8-way or greater VMs.
vCenter can now process double the number of concurrent operations
9x faster HA reconfiguration time
60% more VMs failover in the same time period with the new HA engine
NetIOC - True QoS tagging at the MAC layer and user defined network pools
Splitrxmode can reduce packet loss dramatically under specific circumstances (30K packets per second, more than 24 VMs on a host)
TCP/IP optimizations that boost iSCSI performance
Netflow5 support in the dVS
Multi-NIC vMotion enablement
Storage migration with write mirroring
Host cache - SSDs for swap. Memory hierarchy is: Transparent page sharing, ballooning, compression, then host cache.

30% performance improvement over spinning disk swap

Storage is the root cause for most virtualization performance problems. !!!
(Note, presenter covered many new storage enhancements that I wrote about in previous blogs so I stopped taking notes.)
Software FCoE initiator has nearly the same performance as traditional FC HBAs
An example vMotion improvement for a 28GB Exchange 2010 VM was from 71 seconds on 4.1 to 47 seconds on 5.0 using 10GbE.
VDI workload denisty has also been increased more than 25%

There was a whole bunch of other tidbits that I just couldn't write down fast enough, but the list above is a good start. vSphere 5.0 has over 200 new features, so clear this list is far from complete.

BCO1269: SRM 5.0 What's new and Recommendations

2011-08-30T21:45:00.000-07:00

Unlike the last session I attended, this one was very good and had quite a bit of great technical material that went beyond common sense most IT guys have. SRM 5.0 is a major release that addressed many of the shortcomings of the 4.x releases. Highlights include:

Customers need simple, reliable DR
vSphere Replication is storage array independent and happens at the ESXi host level
RPO is 15 minutes to 24 hours. You cannot do less than 15 minutes, so this is not suited for applications that must use synchronous replication and have zero data loss like the finance sector.
You can take snapshots of the source VM, but they are collapsed at the recovery site
Recommend that you don't use snapshots
The replication engine dynamically changes block sizes and speed to ensure the RPO is met. If your RPO is 1 hour it doesn't wait for 59 minutes then try to burst the data in 60 seconds.
Replication is a property of the VM
Array based replication can have much shorter RPOs, compression, and WAN optimizer friendly. vSphere replication has none of these features.
v1.0 limitations include:

ISOs and floppies are not replicated
Powered off or suspended VMs are not replicated
Non-critical files like swap, stats and dumps are not replicated
pRDMs, FT, linked clones and VM templates are not replicated
Requires VM HW version 7 or 8
A vCenter server is required at both sites

Scalability enhancements for SRM include

Protection of up to 1000 VMs
10 parallel recovery plans
150 protection groups
500 VMs per protection group
These limits are not enforced, just the most VMware has tested and approved

Planned migration is a new workflow

Used when a controlled migration can be used instead of the smoking hole scenario
Will stop if any errors are encountered
Shuts down the VMs gracefully
Very orderly failover process
Application consistent recovery

Failback is a new workflow

Failback in 4.x was in the nicest sense extremely scary and very manual with high risk
Replays existing recovery plan in reverse
"Reprotect" feature in the GUI
Only supported by SRA/LUN-level replication
Failback only supports original VMs, not any new VMs stood up after the original failover

Brand new GUI

Both sites visible in a single pane of glass
Still strongly recommend that customers use Linked Mode
Able to set IPs for both sites in the GUI
IPv6 support
No more sysprep or customization specs needed
Huge Re-IP performance increase (huge!)
Supports in-guest callouts and scripts for custom app control
Extended the APIs for better integration

Dependencies

Increased to 5 priority groups
All actions are parallel within a group unless otherwise specified
Able to now craft more elaborate and controlled dependencies but in a far easier manner
Tip: Don't get TOO creative or it will extend your recovery time and may miss your RTO

For customers that have previously used or looked at SRM before and didn't bite, it's time to look at it again. The failback support with array based replication is huge! You may still find you need a different product, like InMage, but SRM of yesterday is not the SRM of tomorrow. SRM 5.0 is due out "soon" according to VMware. I did a hands-on lab of SRM, and I will stay it was quite slick and I came away very impressed with the changes.

SPO3990: Best Practices for Storage Tiering and Replication

2011-08-30T21:27:00.000-07:00

This session was a bit higher level and more common sense tips, so I split after 30 minutes and went to another session. The speakers had 8 best practice tips, and both of them were from Dell/Compellent. So they had a few minutes sales pitch before they started the meat of the session. Highlights included:

Select an array built for virtualization

Dynamic storage with wide striping
Ability to change RAID levels on the fly, extend LUNs
Rely on metadata to intelligently manage storage

Let tiered storage do the heavy lifting

Sub-LUN data tiering
vSphere 5.0 storage DRS is not sub-LUN tiering aware so it may not behave as expected when measuring latency

Use thin provisioning on the array and with VMDKs

I would actually disagree here and say use array-based thin provisioning and EZT VMDKs if your array supports VAAI, zero detect and thin provisioning.

Leverage storage snapshots

Protect your data
Deploy new VMs (who would really do this?) by cloning a LUN then running sysprep on the cloned VMs. Really?!?

After those four I lost interest and ran over to another session....SRM 5.0 What's New.

SUP1010: Cisco and VMware Innovating Together

2011-08-30T15:58:00.000-07:00

This session was a 'super session' which is basically a vendor touting their wares and how well they integrate with VMware. To that end Cisco went through a number of announcements and innovations that are really industry leading. High points of this session were:

Policy based management is key to deploying clouds. Policies can include security, storage, network, and compute resources.
Clouds must be multi-tenant per-server, elastic, and automatic.
Network, compute and storage have now finally converged
Key tenants to clouds are: Open management, mixed vendor environments, industry standards and partner solutions.
Datacenter designs must have a 10-15 year design life
Policy driven management is fundamental and Cisco has implemented this via service profiles
2011 Cisco achievements: Virtual ASA (firewall), #1 in VMmark 2.1 performance, enhanced VMdirectPath for VM mobility
Cisco UCS has a 21% performance advantage over same-core configurations from other vendors (tested in 4 node, 4 socket config)
New virtual interface card has dual 40Gbps ports and supports upto 256 PCIe interfaces for high-density VDI, multi-tenancy where you can link virtual ASA policies to a service profile.
vCloud Director now has integration with UCS for automated provisioning and configuration
VXLAN is a game changer for the networking industry. The Nexus 1000v will support VXLAN in beta in September 2011, and vCloud director will support VXLAN as well.
VXLAN supports up to 16 million interfaces, up from the 4096 VLANs.
VXLAN is the next generation "VLAN" concept and enables VM mobility across the cloud regardless of physical location.
VLANs are end of life!
VXLAN has been submitted to the IETF as a standard
Virtual ASA firwall appliance was announced yesterday

For multi-tenant datacenters
Uses the Nexus 1000v vPath technology
Same features as the physical ASAs

VM live migration across datacenters

Maintain security policies during and post-migration
Workload mobility

In 2011 Cisco and VMware

Integrated the Nexus 1000v with vCD
Enhanced UCS autodeploy with service profile integration
Overdrive network API
Integrate vCD and vShield manager with OverDrive
vShield Edge and N1K beta in Sept 2011

Future integration: OverDrive for network management, virtual ASA for security, N1K: a complete stack
44 vendors have written products for the UCS XML integration API
Cisco UCS is now the #2 US blade manufacturer, after just 2 years in the market
Cisco is working on a virtual WAAS
ASA will provide tenant-level security down to the VM

In short, I think Cisco is leading the way with unified computing and the other major players (HP, IBM, Dell, etc.) have a lot of catching up to do. No solution stack is perfect, but looking at the currently shipping products and their integration roadmaps, I think Cisco "gets it," It will be interesting to see how the other vendors respond since they are arguably lagging in both vision and shipping products.

VSP1700: vSphere 5.0 Storage Features

2011-08-30T14:22:00.000-07:00

This session covered many of the new and cool storage features in vSphere 5.0. The speaker mentioned that if you think vSphere 4.0 was 'for networks' then vSphere 5.0 is 'for storage.' Many of the cool tools such as DRS that you are used to for compute/memory resources are now extended to storage. Highlights include:

VMFS5 has many great enhancements:

64TB LUN support without any extents
130,689 small files per datastore
64TB pRDM
Consistent 1MB block size. No more trying to choose the right block size.
VAAI ATS is extended to all metadata locks, not just some operations
Sub-blocks are reduced to 8KB, down from 64KB in VMFS3. Up to 30K sub-blocks.
Small file support for files up to 1KB in size. These small files are actually stored in the VMFS metadata, then migrated to sub-blocks between 1KB and 8KB, then migrated to a full block between 8KB and 1MB.
LUNs 2TB and larger use GPT
VMFS3 LUNs that are migrated to VMFS5 and extended to >2TB will be seamlessly converted to GPT
Best practice for upgrading to VMFS5 is to create a new datastore and migrate VMs. Upgraded datastores won't have all the features of a native VMFS5 volume.
VAAI has been extended to NAS and supports full file clone, reserve space, and fast file clone for linked clones. Clones will also be block aligned for better performance.
VAAI also now supports dead space reclamation and out of space conditions (VM stun). Arrays can now de-allocate blocks when a VM is moved to another datastore.
VASA allows array vendors to expose phyical LUN characteristics to vCenter, such as RAID level, disk type, etc. Array vendors must create a provider to enable VASA reporting.
Profile driven storage lets you tag a datastore with characteristics (e.g. Gold, Silver, bronze) so you can place VMs on the correct storage without consulting complex spreadsheets or weird datastore names. Can populate tags manually or leverage VASA.
Storage DRS measures free disk space and latency to determine if a VM should be migrated. This also automates datastore selection and uses advanced load balancing algorithms. Latency is measured over 24 hours and every 8 hours it is evaluated. If over 15ms for 90% of the time, then sDRS will take action.
Datastore clusters let you group together datastores of the same type, which then ties into sDRS.
Datastore clusters have placement rules- Intra-VM VMDK affinity, VMDK anti-affinity, VM anti-affinity rules.
Datastores can now be placed in maintenance mode and sDRS will evacuate the LUN to other LUNs in the cluster using smart placement rules.
storage vMotion now works with VMs that have snapshots and linked clones.
sVM uses a fast migration technique that mirrors writes to source and destination VMDKs so that only one copy pass is required.
vSphere 5.0 supports software FCoE initiator, like iSCSI, but does have hardware dependencies.

Whew...that's just a high level overview of the new storage features. There are other sessions that go into much more depth for many of these features. Needless to say storage has gotten a lot of attention in vSphere 5.0. Be sure to prod your storage array vendor for vSphere 5.0 VAAI and VASA support.

BCA1902: Virtualizing Exchange 2010

2011-08-30T12:57:00.000-07:00

This session focused on virtualizing Exchange 2010 on vSphere 4.x or 5.x. Highlights of this session include:

Bottom line there's no reason why you should be afraid of virtualizing Exchange 2010.
Exchange 2010 now uses 32Kb I/O blocks, vs. 8K for previous versions.
Exchange 2010 I/O is much more optimized than previous versions
VMware fully supports share-nothing clustering with nearly all ESXi features (HA, DRS, etc.). Share-nothing clustering is used by Exchange 2010 and SQL database mirroring.
You can virtualize all Exchange 2010 roles
You can combine DAGs with HA, vMotion, DRS, Fibre Channel, FCoE and iSCSI
VMDKs must be thick provisioned, not thin. Should use EZT VMDKs.
Not supported is NFS for Exchange data or VM snapshots (for roll-back purposes, backup is fine).
VMware internally uses standard load performance tools like jetstress and loadgen (more so loadgen)
On vSphere 5.0 you can average 1000 users per vCPU/pCore, and linearly increases to 12K users
Fibre Channel has the best performance, but iSCSI is fine too
2-7% CPU overhead vice physical hardware
No I/O latency impact on virutalization
Best practices include

vCPUs <= pCores
Exchange is not NUMA aware so keep VM size less than NUMA node size
Use the Exchange processor query tool to determine users per core estimates
Use the Exchange mailbox role calculator for storage/network calculations
DO NOT over commit resources such as memory or vCPUs
Use LSI logic SCSI adaptor unless you've already standardized on the pvscsi driver
Use multiple vSCSI adaptors and distribute the mailbox/log load across them
Only use RDMs if your hardware storage array requires them to do VSS snapshots, otherwise VMDKs are perfectly fine.
For DRS keep the VMs smaller and ensure EVC mode is enabled
Enable HA for all VMs, use host admission control, and enable VM monitoring
Utilize host DRS groups and VM DRS groups

DAGs must be on separate nodes
"should run on" for all other roles

For vMotion set the clusterheartbeat setting to 2000ms, up from 1000ms default if you aren't using jumbo frames
Use the Exchange profile analyzer tool
Design Questions: Dedicated or multi-role VMs? How much HA? DB size? Backups?
Deploy dedicated mailbox VMs, but can easily use other combos like HUB/CAS.
Processor selection has a major impact on users per core. (.e.g. Intel x5470 vs. x5660 shows a dramatic reduction in CPU utilization).
How big should your page file be? Check KB 889654 to reduce page size
Monitor % CPU RDY, KAVG, DAVG and GAVG ESXi counters

Whew...the speaker covered a lot of ground. In a nutshell Exchange 2010 runs extremely well on ESXi, but you do need to be aware of the tweaks/best practices when deploying it on vSphere. Professional services organizations that specialize in Exchange engagements really need to understand the various hypervisors and best practices, or you could run into some issues or cause customers problems.

VSP2247: 10Gb and FCoE Real World Design

2011-08-30T11:20:00.000-07:00

This was an excellent fire hose of a session..far too much data to try and write down. Highlights of this session include:

Ten 1Gb links does not equal a single 10Gb link. With bursting you can more effectively us a single 10Gb link.
With vSphere 5.0 vMotion saturate two 10Gbps links with 8 concurrent vMotions
vSphere 5.0 supports multi-NIC vMotion but this requires multiple vmkernel ports, not just bonding two 10Gb NICS in a vSwitch.
In vSphere 5 NetIOC can use real QoS tags
vSphere 5.0 supports LLDP which helps you map vNICS with HP FlexFabric ports
Speaker's firm did a lot of analysis of 15,000 workloads and on average a single VM only averages 5Mbps of traffic, including IP storage. Of course some traffic will burst, but VM network usage is on average very low.
Manufacturers use two primary means to manage bandwidth: Throttle (HP FlexFabric) and Traffic Priority (Cisco UCS and N1K). Traffic priority is more elastic and flexible.
Use tools like NetPerf to do network load testing
Enabling link failure detection is very important. HP uses SmartLink and Cisco uses Link state tracking.
Major design recommendations

Always check the HCL. Extremely important as not all HW or drivers are on the HCL. A must!
Use the vmxnet3 NIC
Use Twinax cables, but there are some interop issues (Cisco and HP) that you need to watch out for.
Jumbo frames should only be used for special use cases, as it makes traffic shaping harder. Jumbo frames can increase vMotion traffic from 15Gbps to 20Gbps, but do you really need to?
Turn spanning tree port fast on
Leverage VLAN tagging within vSphere
Configure Multi-NIC vMotion in vSphere 5.0
NEVER trust written documentation. There is a lot of wrong information out there. Do you own testing and analysis.

HP FlexFabric recommendations

Use LACP etherchannel on northbound switches
If using the N1Kv, leverage HP FlexNIC throttling, not N1K traffic prioritization as the FlexFabric hardware DOES NOT honor QoS tags. Shame on HP!
Management traffic: 500 Kbps, VMs - 2.5Gbps, vMotion - 3Gbps, Storage 4Gbs
FlexNICs only do outbound trafic throttling..not inbound. Shame on HP.
In FlexFabric make sure to enable the virtualization of the UUID (serial #)
Install Insight Control for vCenter for good network and storage discovery

Cisco UCS

Configure HW QoS on the NIC adapters
Fully configure QoS (end to end)
Enable CDP
In v1.4 template management is greatly improved and works very well (single template for all ESXi servers)
If using jumbo frames set a packet size of 9216, not 9000, to allow for VLAN tag data

Nexus 1000v

Use multiple VSMs, or better yet, the Nexus 1010. Some package deals really bring down the N1010 price.
Use MAC pinning
Be sure to configure system VLANs for critical VLANs that need to be present when ESXi boots

Whew! The speaker talked 100MPH, and also said his slide deck has a lot of backup slides that go through detailed HP FlexFabric and Cisco UCS configuration.

VMworld 2011 Day Two Keynote

2011-08-30T09:47:00.000-07:00

This morning I attended the keynote session by Dr. Stephen Herrod, the VMware CTO. Key points during his presentation were:

It (hypervisors) should just work and work well.
IT industry is moving away from servers to services and toward people, not devices.
DUH: Devices, Universal access, High expectations
We are moving towards the connected enterprise (PC, mobile, tablets, etc.)
Simplify the existing datacenter, extract data from silos, assign policies to the data
Solutions to simplify include: VDI (View), ThinApp Catalog, Data services (Project Octopus)
Solutions to manage: Project Horizon (universal broker)
Solutions to connect: View 5, Horizon Mobile (phone hypervisor), Project Octopus, AppBlast
ThinApp Factory - Supports ThinApp, XenApp, RDS and I saw mentioned elsewhere here at the conference App-V support.
ThinApp factory can automate the software package creation/patching process. Watch out App-v!
Horizon App portal - VMware's version of the App store
Project Octopus - Dropbox for the enterprise. PC, mobile phones, tablets, etc. with policy controls.
AppBlast - Transforms thick applications such as Office into HTML5 with no browser plug-in. Think XenApp, but different under the covers.
View 5 fully supports Aero Glass and Unified communications
VMware bought SocialCast, which is similar to MS Lync....in house collaboration, chat, video.
vSphere 5.0 has over 200 new features, was delivered on time and feature complete (a first)
VMware Go is for SMBs to help quickly migrate servers to ESXi and manage from a browser
Major storage enhancements in vSphere 5.0 - Pooling, automated placement, DRS
VXLAN - New spec submitted to IETF to encapsulate L2 packets into L3 packets for L2 mobility across the WAN. Cisco and other partners are supporting the proposed spec. One of the final pieces for full network virtualization. Can enable scenarios such as DR that don't require re-IPing VMs when they move.
A VM's network identity should not be tied to its physical location.
vShield App will support DLP

Cisco leaks details of virtual ASA Firewall appliance

2011-08-29T22:01:00.000-07:00

Hot off the presses is a Cisco 'announcement' of a virtual ASA product that is in the works, although no details were released about pricing and availability. It will leverage the capabilities of the Nexus 1000v DVS, and is more for edge protection (North-South traffic), vice internal traffic (East-West) which their VSG product is better suited for. You can see the full blog post from Cisco here.

HP 3PAR announces next generation array...P10000

2011-08-29T21:53:00.000-07:00

Last week HP announced their new 3PAR P10000 array, which is the next generation virtualization array. It features a number of enhancements over their previous generation, released when 3PAR was still independent. Today at VMware vmworld HP had an unveiling of the P10000 and former 3PAR CEO David Scott was present. David Scott is now in charge of the HP StorageWorks division, if you didn't know.

Some of the enhancements of the v-Class include:

New 4th Generation 3PAR ASIC which performs much of the controller magic. Each controller node has two ASICs, which triple the bandwidth over the previous T-class controllers.
Support new host-facing connectivity to include FCoE, 10Gb iSCSI, and 8Gb FC. No more PCI-X slots!
HP Peer Motion to move LUNs non-disruptively between arrays.
Redesigned cabinet where all cabling now comes out of the rear of the cabinet for better cable management.
Greatly increased transactional and sequential throughput.
Future support for SAS connectivity.
Major new software version, 3.1.1, which will support the new vSphere 5.0 VAAI and VASA extensions.

I was lucky enough after the unveiling to meet David Scott and talk with him for a few minutes, as a happy customer of 3PAR customer. I look forward to the 3.1.1 release on our T400, which will play very nicely with vSphere 5.0 and provide even better storage support.

VMware General Session, Paul Martiz.

2011-08-29T21:37:00.000-07:00

This afternoon was a keynote general session by Paul Martiz, the CEO of VMware. Some of the highlights from this session are:

As of 2010 50% of all workloads are now virtualized.
20 million VMs run on vSphere. 68,000 VCPs.
In 2014 80% of all internet connected devices will NOT be Windows PCs.
VMware spent 1 million engineering hours and 2 million QA hours on vSphere 5.0. Paul thinks the hypervisor should be like server hardware...highly reliable, you turn it on and forget about it. It should just work.
Shipping shortly will be vCloud director 1.5, vShield Security 5.0, vCenter Operations 1.0, SRM 5.0 and vSphere 5.0 (shipping today).
Paul stated they want to get to a common schedule and tested suite for the above products. Starting with the 5.1 release in 2012, all products will be v5.1 and GA at the same time.
VMware as won some vCloud datacenter deals including Bluelock, CSC, Terremark, Verizon and Dell.
vFabric will expand to Gemfire and SQLfire.
View 5.0 has greatly increased high-latency and low-bandwidth performance, and supports real-time VoIP/Unified communications applications.
Project horizon will provision apps to people, not devices and will support multiple application delivery methods such as ThinApp and App-V.
Coming in the future will be Android phones that are virtualized so you can run two Android instances side-by-side (corporate and person), within the same physical phone.

Nothing earth shattering, but it was very good to hear about the common release schedule and versioning starting with 5.1. VMware is on a yearly cadence of releases, and that seems to be continuing. Much like the Intel 'tick tock', where there is a major release followed by a minor point release.

BCO3420: Avoiding the 16 biggest HA and DRS mistakes

2011-08-29T21:24:00.000-07:00

Wow this session was a riot. The speaker (Greg Shields) could easily double as a stand-up comedian, much like Mark Minasi. Beyond the entertainment, the session was quite technical and had great content. I had a hard time writing everything down he was going so fast. Before he got into the top 16 mistakes he noted that with today's modern servers it is very rare that a server will go belly up due to a hardware failure. Greg also said that bad HA/DRS implementation will also impact vMotion in a negative way.

Drum roll please....the top 16 HA/DRS mistakes are:

1. Not planning for hardware change. The solution is to enable EVC mode on clusters, and try to use hardware that has very similar processors. Do not mix Intel and AMD processors. And even within manufactures, newer processors will support additional instructions not available on older models. Pay attention to the CPUs you buy.

2. Not planning for svMotion (storage vMotion). Snapshots are evil, and you should NOT use them except in rare situations. Make sure your VMDKs are in presistent mode or use RDMs. The servers must see both the source and target datastores, and the cluster must have enough resources to briefly have two copies of the VM concurrently running.

3. Not enough cluster hosts. Plan for adequate cluster resources and build-in a reserve factor. Typically one full server's worth of resources. Solution is to use an admission control policy and set the host failures tolerate to "1".

4. Setting the host failures tolerate to "1". Not all VMs are tier-1 and deserve the same restart priority. Setting aside a full host can be wasteful. Use the percent of cluster resource option and configure a percentage that is less than a single host's contribution to the cluster based on the number of hosts. For example, in a four node cluster each server contributes 25% so set the percentage to something like 20 or 15%.

5. Not prioritizing VM restarts. If you use the suggestion in mistake #4, you must properly configure the VM restart priority since you won't be reserving cluster resources to restart ALL of your VMs. Set your normal VMs to low restart priority then elevate special VMs to medium or high as needed.

6. Disabling admission control. Bad idea! Never, ever, ever do this! Enable the do not power on if inadequate cluster resources are available option.

7. Not updating the % policy. As you add more hosts to a cluster you need to recalculate the host failures percent number, otherwise the system will get out of wack.

8. Buying dissimilar servers. The hosts failure cluster tolerates option bases its calculation on the biggest server in the cluster. If you have six servers with 96GB of RAM then add a server with 384GB of RAM to the cluster, it will really throw out of wack the calculations and you will be left with a lot of unused resources.

9. Host isolation response. This is a confusing subject for many and prior to v5.0, contained some bugs or behaved in a way that customers did not always expect. In many environments you can configure the response to shutdown the guest VMs, but on a per-VM basis change the settings if you have critical apps that you want to ensure don't go down on accident. In v5.0 the datastore heartbeat future is a welcomed change and is only used if the management network goes down. The servers in the cluster need one datastore in common.

10. Overdoing reservations, limits and affinities. Use shares over reservations and limit the use of affinities (or anti-affinities). These restrictions impact the DRS calculations and can impact performance. Use sparingly!

11. Doing memory limits at all. Dont' ever do this, ever, ever! Limit memory usage as close to the app as possible. For example, you can configure SQL to limit the amount of memory it will use within the guest VM.

12. Thinking you are smarter than DRS. No human can calculate all of the variables and come up with the right answer. Let the software do its job.

13. Not understanding the DRS rebalancing equation. Far too complex to repeat here, so do some googling for this one.

14. Being too liberal. Migrations take resources, be they network bandwidth or CPU time. Don't have DRS continually moving workloads between servers. Configure thresholds to do sensible migrations when resources really are out of balance. vMotion was cool 5 years ago, but no need to have DRS continually move workloads just to be cool.

15. Too many cluster hosts. Although the technical limit is 32 hosts per cluster, the sweet spot is 16-24 hosts. Any larger and the calculations DRS does every five minutes become very complex and consume more and more resources.

16. Creating big VMs. This has new meaning with the 5.0 vTax licensing scheme. Assign the right amount of memory and vCPUs to a VM. Don't be too liberal. Right size the VM, don't supersize them.

Besides being the most entertaining session, by far, of the day it also provided great practical information that any VMware administrator should heed.

SPO3962: Enabling business continunity with VMware Metro vMotion

2011-08-29T20:53:00.000-07:00

This session was pretty good and covered many use cases for long distance (metro) vMotion for DR/BC purposes. Highlights include:

Planning for BC/DR is imperative and if you don't and a disaster strikes, statistically you could be out of business in less than a year.
New technologies are enabling disaster avoidance, and fundamental to this is vMotion over long distances.
Brocade has WAN/storage technologies that enable synchronous replication with up to 10ms of latency, up from the native 7ms in VMware Metro vMotion. That amount of latency could allow ~200 miles of separation between the primary and target locations.
79% of natural disasters are from weather, and people have hours to days of notices.
70% to 90% of downtime is planned
Using global load balancing services (GSLB) is key to transparently moving workloads between datacenters.
One example of long-distance vMotion is to lower power costs by migrating VMs to lower cost power during off-peak or off-season times. Specifically, your Seattle datacenter could host most VMs during the spring when hydroelectric produced electricity is cheap, then migrate workloads to another datacenter during other times of the year, or even time of day.
You need synchronous high-speed replication for this to be effective. In the not too distance future companies may need 10Gbps WANs.
Within a datacenter legacy servers move data in a north/south pattern (through the access layer to the aggregation layer and to the core), but virtualization totally changes that and data moves east/west within a datacenter on flat L2 networks.
Datacenters of the future will leverage large L2 networks that may even span datacenters.
The Brocade ADX is a GSLB that integrates with vCenter to help automate the redirection of client traffice to different datacenters as the workloads move.
You need to assume mobility of applications within and between your datacenters, either in whole or sub-components.
Again, Layer-2 architecture is key for virtualization and is radically different from the datacenters of yesterday.

They key takeaway from this session is that server virtualization cannot happen in a vacuum, and the supporting networks must radically evolve to support all of the new technologies. Don't think you can rely on your legacy switching and routers to support a highly virtualized and agile infrastructure.

SPO3049: Advanced SAN Functionality for Data Protection and Disaster Recovery

2011-08-29T20:36:00.000-07:00

Contrary to the title of this session, I didn't think the content was too "advanced." The speaker covered typical SAN features such as snapshots and clones, and how you can leverage them for a virtual environment. Highlights include:

Snapshots are an on-line and point-in-time. It could be of a datastore (LUN-level) or VM level.
If you mount an array-based LUN snapshot you need to select the resignature option.
Clones are a point-in-time copy, usually a full copy, that is used for data mining, quality assurance, test/dev environments, or patch testing. Basically you could clone dozens or hundreds of VMs, bring them up in a fenced environment and perform testing on a copy of your production data but in a controlled environment. This gives you high assurance of what will happen in production.
Many arrays support thin clones which uses copy-on-write technology to save back-end storage space.
Sometimes it may make sense to separate the OS VMDK from the data VMDKs. Why? Maybe you have different protection measures for the OS vs. the application data. You could do this by using several VMFS volumes, RDM, or iSCSI/NFS within in the guest.

The meat of the session only took 30 minutes, and I thought was very high level and pretty common sense.

SEC2284: Securing Government Virtual Environments

2011-08-29T20:25:00.000-07:00

This session was a panel discussion with several industry experts and representatives. Panelists included Texiwill (Ed Haletky), the Catbird CTO, a VMware employee, and others. Topics of discussion included:

There is no ETA on a DISA STIG for vSphere 4.x or 5.0. DISA is stating that they are relying on vendors to do most of the work, such as hardening guides. So for now you are stuck with the 3.x STIGs which are ESX specific, 90% of which does not apply to ESXi.
The NIST 800-125 was released earlier this year and provides some high-level virtualization security guideance.
Horizon data center solution is the first FISMA compliant public cloud.
I-Assure is a professional services firm that has work with customers such as the Naval Surface Warfare (NSWC) and SPAWAR to rapidly deploy secure, template-driven, virtual datacenters and get them through the C&A process. They can provision entire datacenters in 1/4 of the time it would normally take due to their pre-built templates and established processes/procedures. They've even gotten Navy type accrediation for certain products. They have engineers that can STIG a Windows VM or ESXi host in less than 6 minutes using a combination of GPOs and custom scripts.
It is very important to put your ESXi management consoles on their separate VLAN and strictly limit access to only authorized devices.
Create a trustzone for administrators that access vCenter. vCenter holds the keys and must be tightly controlled.
Manage vCenter credentials more tightly than even domain admins or root accounts, since they truly have ALL the keys to the entire kingdom. Virtualization admins must be the most highly trusted people in your organization from both a technical and security perspective.
You should separate your storage admins from your virtualization administrators, so you lessen the chance of a virtualization admin going rogue and deleting all of your VMs, erasing backups, and destroying the LUNs, leaving you quite up the creek.
Cisco has a good whitepaper on virtualizing multi-tenancy networks you can read here.
VMware released updated information for their vCenter Configuration Manager that incorporates DISA STIG findings, which you can find here.
It was mentioned there is no one tool or set of tools that can be universally used to perform scans on a virtual environment for the C&A process. Each DAA is different in what they want to see, so you really need to work with your DAA to understand what body of knowledge they want then find the right tools to do the job.

BCO3334: Site Recovery Manager Futures

2011-08-29T19:38:00.000-07:00

This session covered the future direction for Site Recovery Manager, known as SRM. SRM 5.0 has some major enhancements, and will ship "soon." Among the current and future enhancements are:

Simple, reliable DR which replaces manual runbooks
Array integration with a wide range of vendors, currently supporting 28 arrays
SRM will be able to detect failures, initiate failover, then manage the failover process
In the future they are looking at single click business continuity
Enhancements include application mobility where you can establish DR service levels, RPO, RTO, and tiered storage constraints, then tie applications to the defined DR service levels. For example, tier-1 applications could have stringent RPO/RTO requirements, while a tier-2 application could be much more forgiving.
Multi-site DR is on the road map, so you could enable application mobility across several locations. For example, you could have VMs that "follow the sun" by migrating them to datacenters to take advantage of lower power costs during off hours. Historically SRM was mostly a 1:1 relationship.
SRM can be used as a datacenter migration tool, not just a tool for DR/BC
Features new to 5.0 include automated failback, vSphere host-based replication, planned migrations, a totally new GUI, and application consistency.
SRM can now be used as a disaster avoidance tool, letting you effortlessly migrage workloads from a datacenter in harm's way to one that can sustain operations. Think of weather problems where you have some notice about an impending issue and proactively relocate workloads before a disaster strikes.
5.0 also includes history reports, 5 priority groups, and allows inter-VM dependencies to be configured.
In 2012 a future release will support a cloud DR concept, policy based DR, vMotion anywhere, RTO reduction, support for desktop DR, and per-VM protection with SRAs
Cloud DR is for private/public clouds where a target could be a cloud DR site, or DR between cloud sites. It may also include a self-service DR portal, and single click DR. Basically DRaaS (DR as a service).
The speaker also mentioned VMDK encryption, but didn't go into any details.

For anyone that has used SRM 4.x, the 5.0 release is a major update and sorely needed. SRM can take what is a very complex process and provide a high-level of automation. Now that it supports host-based replication, SRM can be used by SMBs that do not have high-end arrays that do LUN-level replication.

CIM225: Automated Infrastructure using VMware vCenter Operations

2011-08-29T13:58:00.000-07:00

This session covered a product which is a result of an acquisition last year, and has since undergone some significant enhancements. vCenter Operations is a product that provides operations intelligence for your virtual infrastructure. Major points in this session:

The speaker stated that 90% of performance problems for virtualized applications get blamed on the virtualization team, even though typically that is not where the problem resides.
The infrastructure teams and operations teams can be at odds with each other, pointing fingers about the root cause of issues. Even with advanced and costly performance tools, often it is the user that first complains about a problem, not a tool.
Changes to applications and infrastructure changes are the number one cause of downtime. Layer 8 (humans), not hardware, is what causes most of your unplanned downtime.
vCenter ops addresses three major areas: Performance, capacity, configuration
It lets operators easily pinpoint root causes of issues, and provides detailed capacity planning (CapacityIQ is integrated).
Speaker stated that cross-Silo tools are typically lacking in the enterprise (network, storage, compute, applications), and thus correlations are hard to make which translates into it being very hard to find true root causes of problems.
Optimizing IT is very difficult, if not impossible, due to these silos and disparate tools.
VMware's approach is to use analytics, covered by several patents, that distill information from many sources into actionable information that is visually displayed in very eye pleasing graphics.
The visualizations that were shown looked very professional, used heat maps, performance scores, and other business intelligence-inspired layouts that really let you know what is going on at a glance. It's not just a simple up/down dashboard.
The product will have progressive integration with third party monitoring tools such as Microsoft Operations Manager, Tivilo, HP Openview, and many other tools.
It features self learning algorithms that don't require you to manually configure thresholds, and will alert on non-normal conditions and provide technical details.
It will ship in various editions, with enterprise being the most full featured SKU and required to interoperate with third-party tools to manage heterogeneous systems like Windows, Linux and storage.
The many dashboards provide advanced BI features such as Health, risk, efficiency, and performance stats. For example, it can show 'stressed' VM and what is causing the stress. Heat maps let you quickly spot trends.
You can define KPIs (key performance indicators), which show "bound by" conditions, that are your root causes.
It also has guided remediation which can help you determine what the root cause is, and suggest ways to remediate the problem, then do the remediation.
It can coordinate configuration changes to the environment then roll back changes if performance issues occur as a result of the change.
Smart alerts have a root cause pane to help the operator troubleshoot
CapacityIQ has been integrated into the product
The tool shows resource waste and ways to remediate it
There is also a built-in orchestration/automation engine

Overall I was very impressed with the tool, the analytics engine, and the dashboards. In fact, I think they are more informative that say Systems Center Operations Manager, which more "BI" type intelligence with score cards, risk analysis, and root cause details. It will be interesting to see what type integration it has with SCOM and other third-party tools. This is one tool that organization should certainly take a good look at, as I think it's pretty unique in its capabilities.

SEC1980: Department of Defense vShield Architecture

2011-08-29T09:23:00.000-07:00

Here I am again at VMware VMworld, and I'll try to blog about as many sessions as I can. Unfortunately the way VMware has structured the schedule it's not very conducive to a lot of blogging but I'll do the best I can. This session was a very high level (and quick) overview of the VMware vShield products and how they can be used to help secure your networks, based on DISA guidance. However, DISA has not blessed or endorsed these products but the information was good none the less.

In August 2010 DISA published the Network Infrastructure Technology overview which describes how to implement defense in depth for a physical environment. It uses common products like firewalls, VLANs, IDPS, etc. Good perimeter protection, but not a lot internally although there is some. However, managing all of these physical devices can be complicated, error prone, and are not typically designed with VMs in mind.
vShield Edge is a product that provides perimeter protection, designed for multi-tenant internal clouds.

Supports NAT, L2 firewall, DHCP, IPsec, web load balancing, and static routes
It also supports syslog logging

vShield App is designed to protect applications at the hypervisor level. Basically at the vNIC level it can do packet inspection, both inbound and outbound.

VMs are protected as they migrate between hosts - Policy follows VMs
Protects against ARP spoofing, and includes a layer 4 firewall

vShield EndPoint is a framework that third party partners can use to provide addition inspection such as AV scanning, DLP, IDS and other functionality. Trend Micro is one such partner. McAfee/HBSS and Symantec do not yet have supporting products.
These products can allow you to create an 'enclave in a box' where on a single host or multiple hosts you can granularly control network access. For example, you can define a resource pool for VDI where the users can NOT access the internet, or have privileged VMs that administrators use that COULD access internal management VLANs.
These products have not undergone any common criteria testing, but VMware is targeting a future point release at EAL4+, but there are some product enhancements needed to ensure it will meet all criteria.

As I mentioned this session was very high level and didn't really provide any examples beyond birds eye views what a system might look like. The speaker only used 30 of his 60 minutes, so I think he could have covered more content such as real examples to help solidify the concepts and how to implement them. These are not DISA "approved" or recommended products so it's up to your organization to work with your security team to implement a solution that can be accredited.

Automate VMware VMX Security Lockdowns

2011-08-14T17:22:00.000-07:00

When building vSphere VM templates best practices would recommend that a number of security lockdowns be incorporated into the template. There are a variety of sources for recommended lockdowns, such as the VMware vSphere 4.1 Hardening Guide. But what if you already have VMs in production that you need to lock down, or want a simple way to configure your VM template settings?

Using some PowerCLI examples I modified them and the result is the script below. The script is called with a single argument, which can be the name of a VM or a wildcard so you can do mass changes. As always, TEST, TEST, TEST! Before you lock down all the settings below, make sure you understand what they do and determine if you really want to disable the feature.

This script can be very handy for XenDesktop 5.0 deployments, as their MCS engine does not properly copy custom VMX settings from the template, so you are left with unsecured VMs. Use the wildcard feature to hit all of the VMs. Also note that many of the settings require the VM to be power cycled, not just rebooted, to read the new values.

Before you run the script you will of course need to use the connect-viserver command to establish a secure connection to vCenter or an ESX(i) host. After the connection is established you can then run the script and monitor the progress in the vCenter recent tasks pane.

# Configure client VM VMX security settings.
# Version 1.0, August 14, 2011
# Argument can be a single VM or a wildcard

$ExtraOptions = @{
"isolation.device.connectable.disable"="true";
"isolation.device.edit.disable"="true";
"isolation.tools.copy.disable"="true";
"isolation.tools.paste.disable"="true";
"isolation.tools.setGUIOptions.disable"="true";
"Isolation.tools.Setinfo.disable"="true";
"Isolation.tools.connectable.disable"="true";
"isolation.tools.diskShrink.disable"="true"
"isolation.tools.diskWiper.disable"="true";
"isolation.tools.hgfs.disable"="true";
"isolation.tools.commandDone.disable"="true";
"isolation.tools.getCreds.disable"="true";
"isolation.tools.guestCopyPasteVersionSet.disable"="true";
"isolation.tools.guestDnDVersionSet.disable"="true";
"isolation.tools.guestlibGuestInfo.disable"="true";
"isolation.tools.guestlibGetInfoDisable.disable"="true";
"isolation.tools.haltReboot.disable"="true";
"isolation.tools.haltRebootStatus.disable"="true";
"isolation.tools.hgfsServerSet.disable"="true";
"isolation.tools.imgCust.disable"="true";
"isolation.tools.memSchedFakeSampleStats.disable"="true";
"isolation.tools.runProgramDone.disable"="true";
"isolation.tools.StateLoggerControl.disable"="true";
"isolation.tools.unifiedLoop.disable"="true";
"isolation.tools.upgraderParameters.disable"="true";
"isolation.tools.vixMessages.disable"="true";
"isolation.tools.vmxCopyPasteVersionGet.disable"="true";
"isolation.tools.vmxDnDVersionGet.disable"="true";
"isolation.tools.setOption.disable"="true";
"isolation.tools.log.disable"="true";
"log.rotateSize"="100000";
"log.keepOld"="10";
"Tools.setinfo.sizelimit"="1048576";
"tools.synchronize.restore"="false";
"time.synchronize.resume.disk"="false";
"time.synchronize.continue"="false";
"time.synchronize.shrink"="false";
"time.synchronize.tools.startup"="false";
"vmci0.unrestricted"="false";
"guest.command.enable"="false";
"tools.guestlib.enableHostInfo"="false";
"isolation.tools.dnd.disable"="true";
"RemoteDisplay.maxConnections"="1";
"Guest.command.enabled"="false";
"devices.hotplug"="false";
"vmxnet.noOprom"="true"
}
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
Foreach ($Option in $ExtraOptions.GetEnumerator()) {
    $OptionValue = New-Object VMware.Vim.optionvalue
    $OptionValue.Key = $Option.Key
    $OptionValue.Value = $Option.Value
    $vmConfigSpec.extraconfig += $OptionValue
}

# Get all VMs per the argument

$VMs = get-VM $args[0] | get-view

foreach($vm in $vms){
    $vm.ReconfigVM($vmConfigSpec)
}

vSphere 5 VDI Licensing Redux

2011-08-03T22:00:00.000-07:00

Among the licensing kerfuffle surrounding vSphere 5.0, VDI users may have overlooked some interesting information that VMware posted about VDI and vSphere 5.0 today. Myself and other bloggers like Brian Madden have done some analysis on what vSphere 5.0 means for VDI, mostly for non-VMware products such as XenDesktop.

However, VMware's blog post from today contains some very interesting information that I had not seen before. But before we get to that, let me quickly recap the two primary options for VDI on vSphere 5.0. (For more details see my blog post here.)

First, you can buy/use regular vSphere licenses and work within the vRAM entitlement limits. Depending on the number and size of VDI VMs, this may or may not be the best deal. Second, and new to vSphere 5.0 is the vSphere Desktop license which is sold in packs of 100 VMs for $6500. This removes the vRAM entitlement limit, but imposes other limits such as not running server OS VMs on the same hosts as VDI VMs. But overall, this is a better ROI as the per-VM costs are generally lower.

Now here's the new information that I just learned about. According to a VMware FAQ in the blog today "Customers who purchased licenses for vSphere 4.x (or previous versions) prior to September 30, 2011 to host desktop virtualization, and hold current SnS agreements, may upgrade to vSphere 5.0 while retaining access to unlimited vRAM entitlement."

Whoa...stop the train! Did I read that right? You can "violate" the vRAM limitations if you purchase vSphere 4.x licenses before Sept 30, 2011 for VDI? Yes, but what's the catch? Well there are a couple, but they aren't unreasonable. VMware states you must use a separate vCenter instance that is dedicated to VDI in order to re-purpose your vSphere 4.0 licenses for VDI and remove the vRAM caps. VMware states this is required, not optional. You also can NOT run general purpose (non-VDI) server VMs on the same hosts, but you could run VDI broker/monitoring VMs on the same hosts.

Now all the bloggers that did elaborate calculations for VDI can toss much of that work to the wind, recommend people deploy a dedicated vCenter server to manage VDI-only hosts, and be done with it. Companies just getting into VDI probably should go the vSphere Desktop SKU route, but it's nice to know existing VDI customers aren't left in the cold. You could need to pony up for an additional vCenter license, depending on your existing topology.

With XenDesktop in mind, this also makes some sense. Why? Who knows when Citrix will officially support vSphere 5.0 for XenDesktop. I would guess it will be several months after vSphere 5.0 code hits the streets. So you can dedicate a vCenter 4.x instance to XenDesktop, then migrate your other production servers to vSphere 5.0 on your schedule without worrying about XenDesktop impacts.

Along with the new entitlement increases announced today, I think the grandfathering of VDI only hosts into a vRAM entitlement free environment is a great gesture on VMware's part. Thank you!

P.S. It's not entirely clear to me if one leverages the vSphere Desktop licenses whether that also requires a separate vCenter instance or not. I suspect it does, unless there's a way to tell vCenter a host is only for VDI usage.

VMware Changes vSphere 5.0 Licensing

2011-08-03T18:31:00.000-07:00

The story I broke last week about impending licensing changes to vSphere 5.0 turned out to be spot on. In fact CRN published a story about the impending licensing changes and referenced my blog. Today VMware made an official announcement of changes, which you can find here. CRN just put out a story as well about the new changes you can read here.

I've summarized the changes in the table below. As you can see, and what is a bit odd, is the $/GB cost of vSphere Enterprise edition. It's actually more expensive than Enterprise Plus per vRAM GB if you look at it through a vRAM lens.

In addition to the vRAM entitlement changes, as I reported last week, the maximum vRAM entitlement per VM is capped at 96GB, even if the VM is the maximum 1TB in size. In addition, as I also mentioned before, a 12-month vRAM average calculation will be used to determine the amount of licenses you need, so that short lived spikes won't cost you more money indefinitely.

What was not announced was any vRAM-only entitlement SKU, capping vRAM based on pRAM, cross-edition pooling of entitlements, or grandfathering of existing vSphere 4.x customers with additional licenses to cover existing capabilities. With VMware reporting record profits this year, some customers may still wonder why the need for such a complex licensing scheme that may still cost scale-up users more money to use existing hardware. It would almost make sense to kill the Enterprise SKU.

What I also found interesting is that on July 27th VMware announced licensing changes to their VSPP program, which is for cloud providers. You can read the full announcement here. Noteworthy is their movement AWAY from allocated vRAM (which is used for 'regular' vSphere 5.0 instances described above) to reserved vRAM with a 50% minimum floor of VM memory, with a cap of 24GB per VM. According to VMware "We now charge you for the physical memory reserved for the VM, allowing you to vary the memory oversubscription ratio according to the needs of the application and service level." This more closely aligns VM memory usage to pRAM, although is not directly tied to the amount of pRAM your systems have.

VMware goes on to state:

Memory oversubscription works because many applications don’t use all the memory allocated to them, and this is compounded by application deployment guidelines for off-the-shelf applications that tend to over-estimate required memory. In addition, with VMs on the same host running identical copies of the same Operating System and/or application mean many memory pages are duplicates. Under vSphere, those VMs can share just one set of those identical memory pages, effectively “deduplicating” memory.

vSphere makes it possible to reserve less physical RAM for a VM without affecting performance, and has five main patented techniques to maximize memory oversubscription. Of course, it is possible to starve a VM of memory too, so there is a 50% reserved memory minimum (or floor, computed as the reserved RAM divided by allocated RAM). We chose this minimum on the advice of our engineering team. If you try to reserve less than 50% of allocated memory you will still be charged for a 50% reservation.

VMware makes an excellent case for why they are now changing their model from allocated vRAM to reserved vRAM. If this model is so great for cloud providers, and VMware bills vSphere as a cloud operating system, why isn't this new vRAM model good enough for us enterprise users as well? VMware basically admits the allocated vRAM model was not optimal, yet now enterprise customers are forced to use it. It makes you wonder if the left hand is talking to the right hand inside VMware.

I am glad VMware listened to the significant uproar and very emotional customer input, and the revision will reduce price increases for customers. However, I think using their new VSPP model for vRAM tracking makes a lot more sense and would standardize their licensing scheme. I can't imagine VMware making a second revision to their v5.0 licensing model. Maybe in 5.x or 6.x they will unify the vRAM models. But then how many customers will have started a migration to other hypervisors?

P.S. There are also some VERY noteworthy clarifications for non-View VDI users and those upgrading from previous ESX releases to vSphere 5.0. Check them out here.

Impending VMware vSphere 5.0 license changes?

2011-07-28T20:13:00.000-07:00

Update: VMware made an official announcement on August 3, 2011 and I've covered it in detail here.

Since the announcement of vSphere 5.0, and the new licensing terms, much of the focus on the launch has sadly not been around the great new features but the changes in licensing terms. In fact, the many loyal customers in this thread on VMware forums are threatening to, or actually looking at alternatives, such as XenServer, Hyper-V, or KVM.

I have it on good authority that VMware is taking these complaints seriously and next week will announce some changes to address the situation.

The rumored changes may include:

1. Doubling the vRAM entitlements for Enterprise and Enterprise Plus editions to 64GB and 96GB respectively. For example, a dual socket Enterprise Plus server would add 192GB of vRAM to your pool.

2. Essentials and Essentials Plus vRAM entitlement increased from 24GB to 32GB.

3. Capping the vRAM amount that counts against your licensed pool to 96GB per VM, even if the VM is allocated more, such as 1TB. Drops the cost of a 1TB VM from $75K to $3.4K for Enterprise Plus pools.

4. Licensing high water marks will be captured on a yearly basis vice a monthly basis.

Customers that have an ELA (enterprise licensing agreement) may be able to negotiate better terms, and should certainly try to do so with their rep if they feel the new licensing scheme will cost them additional dollars. Of course the details may change before the announcement, so take this information with a pinch of salt until something official comes out, probably next week.

Will this make everyone happy? No, but it is probably a good compromise and shows VMware does take feedback seriously. When the official changes are announced, the community scripts that have been floating around to estimate the licensing impact will need to be updated.

vSphere 5.0 VDI Licensing Changes

2011-07-15T17:12:00.000-07:00

Among the various license changes in vSphere 5.0, the way you can license VDI (virtual desktop infrastructure) has also changed, or rather, now gives you more options. Specifically, if you are a XenDesktop or non-View customer, keep reading. If you use VMware View, then it's pretty much status quo from my understanding. You should note that View licenses are excluded from the vRAM entitlement issues.

I touched on the new VDI licensing option in a previous post here, but I think a dedicated post to VDI with more clarity is warranted. You can also check out that link for more general vSphere 5.0 licensing changes and the vRAM entitlement issues. The official VMware End User Computing web page with more details can be seen here.

vSphere 4.x:

Utilize standard vSphere per-socket license for a ESXi host and you could mix and match VDI and server workloads at you wished. You could use any ESXi edition that matched the feature set you were looking for.
Purchase a VMware View Enterprise license for $150/concurrent user. This eliminates the per-socket ESXi host license, but the VDI hosts can only run the client OS VMs or server VMs that directly support VDI, such as the brokers. No non-VDI VMs are allowed on the VDI hosts. Concurrent user means a user logged into the client VM via the broker, not concurrently powered on client VMs.

You still need a XenDesktop license in either case since you are only using the View license to legally operate your hypervisor. So VMware isn't cutting you any breaks, and could be viewed as unfairly taxing non-View customers if they wanted a concurrent user model. Likely not the route you would want to go, but that would depend on your usage model.

vSphere 5.0:

Utilize standard vSphere 5.0 per-socket/vRAM license for ESXi hosts and you can mix and match VDI and server workloads as you wish. You can use any ESXi edition that matches the feature set you are looking for. You are now subject to the vRAM entitlement limitations. In VDI where you likely have scale-up hosts with lots of memory, you may need to purchase more socket licenses. Do very careful calculations before you jump on the v5.0 bandwagon.

Purchase a VMware View Enterprise license for $150/concurrent user. This eliminates the per-socket ESXi host license, but the VDI hosts can only run the client OS VMs or server VMs that directly support VDI, such as the brokers. No non-VDI VMs are allowed on the VDI hosts. You are excluded from the vRAM entitlement limitations on the ESXi hosts that support VDI. This now may become more palatable given the unlimited vRAM feature.

(New) Buy vSphere 5.0 Desktop packs, which are sold in bundles of 100 VMs, for $6500 each, or $65 per VM. This entitles you to power on a desktop class OS on an ESXi host without any vRAM limitations. This is a per powered on client VM, regardless if anyone is using it or not. You are strictly precluded from running any server OS VMs on these hosts, including any related to VDI that are allowed under the concurrent license model.

(Future??) VMware may release a non-View concurrent user license that costs less than $150 and gives you the same advantage of the View concurrent model, without the rights to use View. No information on possible price, or release date, if this happens at all.

If you elect to use the concurrent model for either version, and don't use View, you need to manually keep track of concurrent user usage through the broker of your choice. Should the VMware license police come around you should have some documentation to show you are in compliance and not cheating. The licensing guy I talked to didn't know how the new per-VM reporting worked and if vCenter would refuse to power on a client VM that violated the licensing maximum, or just nag you.

Also note that the concurrent user license and the new vSphere Desktop license entitle you to the functionality of ESXi Enterprise Plus for the hypervisor. So that's a good deal, which lets you take advantage of all hypervisor features for your VDI.

Depending on your consolidation ratios, usage patterns, number of users, etc. you have a variety of licensing options. There's no one size fits all solution here, but for new vSphere 5.0 customers the Desktop SKU looks like the best solution except if you can live with ESXi standard edition. Also remember that you can't convert an existing vSphere 4.x license to a 5.0 vSphere for Desktop SKU, since it doesn't exist in 4.0. So if you have an existing VDI deployment on vSphere 4.x, you have some tough decisions to make given the new vRAM considerations for the per-socket license.

Brian Madden did a great article with a very detailed analysis of the new vSphere Desktop SKU and desktops with various memory configurations. You can check out his article here. Based on his spreadsheet I did the same calculations as my previous licensing blog (1800 users, 23 dual-core hosts)and got the same results, just in a prettier format:

What's clear from these numbers is that with vSphere 5.0 your minimum VDI cost jumped from $46K to $112K (245% increase) for 1800 users, running the exact same edition of ESXi. For $5K more you can utilize the enterprise plus edition of ESXi (via the desktop SKU) which has vastly more features, although not all are really needed for VDI.

On the flip side, if you were going to run VDI on enterprise plus with v4.x (maybe you wanted to use the Nexus 1000v) and had not yet bought those licenses, you can now run those cheaper on v5.0 IF you buy the new 5.0 desktop SKU. Unfortunately if you already have VDI deployed on v4.x, then you are left with an increased bill, assuming no excess capacity elsewhere in your environment to offset the usage.

vSphere 5.0 Licensing Estimation Scripts

2011-07-14T18:14:00.000-07:00

Wow, the changes VMware made to vSphere 5.0 have really stirred up a lot of passion on the subject from angry customers. Is all of it justified? Probably not all, but certainly some customers will be required to purchase additional licenses for their existing environment when they upgrade to 5.0. But to help take some of the emotion out of the discussion, you first need to see what YOUR environment looks like. Since VMware hasn't yet release their vRAM reporting tool a few VMware community members wrote some scripts to help people out. You will either be pleased by the results, or be stuck with a bill come upgrade time.

You can find a compilation of the scripts here. Users are posting results of their environment..so continue to watch the thread even after you download the scripts. There is also another license validator script that you can read about here.

SQL 2008 R2 SP1 hits the streets

2011-07-13T20:32:00.000-07:00

In the midst of all the uproar about the vSphere 5.0 licensing changes, I missed the fact that Microsoft released SQL 2008 R2 Service Pack 1 yesterday. You can download it here. The master list of bug fixes can be seen here. The new features are listed below. For the full release notes, click here.

Dynamic Management Views for increased supportability. sys.dm_query_stats DMV is extended with additional columns to improve supportabilities over troubleshooting long-running queries. New DMVs and XEvents on select performance counters are introduced to monitor OS configurations and resource conditions related to the SQL Server instance.

ForceSeek for improved querying performance. Syntax for FORCESEEK index hint has been modified to take optional parameters allowing it to control the access method on the index even further. Using old style syntax for FORCESEEK remains unmodified and works as before. In addition to that, a new query hint, FORCESCAN has been added. It complements the FORCESEEK hint allowing specifying ‘scan’ as the access method to the index. No changes to applications are necessary if you do not plan to use this new functionality.
Data-tier Application Component Framework (DAC Fx) for improved database upgrades. The new Data-tier Application (DAC) Framework v1.1 and DAC upgrade wizard enable the new in-place upgrade service for database schema management. The new in-place upgrade service will upgrade the schema for an existing database in SQL Azure and the versions of SQL Server supported by DAC. A DAC is an entity that contains all of the database objects and instance objects used by an application. A DAC provides a single unit for authoring, deploying, and managing the data-tier objects. For more information, see Designing and Implementing Data-tier Applications.
Disk space control in PowerPivot for SharePoint. This update introduces two new configuration settings that let you determine how long cached data stays in the system. In the new Disk Cache section on the PowerPivot configuration page, you can specify how long an inactive database remains in memory before it is unloaded. You can also limit how long a cached file is kept on disk before it is deleted.
Support for 512e Drives. SQL Server now correctly detects and supports hard drives with the new 512e format. These drives report 512 byte logical sector sizes, but they are formatted internally using 4KB sectors. When SQL Server 2008 R2 SP1 is installed on Windows Server 2008 R2 or higher, we will correctly detect these drives and adjust automatically.

vSphere 5.0 Storage Improvements

2011-07-12T20:50:00.000-07:00

If you a regular follower of my blog, you will probably notice I'm a bit of a storage geek. VAAI, FCoE, WWNs, WWPNs, VMFS, VASA and iSCSI are all music to my ears. So what's new to vSphere 5.0 storage technologies? A LOT. That team must have been working over time to come up with all these great new features. Here's a list of the high level new features, gleaned from a great VMware whitepaper that I have a link to at the end of this post.

VMFS 5.0

64TB LUN support (with NO extents), great for arrays that support large LUNs like 3PAR.
Partition table automatically migrated from MBR to GPT, non-disruptively when grown above 2TB.
Unified block size of 1MB. No more wondering what block size to use. Note that upgraded volumes retain their previous block size so may want to reformat old LUNs that don't use 1MB blocks. I use 8MB blocks, so I'll need to reformat all volumes.
Non-disruptive upgrade from VMFS-3 to VMFS-5
Up to 30,000 8K sub-blocks for files such as VMX and logs
New partitions will be aligned on sector 2048
Passthru RDMs can be expanded to more than 60TB
Non-passthru RDMs are still limited to 2TB - 512 bytes

There are some legacy hold-overs if you upgrade a VMFS-3 volume to VMFS 5.0, so if at all possible I would create fresh VMFS-5 volumes so you get all of the benefits and optimizations. This can be done non-disruptively with storage vMotion, of course. VMDK files still have a maximum size of 2TB minus 512 bytes. And you are still limited to 256 LUNs per ESXi 5.0 host.

Storage DRS

Provides smart placement of VMs based on I/O and space capacity.
A new concept of a datastore cluster in vCenter aggregates datastores into a single unit of consumption for the administrator.
Storage DRS makes initial placement recommendations and ongoing balancing recommendations, just like it does for compute and memory resources.
You can configure storage DRS thresholds for utilized space, I/O latency and I/O imbalances.
I/O loads are evaluated every 8 hours by default.
You can put a datastore in maintenance mode, which evacuates all VMs from that datastore to the remaining datastores in the datastore cluster.
Storage DRS works on VMFS and NFS datastores, but they must be in separate clusters.
Affinity rules can be created for VMDK affinity, VMDK anti-affinity and VM anti-affinity.

Profile-Driven Storage

Allows you to match storage SLA requirements of VMs to the right datastore, based on discovered properties of the storage array LUNs via Storage APIs.
You define storage tiers that can be requested as part of a VM profile. So during the VM provisioning process you are only presented with storage options that match the defined profile requirements.
Supports NFS, iSCSI, and FC
You can tag storage with a description (.e.g. RAID-5 SAS, remote replication)
Use storage characteristics or admin defined descriptions to setup VM placement rules
Compliance checking

Fibre Channel over Ethernet Software Initiator

Requires a network adaptor that supports FCoE offload (currently only Intel x520)
Otherwise very similar to the iSCSI software initiator in concept

iSCSI Initiator Enhancements

Properly configuring iSCSI in vSphere 4.0 was not as simple as a few clicks in the GUI. You had to resort to command line configuration to properly bind the NICs and use multi-pathing. No more! Full GUI configuration of iSCSI network parameters and bindings.

Storage I/O Control

Extended to NFS datastores (VMFS only in 4.x).
Complete coverage of all datastore types, for high assurance VMs won't hog storage resources

VAAI "v2"

Thin provisioning dead space reclamation. Informs the array when a file is deleted or moved, so the array can free the associated blocks. Compliments storage DRS and storage vMotion.
Thin provisioning out-of-space monitors space usage to alarm if physical disk space is becoming low. A VM can be stunned if physical disk space runs out, and migrated to another datastore, then resume computing without a VM failure. Note: This was supposed to be in vSphere 4.1 but was ditched because not all array vendors implemented it.
Full file clone for NFS, enabling the NAS device to perform the disk copy internally.
Enables the creation of thick disk on NFS datastores. Previously they were always thin.
No more VAAI vendor specific plug-ins are needed since VMware enhanced the T10 standards support.
More use of the vSphere 4.1 VAAI "ATS" (atomic test and set) command throughout the VMFS filesystem for improved performance.

I'm excited about the dead space reclamation feature, however, there's no mention of a tie-in with the guest operating system. So if Windows deletes a 100GB file, the VMFS datastore doesn't know it, and the storage array won't know it either so the blocks remain allocated. You still need to use a program like sdelete to zeroize the blocks so the array knows they are no longer needed. You can check out even more geeky details at Chad Sakac's blog here.

Hopefully VMware can work with Microsoft and other OS vendors to add that final missing piece of the puzzle for complete end-to-end thin disk awareness. Basically the SATA "TRIM" command for the enterprise. Maybe Windows Server 2012 will have such a feature that VMware can leverage.

Storage vMotion

Supports the migration of VMs with snapshots and linked clones.
A new 'mirror mode', which enables a one pass block copy of the VM. Writes that occur during the migration are mirrored to both datastores before acknowledged to the OS.

If you want to read more in-depth explanations of these new features, you can read the excellent "What's New in VMware vSphere 5.0 - Storage" by Ducan Epping here.

vSphere 5.0 Virtual Storage Appliance

2011-07-12T19:37:00.000-07:00

One of the new features of vSphere 5.0 is a VMware VSA, or virtual storage appliance. VSAs are nothing new, as HP and FalconStor have offered VSAs for vSphere for a number of years. VSAs work by using DAS (direct attached storage, e.g. SAS or SATA) and turn it into shared storage that enables HA features like storage vMotion, HA, FT and vMotion. Of course the primary reason to do this is cost. If you are a SMB or have a remote office, you can deploy a VSA for less money than a physical iSCSI SAN.

Basically you can install the VSA on two to three servers (one server configuration is NOT supported), and it will pool their storage using local RAID, and do network RAID across the physical servers. VMware claims 99.9% availability using vSphere HA. It also has tight integration with vCenter, so you can manage it in a single pane of glass, which is pretty cool.

The VSA is separately licensed, and not included in any vSphere edition. Each instance supports up to three nodes. However, vCenter will only support one VSA instance. So if you have a lot of remote offices and want to use VMware VSAs at them, you really won't be able to do that. You would need to look at alternatives like HP. List price of the VMware VSA is 5,995 per server. You can also buy it with the vSphere 5 Essentials Plus SKU for a total of $7,995 for a limited time.

It is interesting to see VMware now directly competing with partners such as HP for storage business. The P4000 VSA is very feature rich, just like the physical P4xxx servers and include VAAI support. The VMware VSA v1.0 only supports NFS, so you don't get any VAAI 1.0 features that you do with the P4000 VSA since it's iSCSI based. You do get NFS storage I/O control, which is new to vSphere 5.0. The VMware VSA also will have a separate HCL, and pretty short at GA, but VMware says the list will rapidly expand as partners validate the solution.

During the Q&A of the live session it was a bit unclear how VMware calculates usable capacity of the VSA. So stay tuned for more details, once I find more information on the subject. Basically there's a combination of RAID 10 and RAID 5 going on to provide solid data protection in the case of disk or node failure.

As a side note, some SKUs of the HP P4500 physical arrays come bundled with 10 VSA licenses that support up to 10TB each. And there's no vCenter limitation of the number of P4000 VSAs you can use, so that becomes an excellent branch office solution which can scale up very nicely.

You can buy a P4500 model BQ888A which includes the 10 VSA licenses for $43K, so in essence you pay $4.3K for each VSA and get a free 14TB hardware SAS iSCSI array. The VMware pricing reinforces that the VSA is really for Essentials Plus customers, which probably wouldn't pay $43K for a hardware iSCSI array.

vSphere 5.0 Licensing Changes

2011-07-12T18:38:00.000-07:00

One of the major announcements today that has gotten customers worked up a little bit is the change in the licensing model with vSphere 5.0. In previous versions licenses were tied to the number CPU sockets, and the various editions had limitations on maximum physical memory supported and the number of processor cores. These limitations were spread over six major SKUs. With v5.0 that's history, and you now need to keep track of what they call vRAM entitlements and CPU sockets. Oh ya, one SKU got dropped, Advanced edition.

Lifted are the limitations on CPU cores and maximum physical memory. Have a 256GB server with 32 cores? No problem, you can use Standard edition. Nifty? Well, remember the new vRAM licensing concept. This gets a bit confusing at first, so stick with me.

vRAM is the amount of consumed VM memory across the entire environment (within a given SKU, such as Enterprise edition). This not the amount of physical RAM, mind you, but the total amount of RAM allocated to all running VMs. For example, if you have 10 4GB VMs, that would be 40GB of vRAM. Transparent page sharing, ballooning, or other memory conservation features will not help you here.

Now the kicker is that each licensing SKU, such as Enterprise edition, come with a fixed amount of vRAM entitlements. For example, Enterprise edition has a 32GB per socket entitlement. If you have four dual socket servers then you get 256GB of vRAM entitlements (4 x 2 x 32GB). This means on the four physical servers all of your powered on VMs cannot use more than 256GB of RAM. VMware provided the slide below that compares vSphere 4.1 and 5.0 licensing.

Remember vRAM is a pooled asset, so vCenter will manage and track the pooled usage. Linked vCenter instances will pool their memory together. Pools are based on the license SKU, so you have separate pools for each of the five editions. Since it is a pooled asset, you can legally exceed the entitlement on one or more servers, as long as there is excess unused capacity on other servers in the same pool.

What I also found interesting is that VMware will not be selling vRAM only entitlement SKUs. The only way to buy more entitlements is to either upgrade to the next SKU level (.e.g. Enterprise to Enterprise Plus) or buy more socket licenses. According to VMware this change will only result in increased costs for 4% of customers. The VMware chart below shows the list price for the licenses, vRAM entitlements, and features.

How VMware thinks this makes licensing simpler is beyond me. This major licensing change will likely impact large scale designs. So servers like Cisco UCS or large HP blades that can support 384GB or more of RAM could require a lot of new licenses to fully utilize their memory.

Another interesting consideration is VDI, such as XenDesktop. Typically these servers have a lot of memory (96GB to 144GB), and are packed to the gills with running VMs so you can reduce the per-VM hardware costs. The sweet spot for VDI servers has been 2-socket servers packed with memory. For a large scale VDI deployments that has high concurrent usage, licensing could become complex to manage. VDI may not need fancy features like I/O control or storage DRS, so customers may look at lower SKU editions like standard edition. But with the lower vRAM entitlements, you really have to do some careful calculations and likely increase the number of licensed sockets to stay compliant.

--
Update 2: VMware has now announced their vSphere 5.0 Desktop license for VDI. Basically you license desktop VMs in bundles of 100 for $65 each. The host must be dedicated to VDI (no server VMs), and the ESXi functionality is that of enterprise plus, and NO vRAM entitlement limitations. See their blog post here. However, this is only good for NEW vSphere 5.0 Desktop licenses. Customers with existing vSphere enterprise plus licenses that are upgraded to 5.0 are still bound by the vRAM entitlement restrictions.

The calculations below were made prior to the new Desktop SKU. I am glad VMware realizes there are non-View VDI solutions and that customers needed a price break to make VDI more affordable. Thank you VMware!

Update 3: Brian Madden did a more exhaustive VDI cost calculation matrix in his post here.

---

Let's take an example of ~1800 VDI users. Using current 12-core servers, you could probably get ~90 users per server with 144GB of RAM, allocating 1.5GB per VM. For 1800 users that equates to 20 servers, with no spare capacity. Let's throw in three servers for extra capacity, for a total of 23 servers.

Standard:
vSphere 4.x: 23 x 2 x $995 = $45,770 (46 licenses)
vSphere 5.0: (1800 VMs x 1.5GB) / 24GB = $111,937 (112 licenses)
$/VDI VM = $25 vs. $62 (240% increase)

Enterprise:
vSphere 4.x: 23 x 2 x $2875 = $132,250 (46 licenses)
vSphere 5.0: (1800 VMs x 1.5GB) / 32GB = $244,375 (85 licenses)
$/VDI VM = $74 vs. $136 (84% increase)

Enterprise Plus:
vSphere 4.x: 23 x 2 x $3,495 = $160,770 (46 licenses)
vSphere 5.0: (1800 VMs x 1.5GB) / 48GB = $199,215 (57 licenses)
$/VDI VM = $89 vs. $111 (25% increase)

Of course these costs do not take into account other unused capacity in the environment, so this is 'worst case' pricing. But remember that vRAM entitlements are SKU specific. So if you use standard edition licenses for VDI, but enterprise plus for servers, you need to keep track of two separate vRAM pools. What's interesting is that the enterprise edition costs substantially more per VDI VM with v5.0 ($136) than enterprise plus ($111), because of the vRAM entitlement differences. VMware View users may not be hit as hard, but I'm not as familar with those licensing specifics as I support a XenDesktop environment.

As noted in a VMware FAQ, there is no "hard stop" if you reach the vRAM limit in the standard, enterprise and enterprise plus SKUs. There is a hard stop for the vCenter server for Essentials SKU. So in most editions vCenter will only nag you if you are out of compliance, but you won't be left in the lurch unable to power on VMs. Of course you should adequately predict vRAM usage and purchase licenses in advance to stay ahead of the curve and be completely compliant.

It will be interesting to see how much customers push back on this new 'simplified' licensing model, and if VMware changes direction by the time it hits the streets in late Q3 2011. I think this will make customers look at alternatives such as XenServer and Hyper-V more closely. There's a very active thread on VMware forums about the licensing changes, and its mostly shock and awe.

VMware vSphere 5.0 Announced!

2011-07-12T16:59:00.000-07:00

In case you were living under a rock today, or don't have lots of RSS subscriptions for virtualization blogs, you may not have heard that VMware announced their vSphere 5.0 product today. Although not shipping until late in Q3 of 2011, the cat is now out of the bag and technical details are abundant. This is a huge release with hundreds of new features and tweaks, so I'm sure the blogosphere will be crammed with great details over the coming months.

VMware had an online virtual product release with several webinars, live Twitter feeds and live Q&A. So in a series of posts I'll just cover some of the very high level new features, so you get a feel of the magnitude of the updates and hopefully get you interested in reading more on your own.

A few of the major feature enhancements include:

Exclusive use of the ESXi hypervisor. No more ESX.
Auto Deploy. Uses host profiles to provide stateless computers with no local storage. Enables you to rapidly provision new servers and centralize patch management. No no longer really patch servers, you reboot the server and it will download a whole new image.
Storage DRS. Tiered storage based on performance characteristics. Load balance VMs based on I/O profile and align with SLAs. You can put a datastore in maintenance mode and all VMs will be vMotioned to other datastores.
Added support for NFS storage I/O control (previously limited to block storage)
Per-VM network I/O controls, to help eliminiate noisy neighbors.
VMs can now support 3D graphics
Supports client-connected USB devices
Support for USB 3.0
Supports smartcard readers
Mac OS X server support
Hardware VM version has been increased to v8.0 and EFI virtual BIOS
VM limits increased to 32 vCPUs, 1TB RAM, support 1,000,000 IOPS, >36Gb/s network throughput
Brand new HA architecture. Supports larger clusters, simplified setup, and more reliable.
vCenter appliance running on Linux. Only supports Oracle DBs. Didn't VMware learn from vCloud? Not as full featured as the Windows version.
Brand new web client to manage vSphere from anywhere.
Networking supports Netflow, SPAN support and LLDP
ESXi now has a built-in firewall
VMFS version increased to 5.0 (online non-disruptive update from prior versions)
VMFS support for datastores up to 64TB without using extents
VAAI v2
Software FCoE initiator
vMotion support for higher latency links (up to 10ms)
Dropped the "Advanced" licensing SKU
Licensing is now based on CPU sockets AND vRAM (see my licensing post here). No more core/memory limitations.
vCenter Heartbeat 6.4 supports SQL Server 2008 R2 and vCenter plug-in for monitoring.
New vSphere Storage Appliance

Nearly each feature could have a dedicated blog post about it, so this is just a small snapshot of some features. Other products like SRM and vShield have also undergone major updates. Stay tuned for a lot more post about new features.

Get Your Nerd On T-shirt now!

2011-07-02T12:58:00.000-07:00

A friend of mine, Chris McCain, has some cool t-shirts for sale on his site, Get Your Nerd On.

Check 'em out and get one today!

SQL 2008 R2 Cumulative Update 8 Released

2011-06-26T11:12:00.000-07:00

Microsoft recently released SQL Server 2008 R2 cumulative update package 8 to the web for downloading. You can request the hotfix from this page. Also remember that the June 2011 monthly security patches included some for SQL Server (2005 through 2008 R2). You can find the whole bulletin list here. SQL security updates are fairly rare, so you might not have checked the bulletins this month.

Citrix NetScaler Active Directory Authentication

2011-06-26T09:23:00.000-07:00

The Citrix NetScaler can be configured to authenticate users against a variety of sources including RADIUS, LDAP, TACACS, and PKI certificates. If you are going to use the NetScaler as an Access Gateway (proxy) between and untrusted network such as the internet and your corporate network, you will probably want to have the NetScaler perform authentication functions.

Configuring the NetScaler for AD authentication is not difficult, but there are a few settings you should watch out for. I was using NetScaler v9.3 for these configuration steps, so other versions may have slightly different options or windows.

1. In Active Directory create a group that the members of which need to be permitted inbound access to your network. For my environment I used AccessGateway_RemoteUser. Create a service account in AD that will be used to bind to Active Directory, such as SVC_NetScaler_Admin.

2. In the NetScaler GUI go to the System folder and click on Authentication. Next, click on the Servers tab, then right click in the window and select Add.

3. Enter a name for this authentication server. I use the hostname of the AD server I'll be authenticating against. Change the authentication type to LDAP then enter the IP address of your Active Directory server. Don't configure the port number as we will do that later. Configure the base DN and Administrator bind DN according to your environment, and type in the password for your service account.

4. In the lower half of the window you need to configure the Search Filter and SSO Name Attribute. The search filter maybe a little confusing at first. Open option is using of memberof=cn=. If you know LDAP well you can create different filters as needed. For the SSO Name Attribute, use samAccountName.

5. At this point you need to configure the security for the LDAP services. The exact configuration will depend on your Domain Controller configuration. The most secure is the SSL option which uses port 636, but your DC must have a server authentication certificate. The next best is TLS, where it uses port 389 but tries to use the LDAP StartTLS command to encrypt communications.

To verify which one will work click on Retrieve Attributes under connection settings and verify a connection can be established. After you know which setting works, click OK. Note that if you use the PlainText option that the NetScaler will disable the ability of users to change expired passwords during the logon process.

6. In the NetScaler GUI go to the Authentication Policies tab, right click in the window and select Add. Input a logical name for the authentication type (e.g. Active Directory), change the Authentication Type to LDAP and pick the server name you just created.

7. In the lower half of the window select True Value from the drop down and click Add Expression. ns_true should now appear in the Expression window.

8. Your configuration should now look very similar to the window below.

9. At this point I would bind this authentication mechanism globally to the NetScaler. To do that you right click in the Policies window and select Global Bindings. Select the policy name from the drop down then click OK.

Now you are ready to rock and roll. NetScaler services such as Access Gateway can now take advantage of your Active Directory authentication services you configured. If you want to provide high availability for your authentication services, you could configure LDAP load balancing as I describe here and use that VIP instead of the IP address of your domain controller back in step 3.

LDAP Load Balancing with Citrix NetScaler v9.3

2011-06-25T08:00:00.000-07:00

When using a load balancer in an enterprise environment it opens up the possibilities for service level redundancy that you may not have thought of before. For example, maybe you have appliance devices on the network that can be Active Directory integrated, but only allow you to specify one LDAP server (HP iLO, for example). Maybe you have multiple datacenters and you want to provide seamless datacenter failover in case of an outage for a service, such as a web site. Or maybe you have a global network and want to direct users from a particular region of the world to the nearest server to provide the best response times. Advanced load balancers can do all of this, and more.

Out of the box the Citrix NetScaler has a the capability to load balance LDAP requests, and also has intelligent monitors that do more than just see if the TCP port LDAP uses (389) is alive. The monitor can perform a query against the LDAP server to ensure the LDAP service is actually returning valid data. So let's build a load balanced LDAP virtual server in the NetScaler and utilize the intelligent LDAP monitor provided by Citrix. A future blog article will cover the same configuration but for LDAP over SSL. These instructions are written using NetScaler v9.3, but should be fairly similar in other releases.

1. Create a service account in AD that will be used for the LDAP monitor. It should not have any special privileges. Let's call ours SVC-NS-LDAP.

2. Open the NetScaler management GUI and open the Load Balancing folder. Go down to the Servers container and create a new server object. Enter a logical server name. I would use the FQDN of your first Active Directory server. Next you can enter the IP address or domain name of the server. I prefer using the domain name so if a server's IP changes you don't wonder why your monitor or load balanced service is broken. Click on Create. Repeat the process for your other AD servers.

3. Under the Load Balancing folder on the NetScaler click on the Monitors container. Create a new monitor. On the first window enter a logical name, such as LDAP_389 and change the monitor type to LDAP. Leave all other parameters on this window alone.

4. Click on the Special Parameters tab then click on Browse and locate the nsldap.pl script. For the remaining fields use:

Dispatcher IP: 127.0.0.1 (Do not change this IP)
Dispatcher Port: 3013 (Any unused NetScaler port will work but 3013 seems popular.)
Base DN: dc=contoso,dc=net (Substitute your domain information of course.)
Bind DN: cn=SVC-NS-LDAP,cn=users,dc=contoso,dc=net (Use your path.)
Filter: cn=builtin (This is a standard object in AD.)
Password: xxxxx (Enter the password of your service account)

Note that the filter parameter is very important so the LDAP server doesn't return every object in your domain. You only need a single object to return from the query to ensure LDAP is working. Do NOT leave this field blank!

5. Under Load Balancing in the NetScaler GUI open the Virtual Servers container. Add a new virtual server and use a logical name such as ldap.contoso.net_389. Change the protocol to TCP, enter the IP address of the new virtual server and use port 389. Click on the Service Groups tab and select the LDAP_389 group.

6. If all goes well you now have a functioning monitor that shows an UP state.

7. Optionally you can now create a DNS entry for the new virtual server, say ldap.contoso.net, so now any devices that need load balanced LDAP services can simply point to this DNS name. Of course if the device doesn't support DNS you can specify the virtual server IP address. Just like the rationale behind creating 'servers' based on DNS entries in the NetScaler, use DNS names when possible to lessen the work required when IP address changes occur.

8. To test out that the new virtual server is actually working, hop on one of your servers that has the ldp.exe tool installed. This is baked in starting with Server 2008 and later. Launch ldp then select connect. Enter the new LDAP DNS name or the virtual server IP address. Next select bind, leave the rest of the options, and click on OK. You should see messages showing the connection was successful.

9. If you want to get really geeky and verify that the search LDAP search results for the LDAP monitor are correct you can whip out WireShark and do a network trace. Look for "searchResEntry" to see the results of your query.

And there you have it! Load balanced LDAP! You should now do some testing by bringing down one of the AD servers you are load balancing across then reconnect with the ldp tool and verify you can still connect. As mentioned earlier, if your load balancer supports global load balancing, you can get really fancy and have geographically redundant LDAP. LDAPsoft also has a nifty LDAP browser you can use free for 15 days that is worthwhile to check out if you are a LDAP geek.

Updating your NetScaler Management Interface SSL Certificate

2011-06-23T21:19:00.000-07:00

When you install the Citrix NetScaler it comes with a self-signed certificate which is bound to the management IP interface for the purposes of encrypting management traffic. However, using self-signed certificates are not recommended in anything but a lab environment. So that means you need to install and configure the NetScaler to use a new certificate for all management traffic. Thankfully Citrix has made this super easy! These steps were performed on NetScaler v9.3, other versions may slightly vary.

Here's how!

1. Create a trusted SSL certificate and upload it to the NetScaler. The certificate should be for the FQDN that you want to use for the NetScaler management interface, not any of your Vservers. To do that follow my blog article here.

2. In the NetScaler GUI interface navigate to the Certificates folder under SSL, right click on ns-server-certificate and verify that it is bound to several interfaces. The bindings indicate that the certificate is in use, which is good.

3. Right click on ns-server-certificate and select Update.

3. On the following screen navigate to the certificates located on the appliance that you created in step one and click OK.

4. If the update goes as planned you will now see the new certificate names in the certificate list.

5. Close out the NetScaler management interface and reconnect via HTTPS. Open the certificate properties in your browser and verify that the trusted certificate is being used.

Load Balancing XenDesktop 5 with NetScaler 9.3

2011-06-22T20:31:00.000-07:00

As I mentioned in yesterday's blog post, any enterprise VDI deployment needs redundant broker services for high availability. Other enterprise applications such as Exchange, Lync, and SharePoint can all benefit from a load balancer, be it virtual or physical. Building on yesterday's post about configuring SSL on the NetScaler, it's now time to configure load balancing for the XenDesktop DDCs and Web Interfaces.

I'm making a few assumptions here. First, you already have XenDesktop 5 installed and functioning in your environment, hopefully with redundant WI and DDC servers. Second, you've configured the WI servers for SSL. Third, you've already deployed the NetScaler and using at least version 9.3. Fourth, you've installed a SSL certificates on the NetScaler for the DNS names you've assigned to your WI and DDC virtual IPs.

Environment:
XenDesktop combo DDC/WI: 192.168.0.200 and 192.168.0.201
Web Interface virtual IP: 192.168.0.100
DDC virtual IP: 192.168.0.101

1. Download the Citrix AppExpert template for the Citrix Web Interface here.
2. In the NetScaler open the AppExpert folder, right click on Templates and select Manage Templates.
3. Click on the Upload button and locate the XML file you downloaded in step one.
4. After the template imports click on Load Balancing in the NetScaler GUI. You should now see two new wizards under Getting Started.

5. Start the XenDesktop wizard and enter the appropriate information in the WI server wizard screen. The IP addresses are pretty self explanatory. I would recommend configuring a health monitoring service account. This will allow the NetScaler to actively attempt to authenticate to ensure the WI is actually functional. One critical change you need to make to the form is the site path. You MUST remove site/default.aspx, as shown below.

6. For the DDC configuration page it's pretty clear what you need to input. Remember you will need to use a unique IP address for the DDC virtual server. And again, I'd configure a service account for health monitoring. You could use the same account or a different one.

7. Close the wizard and if everything is correct, it will create the virtual servers, service groups, monitors, and servers for you. It is very likely though that the WI monitor will show a down status, while the DDC monitor may show as UP. If that happens, it's probably an SSL issue which we can easily resolve.

8. Open the WI virtual server and see if you see the error below, certkey not bound, you are in luck as this is an easy fix.

9. Click on the SSL Settings tab and select the appropriate WI SSL certificate that you either created from my blog yesterday if you are just testing, or your real one if this is a production deployment. Click on Add to move it to the configured column.

10. Close the window and now your WI State should be UP and 100% health.

11. Repeat the SSL assignment exercise for the DDC monitor using another certificate which matches the DDC DNS name you chose earlier.

Next up, open your browser and go to the FQDN for both virtual servers and verify that the XenDesktop login screen appears with no SSL warnings. If so, you've now created two VIPs for load balancing critical XenDesktop services and enabled health monitoring. High availability baby!

Creating a SSL certificate for Citrix Netscaler

2011-06-21T20:03:00.000-07:00

A high availability VDI deployment, such as XenDesktop 5, demands that you use multiple servers to provide broker redundancy. As such, a load balancer such as the Citrix Netscaler comes in mighty handy. The NetScaler can also act as an ICA proxy between a trusted and untrusted network, such as the internet and your corporate network. Now that I've gotten XenDesktop 5 running in my lab, I wanted to see what it takes to configure the NetScaler Access Gateway feature to allow external inbound connections and serve up a nice VDI desktop.

As the configuration is somewhat complex, let's start with the easy part, creating your own SSL certificate and importing it into the NetScaler. Now in the real world you'd need to use a trusted CA like Verisign, or your clients won't trust the Access Gateway and the Citrix receiver will not launch. However, if you are in a lab or home environment you can use your own CA just to get the flavor how it works.

In my lab I'm using the latest NetScaler VPX release, which is v9.3 build 48.6.nc. First we need to use OpenSSL to create a private key, then a certificate request, convert the private key, then submit to my Microsoft CA, and finally import into the NetScaler. Figuring out this process was a bit easier than VMware makes it for importing certs into an ESXi host, so you have that going for you.

1. Login to the NetScaler and click on the SSL folder in the left pane.
2. Generate a private RSA key by clicking on Create RSA Key. Use a filename that is easily associated with the FQDN of the certificate and I would use a .key extension to denote it's the private key. 2048 bits is the maximum keysize, so I'd go for that. Change the format to DER. Click on Create then Close.

3. On the NetScaler SSL page click on Create CSR. Type in a file name for the certificate request (I'd suggest a .req extension), then browse to the private key file you just created. In the Common Name field enter the FQDN you want your certificate to be bound to. Fill in the other information as needed. Click on Create then Close.

4. Back on the SSL page click on Manage Certificates then locate the REQ file, highlight it, then click on View. Copy the contents to the clipboard. Close the window.
5. Assuming you are using a Windows Server 2008 R2 CA, perform these steps:

Go to the certificate home page and click on Request a certificate.
Select Advanced certificate request.
Select Submit a certificate request by using a base-64-encoded....
Paste the certificate into the window and change the template to web server.
Download a DER encoded certificate (not the certificate chain) using a logical name like xd-contoso-net.cert.

6. Back on the NetScaler and open the SSL folder then click on Certificates.
7. Right click in the SSL window and select Install.
8. I would suggest the FQDN for the pair name, browse locally to the certificate file name, then browse on the appliance for the private key, and change the certificate format to DER.

9. Click on Install and hope that the certificates import successfully. Once the certificate imports, you should delete the certificate from wherever you downloaded it to on your workstation.

And there you have it! You've created your own private key, certificate request, generated a SSL certificate, then imported it to the NetScaler. The private key and public key file names are important, since the files are stored on the NetScaler and each certificate must have a unique name. You can repeat this process for any number of certificates, as needed.

Windows 8: Hyper-V 3.0 baked in!

2011-06-20T09:09:00.000-07:00

Wow this is pretty darn cool..in the latest leaked build of Windows 8 x64, a blogger found Hyper-V 3.0 is baked in and is sporting a number of new features. This is the first time MS has baked in a hypervisor to a client operating system. While some of the new features aren't really relevant to a desktop user (like a virtual fibre channel adapter) or 16TB VDHX files with power fail resiliency, it does open up a world of possibilities for handling application compatibility issues. For the more geeky folks that like to use a type-2 hypervisor like VMware Workstation, there's finally a MS solution for running 64-bit VMs on your desktop operating system.

You can check out all of the new features and some screen shots here. A short summary of enhancements include:

Support for more than four cores
Virtual Machine Queue and IPsec offload
Bandwidth management
DHCP Guard
Router Guard
Monitor Port
Virtual Switch extensions
Network Resource Pools

Since Windows 8 RTM isn't expected until mid to late 2012, there is plenty of time for Microsoft to add additional features. Of course Microsoft could also pull Hyper-V 3.0 from the client OS too, but let's hope not.

Outlook in a VDI environment? Think Exchange 2010!

2011-06-18T21:20:00.000-07:00

When migrating towards a VDI environment you really have to re-think your entire architecture. Servers, hypervisor, storage, application delivery, network, and everything in between. Simple things like anti-virus can wreck havoc and cause massive I/O storms that bring your VDI environment to its knees. One aspect that I hadn't thought about was Outlook performance with VDI. You may think, so what? What's different about using Outlook with VDI? A LOT!

I came across this great blog post by Kraft Kennedy which VDI architects really need to review if your organization uses Outlook and Microsoft Exchange. His summary really hits home:

"If you’re considering VDI and are concerned about Outlook performance, I’d strongly recommend moving to Exchange 2010. Many of the problems are addressed in Exchange 2010 and it can deliver a good Outlook experience for all VDI users."

Still running Exchange 2003 or 2007 and moving full steam towards VDI? Start planning your Exchange 2010 migration now!

Align your partitions with VMware Converter 5.0 Beta

2011-06-15T19:52:00.000-07:00

Update: VMware released the GA version of 5.0 and you can download it here.

A few days ago VMware released a significant update to their standalone converter utility, Converter 5.0 beta (download here). One of the cool new features is the ability to re-align partitions. By default Windows Server 2003 and Windows XP do not have properly aligned partitions. This can cause addition IOs and poor VM performance. Windows Vista and Server 2008 and later are smarter and automatically align partitions on a 1MB boundary.

So I decided to try out the new feature and verify that a conversion process did in fact align the partitions. To perform the test I already had a Server 2003 VM in VMware Workstation 7 that had an improperly aligned partition:

Here you can see a starting offset of 32,256 bytes which is 31.5KB. No good! You ask how did I get that information? Simple...from a command prompt type:

wmic partition get blocksize, startingoffset, name, index

Next I fired up Converter 5.0, ran through the wizard to convert it to an ESXi VM, and saw this nifty screen:

The 'create optimized partition layout' appeared when I selected a volume copy option. Whoohoo! I ran through the rest of the wizard, waited 4.5 hours (gotta be a beta bug to take this long), and viola, ended up with a newly converted VM on my ESXi host. Now did the converter actually work? Let's see:

By George, yes the starting offset is now divisible by 32K. You can now sleep better knowing that your disk subsystem is working as efficiently as it can. This tweak can be really important in a Windows XP VDI environment where there's a lot of disk IO and any savings can be substantial when multiplied by hundreds or thousands of VMs.

If you want to get really fancy and change the NTFS cluster size during the conversion process, you can click on the Advanced button in the figure above and tweak as shown below:

PowerCLI Script to dump VDI VM IO Stats

2011-06-11T20:29:00.000-07:00

During a VDI pilot you really need to gather some real-world stats on how your client VMs are performing. People generally underestimate the workload VDI can put on a SAN. Thankfully if you are using vSphere 4.x, you can easily pull the stats with a short PowerShell script. I took a great script LucD has overe here and changed it up a bit to be more VDI specific. I'll be the first to admit PowerShell is an area I need to learn more in, so the script could probably be more efficient, but hey, this works for me!

Unlike the LucD version, this requires a couple of command line arguments so you can more rapidly change the VM names and time period you are reporting against. The first argument is the VM name and the second is whole number which represents the number of minutes the stats need to be displayed for. The report is displayed both on the screen and dumped to a CSV file with a unique date stamp, so you don't accidentally overwrite your results.

Unfortunately VMware sets the minimum realtime stats sample size at 20 seconds, so the IOPSmax value is over a 20 second interval. Within that 20 seconds there could well be higher spikes, so don't take that value as the absolute max. Make sure you take into account the 20 second window when setting the sample time, so you don't lose some data.

Update: Script has been modified a bit so that you can use wildcards for the hostname and it will properly calculate all the stats for each VM.

-------

$vms = $args[0]
$time = $args[1]
$metrics = "disk.numberwrite.summation","disk.numberread.summation"
$start = (Get-Date).AddMinutes(-$time)
$report = @()
$stats = Get-Stat -Realtime -Stat $metrics -Entity $vms -Start $start
$interval = $stats[0].IntervalSecs
$date = get-date -format "dd-hh-mm-ss"
$report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{

  $AvgIOPS = [math]::round((($_.Group | `
    Group-Object -Property Timestamp | `
    %{$_.Group[0].Value + $_.Group[1].Value} | `
    Measure-Object -Average).Average / $interval),2)

  $MaxIOPS = ($_.Group | `
    Group-Object -Property Timestamp | `
    %{$_.Group[0].Value + $_.Group[1].Value} | `
    Measure-Object -maximum).maximum /$interval

  $WriteIOs = ($_.Group | Group-Object -Property Timestamp | `
   %{$_.Group[0].Value} | Measure-Object -sum).sum

  $ReadIOs = ($_.Group | Group-Object -Property Timestamp | `
   %{$_.Group[1].Value} | Measure-Object -sum).sum

  $TotalIOs = ($_.Group | `
    Group-Object -Property Timestamp | `
    %{$_.Group[0].Value + $_.Group[1].Value} | `
    Measure-Object -sum).sum

  $ReadRatio = [math]::round(($readios / $totalios),2)

  $WriteRatio = [math]::round(($writeios / $totalios),2)

New-Object PSObject -Property @{
  VM = $_.values[0]
  AvgIOPS = $Avgiops
  MaxIOPS = $MaxIOPS
  WriteIOs = $writeios
  ReadIOs = $readios
  TotalIOs = $totalios
  ReadRatio = $readratio
  WriteRatio = $writeratio
}
}

$report | Export-Csv "D:\IOPSMax-report-$date.csv" -NoTypeInformation -UseCulture

VDI Storage - Right size it or anger your users!

2011-06-04T18:15:00.000-07:00

According to Citrix, the number one mistake people make when deploying VDI is not sizing their storage properly for performance. However, calculating a target IOPS for your environment is far from easy and you really need to understand VDI, its unique workloads, and your storage subsystem. There are many blogs about this topic, so I'm not going to rehash them. But I will provide a good list of what I think is required reading if you are going to deploy VDI in production on any scale, above a few dozen VMs.

Great links to check out:

VDI Calculator
Finding a Better Way to Estimate IOPS for VDI
Windows 7 IOPS Deep Dive
Virtual Desktop Resource Allocation
Improper Storage Design for Virtual Desktops is a Killer
Deciding on Local or Shared Storage for your Desktop Virtualization Solution
Estimate IOPS for Virtual Desktops
Data Storage for VDI - Part 2 - Disk Latencies
Data Storage for VDI - Part 3 - Read and Write Caching
Data Storage for VDI - Part 4 - Impact of RAID on Performance
Video: Storage Infrastructure Design Guidelines
HP XenDesktop 1,000 User Reference Architecture
RAID options with Desktop Virtualization
Local or Shared Storage - that is the question
Does Cache Trump IOPS
VDI Storage Calculator Spreadsheet
VDI & Storage - Deep Impact
VDI IOPS Calculator

Interesting VDI storage products:
xiotech Hybrid ISE
WhipTail
Atlantis Computing
IntelliCache and the IOPS Problem

Below is a sample calculation for 2,000 users using a moderate IOPS profile. According to these calculations for RAID-1 your disk array would need 273 15K disks!

Enjoy!

Threats and Countermeasures Guide for Windows 7 and Server 2008 R2

2011-06-03T17:50:00.001-07:00

Finally, Microsoft has released their Threats and Countermeasures guide for Windows 7 and Server 2008 R2. It is a very lengthy document going into gory details on hundreds of security settings, mostly covered by GPOs. It's a great resource for understanding WHY Microsoft recommends a certain setting, and understanding implications of the lockdown, such as breaking compatibility.

You can download the full guide here.

Use a Service Account with XenDesktop 5

2011-06-02T19:27:00.000-07:00

During the testing process of Citrix XenDesktop 5, we were using the built-in SQL express database so we can do a quick lab setup. Of course a production deployment would use a full blown SQL 2008 R2 enterprise edition instance with database mirroring. During the XenDesktop 5 SP1 upgrade process, we ran into an interesting error that was related to how I did the original XD5 installation.

During the original installation process I logged into the freshly provisioned VM that was soon to become our all-in-one XD5 server. So of course I logged in with my admin credentials and performed the installation, using the free built-in SQL Express option. All was fine and dandy, until another administrator tried to install SP1.

During the SP1 installation process the other administrator ran into a problem, that was tracked down to SQL. As it turns out, my account (original installer) was automatically configured as the CitrixXenDesktopDB DBO.

So naturally when someone else came along and ran SP1, which needs to update the XD database, he ran into problems. While there are several solutions to the problem, I will propose a solution that solves two problems at once.

As we also discovered, the original installer's account credentials are also used to connect to vCenter. As it turns out I'm a vCenter admin, so that process was transparent. However, when my password expired, XenDesktop broke because it couldn't contact vCenter. Bad!

One elegant solution is prior to installing XD5 is to create a service account, and configure it for a non-expiring password. Next, give that service account local admin rights on your XenDesktop server. If you are using MCS with vCenter, give that account the required vCenter rights. Finally, login with that service account on the XenDesktop server and proceed with your installation process. This way both SQL Express and vCenter credentials are using those of the service account, not your personal admin account.

I really wish the XD5 installer prompted for service account credentials, so both of these problems could be automatically avoided. Only after several weeks of testing and a new service pack release did we run into these issues.

Radical new Windows 8 GUI

2011-06-01T17:04:00.000-07:00

So here's an official video from Microsoft showcasing a preview of their new Windows 8 GUI. All I can say is wow...is it different! Built from the ground up for touch..looks pretty darn cool.

You can watch the HD video here. It will be exciting to see more information about Windows 8 and Server 2012 over the coming year.

Free vSphere Compliance Checker

2011-05-31T19:28:00.000-07:00

Yippee..a free tool from VMware! This nice little tool runs compliance scans against vSphere hosts and compares the results to the VMware Hardening Guidelines. Almost a year ago I wrote a short blog announcing their hardening guide here. Since then, VMware released a hardening guide for vSphere 4.1, which you can find here.

This tool beats trying to do manual scans to see how compliant your environment is. The free tool only scans five hosts at once, and I can't find a way to display which VMs are not in compliance. It just gives the server an overall score for each items. So it has very limited utility, IMHO. If you want more detailed information, then you step up to their paid product, vCenter Configuration Manager or a third-party tool.

You can download the free tool here. Be aware that you need Java installed on the computer you run the scan from, and on 64-bit systems it may default to the wrong Java directory path. Scanning my lab host took less than a minute, and came up with several non-compliant settings, most I was aware of and accepted the risk since it's just my home ESX server.

XenDesktop 5.0 Service Pack 1 Released

2011-05-29T13:14:00.000-07:00

A couple of weeks ago Citrix released XenDesktop 5.0 SP1, which fixes a number of bugs. Last week I attended Citrix Synergy 2011, and talked to an employee about SP1. He said that the service pack fixes many more bugs than the short list included in the release notes. In fact, his comment was that more than 600 bugs were fixed. So given my experience with the GA release, which you can read about here, I thought I'd give SP1 a whirl to see if the bugs I encountered were fixed.

Using a PVSCSI controller in the guest VM on ESX 4.1. Fixed!
Using a dvSwitch (such as the Nexus 1000v) on ESX 4.1. Fixed!
Guest VM does not unmount an ISO image when cloned on ESX 4.1. Fixed!
ESX 4.1 VMX/nvram settings not copied from master VM template to clones. Still Broken.

It's great to see Citrix fixing many of the VMware related bugs. SP1 officially supports ESX 4.1 U1, so that's also good news. But it is disappointing that the VMX and nvram settings are not copied to the cloned VMs. This is a big security issue, since our VMX files contain dozens of required lockdowns. The nvram settings also control floppy drive settings. Since that gets reset all of our VDI VMs have a floppy drive shown. Yes we can use a GPO to hide the floppy, but this really should be handled by Citrix.

In case you missed, it, Citrix also has publicly released a 'technology preview' of the next version of XenDesktop, likely version 5.5. Details on this release can be viewed here. Cool new features include Windows Aero redirection, enhanced Flash redirection, improved WAN scanner support, and HDX 3D Pro support. You can download the full package from here.

VIR401: RDP, RemoteFX, ICA/HDX, EOP and PCoIP: VDI Remoting Turned Inside Out

2011-05-22T20:27:00.000-07:00

This session was presented by two non-Microsoft speakers: Benny Tritsch from AppSense and Shawn Bass from Syn-Net. After seeing their extensive testing methodology and the presentation of the results, I feel confident that the results are accurate and as neutral as one could get. The whole session was presenting the results of a battery of tests against the RDP, RemoteFX, ICA/HDX, PCoIP, EOP, Blade and RGS remoting protocols. The battery of tests included 2D graphics, video and animation, 3D graphics, LAN performance, and two WAN performance scenarios using a WAN simulator. They clearly went to great lengths to fairly test the protocols and compare the results.

One of their comments was that these remoting protocols have undergone significant changes in the last three years, and they would have not predicted the rapid advances and very good WAN performance. So for those organizations looking to implement VDI across the WAN, this is very good news. While all is not perfect for VDI WAN usage, the industry is pouring a lot of money and resources into advancing the technology.

Given the nearly all of the session was presenting video clips of all the tests, it's really hard to summarize the results in a meaningful way. Thankfully, Microsoft has links to the audio/video recordings of the session so you can see the results for yourself. You can check out my TechEd 2011 trip report here, which has links to all of the sessions I put on my schedule. Use your browser and search on that page for VIR401.

In case you don't want to listen to the entire 75 minute session, here's a tiny snipit of the results:

When using the Microsoft RDP client (mstsc), it is VERY important to select the appropriate 'experience' if you are using it over a WAN. Selecting the Satellite setting can greatly improve the performance over WANs by using latency 'hints.'
VMware View does not support any type of multi-media redirection with Windows 7. If you want MMR, look at Citrix HDX (or EOP).
None of the protocols can redirect SilverLight or WPF content.
Most of the protocols are very comparable on the LAN. Bandwidth utilization does very somewhat by protocol. For example, watching a SilverLight video over the LAN you can see bandwidth spikes to 45Mbps or more with RemoteFX.
At 50ms WAN latency, you start getting a degraded user experience. Sometimes even at 20ms the experience can suffer.
Packet loss is a huge factor in perceived WAN performance. Their tests used a .01% packet loss, which is typical of a MPLS circuit, but lower than a regular internet connection.
Newer protocols like RemoteFX require additional hardware, be it GPUs or ASICs. Hardware assist generally does provide better results, so expect to see more hardware dependencies in the future.
In the WAN scenarios there's not a huge user experience difference between Citrix HDX and VMware View. In the charts shown, VMware View did use more bandwidth than HDX.
Not all protocols support all versions of DirectX and OpenGL. So you really need to look at the applications you are using and what graphics subsystem they require. OpenGL 1.1 is pretty broadly supported. However, RemoteFX only supports 100% of OpenGL 1.1 features, and at 1.2 and higher it's basically unsupported. On the other hand, Citrix HDX 3D supports all features through OpenGL 3.3 and much of OpenGL 4.1.
For an optimal user experience, 2Mbps of bandwidth is best.
If you are just using Office type applications for text manipulation, over the WAN most of the protocols do a really good job. Multi-media and graphics manipulation is where the differences really start to show up.

I'd really encourage you to watch the videos at my link above, so you can see the visual differences between the protocols. No one protocol stood out as horrible in every test, and no one protocol wiped the floor either. You need to look at the applications you want to use, performance requirements, hypervisor requirements, and other factors to make a determination on what VDI protocol you want to select.

In reality your primary choices are Microsoft RemoteFX, Citrix HDX, and VMware PCoIP. VMware View is limited to the ESX/i hypervisor, and RemoteFX is limited to Hyper-V. So that leaves HDX as the primary hypervisor independent protocol.

SIM210: Sneak Peek at Service Manager 2012

2011-05-22T14:34:00.000-07:00

One of the objectives for SM 2012 is to deliver IT as a Service (ITaaS). ITaaS objectives are to reduce costs, increase service levels, faster time to delivery, provide more data, more transparency, and increased compliance. Implementation components include automation, standardization, self-service, and compliance. The deployment of these components are guided by process designs such as MOF, ITIL and COBIT.

Highlights of this session include:

Integration with Operations Manager 2012, Configuration Manager 2012, Virtual Machine Manager 2012 (new to 2012), and Orchestrator 2012 (new to 2012).
Service manager enables self-service through a portal, reports and dashboards, Excel, and email.
Business processes can be defined in templates and the Configuration Management DB (CMDB), provides a common model and reconciliation of data.
A new compliance library maps legalese to actionable IT control activities. Compliance is continuously and automatically evaluated in real time through SCCM integration.
Improvements from SM 2010 include tracking of incident SLAs, parent/child work items, AD connector improvements, PowerShell integration, parallel activities, and performance improvements.
A new Service Catalog and self-service portal deeply integrate with Orchestrator and VMM to enable ITaaS.
SM imports cloud objects, VMM templates, and runbooks. An admin then creates templates to capture business processes and the role of runbooks within the processes. The admin creates default values to standardize offerings. Roles are then mapped to offerings, to limit access. Born is a self-service portal.
Request processes drive automation. By using a request template, runbook activity, service requests, and connectors, business processes become automated.
The SM portal is completely new and based on SharePoint Foundation 2010 and a Silverlight interface. You can customize webparts using SharePoint admin tools, and it's very extensible. It features service catalog scoped to users, and customizable dynamic forms.
The System Center Data Warehouse replaces the not-often-used System Center Reporting Manager. This enables self service report and dashboard authoring with OLAP cubes. Report authoring with Office integration for knowledge workers. Although you still may need to have some SQL reporting expertise for super-custom reports, this new OLAP model when combined with Excel really enable powerful data slicing and dicing with limited skills.
There's a new Exchange connector for enhanced email integration.
You can more easily report on KPIs (Key Performance Indicators).

I was pretty impressed with the amount of enhancements in SM 2012. Given that the shipping product is just over a year old, MS clearly invested a lot of developmental resources into SM. It feels like a new rev of the product, not just a R2 version with minor tweaks. One of the demos that really blew me away was the ability to slice and dice the OLAP cubes in Excel to create custom reports, then upload those report forms in SharePoint to create a live dashboard. With just a few clicks of the mouse Excel was able to instantly drill down into complex data sets, visualize the data, and present meaningful data to the end user.

For organizations that have an existing ticketing system, like Remedy, Service Manager should be seriously considered for your environment. The integration with the entire System Center suite, SharePoint, self-service portal, and reporting is amazing. If you are serious about automating your processes, enhancing compliance, SM 2012 should be at the top of your list for consideration.

VIR313: RemoteFX GPU Virtualization Deep Dive

2011-05-22T13:44:00.000-07:00

This session went into great gory depth on how RemoteFX works, and the hardware requirements. What is RemoteFX? RemoteFX is a new technology in Windows Server 2008 R2 SP1 and Windows 7 SP1 that when combined with a VDI environment allows efficient host-side rendering of graphics and multimedia. It requires the use of Hyper-V, and thus does not work on other hypervisors.

It also enables USB device redirection, something not available with previous versions of RDP. Host side rendering has several advantages, including rendering of any content including WPF, Flash, SilverLight, Quicktime, WMP, 3D applications, etc. RemoteFX bumps up the RDP version to 7.1.

Why is RemoteFX important? Well with RemoteFX you invest in server hardware with GPUs, but this allows you to buy very cheap and disposable thin or zero clients that don't require much computing or GPU power. This is in contrast to some technologies like Citrix HDX that purposefully offload some CPU/GPU processing to the client devices. There's no one right or wrong way to handle rich graphics, so both architectures have their place.

Enough background, so here are some of the gory technical details covered:

Virtualized GPU. A single GPU can be utilized by multiple Hyper-V guest operating systems. It uses intelligent screen capture and hardware-based encoders. It can utilize hardware-based decode, but this is optional.
The CODEC is designed for text and image-based content. A single CODEC works for VDI, RDS, and WMS (Windows multi-point server) sessions.
USB redirection supports nearly all USB devices, and no client side drivers are required. Admins control what devices are or are not allowed to be connected. It's integrated with PnP/Windows update so applications do not know the device is redirected.
The virtualized GPU supports Direct-X 9 and GDI. No support for DX10 or higher, in this release. Most applications can use DX9, so this is not a big limitation.
The physical video card must support DX10, since some new APIs are used for more efficient encoding.
Although the GPU is used for much of the encoding work, the server CPU still has some processing load. Hardware manufactures are working on a RemoteFX ASIC which offloads this processing. The ASIC is totally optional, and some should ship mid 2011.
Previous incarnations of RDS used a kernel-mode architecture. With RemoteFX and RDS, there are now user-mode and kernel-mode components.
For VDI, the hardware requirements include a CPU with SLAT (second-level address translation), GPU installed in the server, and a Windows 7 SP1 VM. With this the benefits include GPU virtualization, USB redirection, full Aero glass support, RemoteFX compression using CPU and GPU, and you can offload client hardware decompression to an ASIC.
RemoteFX is NOT another protocol, RemoteFX is built into RDP 7.1 and does not require additional ports to be opened up, or protocols to be allowed. It's simply an RDP extension. As such it supports all RDP features such as security (SSL, Kerberos, etc.) and virtual channel multiplexing
RemoteFX enables a new class of ultra lightweight and low-power devices. Operating systems could include Windows CE, Linux and custom OS like Wyse ThinOS. The devices can draw less than 5W of power and could have a dedicated ASIC for CODEC acceleration.
The limitation of how many RemoteFX enabled virtual desktops per server comes down to video card memory and screen resolution. For example, a 2GB ATI V7800 card could support up to 11 VMs running a single 1600x1200 screen. All of the VDI VMs on a server do not need to be RemoteFX enabled, so you can assign power users to RemoteFX VMs, and max out the server with non-RemoteFX VMs to increase VDI density.

Now one of the 'tricks' to implementing RemoteFX is putting a very beefy video card into your VDI servers. If you are using blade servers, this really restricts your options. For example, with HP you can only use their "workstation" blade PC with a side-car PCIe expansion slide, which cuts the server density in a 10U chassis from 16 servers down to 8. Given high heat load and the number of PCIe lanes required, blade servers are at a distinct disadvantage for the time being. One solution would be a traditional 1U rack-mount server, a heavy-duty graphics card with 4GB of RAM, and a single 10Gb CNA so you reduce your cabling requirements.

Since the primary scalability factor for RemoteFX is the amount of video RAM on the graphics card, I suspect AMD and nVidia will come out with RemoteFX oriented cards that increase the memory to 8GB or more. For example, a 8GB video card should be able to support over 35 RemoteFX sessions using 1920x1200 screens (typical 24" monitor).

In most cases not all users need RemoteFX. Task workers such as those using Office products or basic web surfing really won't benefit that much. But for users that require 3D applications, video playback, or USB redirection, RemoteFX is something to consider. Citrix has stated they are working on incorporating RemoteFX into XenDesktop, and that integration should be completed later this year.

Finally, not all graphics cards are certified for RemoteFX. Microsoft said the initial drivers AMD and nVidia provided were pretty buggy. So be sure to check out the Microsoft HCL for graphics cards and ensure it has been certified for RemoteFX. Also, it was stressed the RemoteFX was designed mostly for LAN based environments. If you have remote offices/branch offices, then RemoteFX may not be a good fit. Future versions may address bandwidth/latency issues, which are the primary reasons why this first version is intended for LAN usage.

VIR315: Modeling and Maintaining Virtualized Services in VMM 2012

2011-05-22T11:28:00.000-07:00

This session focused on the brand new services template module in VMM 2012. Service templates are a method to rapidly, consistently deploy, and maintain applications regardless of what hypervisor (Hyper-V, ESX or XenServer) that you use. So this entire session applies to customers even if you are a VMware shop, or a mixed environment.

Session highlights include:

A services template is the starting point for services and source of truth. It specifies machine and connectivity requirements. Deployed services are always linked to their templates. This enables servicing of instances, not just individual VMs.
An instance is a group of VMs working together, it includes machine specific definitions as well as applications. Native application support for web applications (WebDeploy), virtual applications (server App-V package), and database applications (SQL DAC). Future versions of VMM will likely support more types of applications (.e.g. Exchange, etc.).
Why use service templates? You can manage multi-tier applications across multiple servers as a single unit. This allows you to scale out on demand. It also allows you to manage fewer OS images, since you can customize the OS deployment at provisioning time.
The lifecycle includes creating a template, customize the deployment, deploy a service, then update template and apply to a service.
The service templates are authored in a new Service Designer. This defines the tiers, VM hardware, logical networks, OS, applications, load balancer configuration, etc.
The Service Designer is a really slick application with a ribbon interface that let's you graphically construct your application, link to networks, define instance counts, deployment order, upgrade domains, and servicing order.
A customized deployment of a template lets you define OS settings (computer name, admin password, etc.), application settings (SQL connection string, service account names, passwords, etc.), and lets you use the same template in various environments (dev, staging, production, etc.).
The deployment has several integration points where you can inject scripts to even further customize the deployment, which run in the guest OS.
There are two update types. The first is a regular update where the template changes are applied without replacing the OS image. Changes can include increasing memory, application updates, etc. The second method is an image based update. This replaces an old OS image with a new OS image. This re-installs the application, while preserving state. For example, you could migrate your webapp from Server 2008 to Server 2008 R2, with little to no downtime.
Regular updating and image updating both have extensive integration points where you can inject custom scripts to change the update process.
Service templates can be imported and exported and use a XML file. This lets you share templates between different environments, backup templates, or synchronize them in a multi-VMM environment. Like a GPO import, you can map resources to the template during the import process. For example, you can map network names or storage tier levels, even if they don't have the same name in the different environments. Very slick!

What was really slick about this session was the demos. While all of the information above is great, you could easily gloss over the potential impact of VMM. One compelling demo was updating a web app.

In the demo the original template was defined with v1.0 of the app, and used a scale out approach with a hardware load balancer. This created multiple instances of the web app, in a HA (high-availability) configuration (let's say four web servers). Now let's say you have v2.0 of the application you want to roll out but maintain HA. You open the service template and update the application to v2.0. Nothing has happened in production, since you've only updated the template. You now click a button to deploy v2.0 of the template. The template has defined service domains, which in this case, takes down two of the four web servers, pulls them out of the HLB, updates the web app, then adds them back into the HLB. After the first two are done, it takes down the next two, using the same process. 100% automated and no downtime!

Now let's say v2.0 of the WebApp has some bugs, and you need to revert back to v1.0 until you can fix the issues. No problem! Click on the service template, and you see the full version history. With a single click you revert to v1.0, and it repeats the same deployment process of removing the VMs from the HLB, down-rev the app, and add them back to the HLB. Completely automated roll-back, without service interruption.

And remember...service templates are hypervisor agnostic, so VMware shops get all of these service template features. Since this is an automated process, it minimizes human errors, limits configuration drift, and orchestrates the updates in a HA manner. What will be really exciting is when more products are supported, such as Exchange, so you can roll out service updates in a similarly automated fashion. Very slick indeed.

Service Templates support Windows Server 2003 R2 SP2 and later VMs, and SQL 2008 R2 (not SQL 2008 since it's not sysprep aware). As Microsoft gets onboard with Server App-V, those applications will be fully supported as well.

SIM209: System Center Service Manager: Automating ITIL and MOF

2011-05-22T10:16:00.000-07:00

This session covered how System Center Service Manager 2010 really integrates many of the System Center components to allow an organization to follow industry best practices, such as ITIL and MOF. Without an integrated solution, it's really hard automate these processes to reduce operational expenses and meet defined SLAs or customer expectations. Later in the week there was a Service Manager 2012 sneak peak session, which showed even more integration points such as VMM 2012 and Orchestrator 2012. I'll cover that session in another blog post.

Highlights of this session include:

Service Manager is the power of integration. It brings together Operations Manager, Configuration Manager, Active Directory, and Opalis (renamed Orchestrator). On top of these products it layers a single configuration management DB (CMDB), workflows, a portal, data warehouse, and forms.
SM supports many processes, but not the entire stack found in ITIL or MOF. The processes it does support include risk management, compliance management, service asset & configuration management, change management, knowledge management, incident management, problem management, event management, request fulfillment, and service level management.
First up is the Configuration Management process. The SM connector framework to AD, SCCM, SCOM allows automatic identification of configuration items (CI) in the environment. Regular synchronization makes sure the data (CMDB) is up to date. An audit trail is maintained for each CI.
Second is up is Incident Management. Incidents are 'daily fires' that the service desk puts out each day. Incidents are not part of the standard operation and need to be addressed as quickly as possible. Incidents can be opened automatically by SCOM, from DCM (Desired Configuration Manager) non-compliance, email from an end user, phone call, or the web portal. You can categorize incidents, assign an impact and urgency. Impact + urgency = priority. You can configure standard templates. Built-in links in the CMDB show related items. It also supports knowledge articles (provided by MS or custom entries), incident tasks, and capture of resolution information.
Third up is Problem Management. Problem management deals with resolving the underlying cause of one or more incidents. The focus is to resolve root cause errors and find permanent solutions. SM can log problems against related CIs, and you can manually create problem records, or they can be automatically created. You can categorize and set priorities through various data fields, and see the impacted CIs. You can define and relate known errors, and define and relate knowledge base articles within SM. Integration with the incident management engine can close incidents when they are resolved, and integration with the change management module ensures that proper change processes are followed.
Fourth up is Change Management. Change management records changes in the environment, affected services and computers, authorization to proceed, captures planning work, coordination of change implementation, and review of change completion. Details such as title and description are captures, including related CI items, and you can define change templates. CM contains activities, and you can review their current state. Planning changes are captured as well. Fields can capture scheduling details of the change.

The speaker went through a number of demonstrations of SM, and all of the integration modules. I thought it did a good job of showing the power of the integrated CMDB and how it can help you streamline and automate your processes. However, SM won't magically create your processes. So if your organization has poor processes to start with, you won't get as much out of this tool. You really need to clearly define processes, socialize them to all staff, then enforce the processes. That's when the power of SM can really be seen. For a v1.0 product, I think MS took a good stab at the problem set. The enhancements to SM 2012 close some of the gaps in 2010, and make the product even better.

SIM307: Securing your Windows Platform

2011-05-22T08:14:00.000-07:00

This session covered a number of free Microsoft tools that can be used to secure the Windows operating system, and to a lesser extent, applications. For the most part they presented tools that I was familiar with, and even have written a blog about (such as EMET). However, I did learn about a new tool that I think is pretty slick that your IA/security guys might really like.

The session started off with a brief background on security, then went into the specific tools and a few demos. Highlights of this session included:

Active Directory compromise is BAD. 100% cleanup assurance is extremely difficult, if not nearly impossible. Rebuild is expensive and embarrassing.
Malware is a profit driven industry and assume attackers are well funded and highly motivated. These are not just script kiddies trying to compromise a PC for fun.
Sophisticated techniques are getting more efficient, obfuscation techniques are constantly evolving, and the number of malware variants exceeded 286 million in 2010.
Attackers want to gain a beachhead, install malware, escalate their privileges, introduce redundant access into your environment, and exfiltrate data or other nefarious actions.
They then presented a graphic showing the cost of defending a network and the return on benefit. For most organizations the optimal point is 'commercial reasonability', after which costs dramatically increase with diminishing security returns.
If you aren't even doing due diligence then you are really up the creek. This includes limiting domain admin privileges, limiting local administrator access, don't allow internet browsing from administrative workstations, run 64-bit clients, patch, anti-virus software, and a firewall. In addition you should require two-factor authentication for administrators.
A concept they introduced is the "trusted virtual machine client." This is a client that is highly hardened and is what admins use to administrate the domain. Goals of this VM include lowing risks, prevent malware infections, limiting damage should the VM become compromised, and easy to use.
This trusted admin VM should run Windows 7 x64, be joined to the domain, member of a hardened workstation OU, use the SSLF security profile, and NOT have browser access to the internet. Normal users should never login to these admin workstations. Only regular users should login to regular workstations. No server or domain admins allowed on regular workstations.
You should have a concept of server admins, which is NOT a domain admin, and does NOT login to any clients. The account can only logon to authorized SERVERS.
Next up is the first security tool, Security Compliance Manager (SCM). SCM lets you configure a security GPO baseline, maintain version control, then export to a GPO to use in your domain. Microsoft provides many baselines that you can copy and modify to fit your security requirements. It also has a lot of built-in knowledge to help you understand what the settings do. It can also work with SCCM's DCM (desired configuration manager).
Second up is EMET, the Enhanced Mitigation Experience Toolkit. A new version of EMET (v2.1) was just released a few days ago, that you can download here. EMET protects against unknown vulnerabilities, blocks entire classes of exploits, and is easy to install. In just the last year EMET mitigated several zero day Adobe and IE vulnerabilities..all before Adobe and Microsoft released patches. Unfortunately for enterprises there is no centralized control or native reporting. Enterprise enhancements are in the works, but no ETA.
Applocker is a new feature in Windows 7 and Server 2008 R2 which can let you easily create whitelist and blacklists of applications. Unlike SRP (software restriction policies) in previous generations of OSes, Applocker is easy to configure, can automatically create rules, and is far more flexible.
The last tool, which was new to me, is Attack Surface Analyzer (ASA). ASA identifies the changes in system state, runtime parameters, and securable objects in Windows. It's part of Microsoft's internal SDL (secure development lifecycle) process. Basically you execute the tool on a computer, and it will report any insecure findings such as weak ACLs on objects. You can also schedule snapshots of systems and do a historical comparison. You can download a beta here.

ASA is a great tool for analyzing golden images before you deploy them, or hosts in high risk environments like DMZs. It analyzes far more than just filesystem or registry ACLs, such as COM+ objects, named pipes, GAC assemblies, network shares, threads, handles, ports, and other deeply buried Windows features that you can't possibly analyze manually. Microsoft uses it on 100% of all shipping products and any severity 1 findings prevent a product from shipping without a senior VP within MS granting a waiver.

Typical use scenario would be to run the tool on a virgin OS image (without any apps), then install all of your apps, then re-run the tool and look for any insecure settings that your applications created. You will probably see some false positives for weak ACLs involving the TrustedInstaller account. You can ignore those. Microsoft wanted to be transparent and not hide these findings, although I do think a check box to suppress the findings would be useful.

Fully supported platforms include Windows 7 and Server 2008 R2. You can do command line analysis and collection of Windows Vista and Server 2008 systems. Windows 8 and Server 2012 will require a new version of the tool.

Security is not just a matter of applying the right GPOs, installing anti-virus software, running a few tools, and calling it a day. Threats are constantly evolving, and policies and procedures are extremely important. User education is also critical, and often over looked. However, the tools covered in this session are a great start for hardening your base operating system.

SIM354: Systems Center Operations Manager 2012 Network Monitoring

2011-05-19T11:35:00.000-07:00

This session focused on the new (and pretty robust IMHO) networking monitoring enhancements in OpsMgr 2012. Previously network monitoring in OpsMgr was very, very basic. So basic, that I suspect not many people really used OpsMgr to monitor network devices. That all changes in OpsMgr 2012, where the OpsMgr team developed their own SNMP (v1, v2, v3) discovery engine, and are working with network vendors to have an extensive list of certified devices. The most in-depth monitoring is for Cisco devices.

Highlights of this session include:

OpsMgr 2012 supports network discovery, network monitoring, visualization, and reporting.
Key takeaway is that IT operations can now gain visibility into the network to reduce mean time to resolution.
Out of the box it will include multi-vendor support (Cisco, Foundry, etc.), supports IPv4 and IPv6, and partners can also build on the platform to further extend the feature set.
Network discovery finds things such as connectivity, VLAN membership, HSRP groups, server NIC discovery, port/interface details, processor details, memory.
Network discovery can be explicit, or recursive (using ARP, IP topology, MIB).
Discovery can run on demand, or on a scheduled basis. Some discoveries can be initiated by device traps.
Network monitoring stats include up/down, volume of inbound/outbound traffic, % utilization, drop and broadcast rates, processor % utilization, in-depth memory counters for Cisco (including fragmentation), and free memory.
You can monitor connection health (looks at both ends of the connection), VLAN health (based on switch status), and HSRP groups.
Built-in are a number of dashboards including network summary, network node details, network interface details, and vicinity views.
Built-in reports include memory utilization, processor utilization, port traffic volume, port error analysis, and port packet analysis.
OpsMgr network monitoring is not meant to replace network engineer monitoring tools. Although, I think it is idealy suited for service desk level 1/2 to monitor the network. Combined with SharePoint or Visio dashboards, you could do some really nifty real-time dashboards for a variety of groups in your organization.
No MIB import support. Microsoft will certify devices and release device updates in cumulative updates and service packs. CUs are typically on a quarterly basis. They are working on a process for customers to request specific devices to get certified.
No current support for NetFlow stats. It relies on snmp gets for performance counters. Netflow may be added in the future.
One OpsMgr network management server can probably support approximately 500 devices. More testing will be performed and MS will come out with performance details closer to RTM.
For 2,500 devices you can expect 15GB of additional storage required in your OpsMgr operational database, and about 100GB in your data warehouse for 1yr of data.
No Fibre Channel or SAN switch support. May be added in future releases.
OpsMgr only needs read-only SNMP access, NOT read/write.
Currently there is no event correlation between a switch going down and the servers connected to it. All objects will alert in OpsMgr. Future releases may have a correlation engine to suppress alerts.

In short, even though Microsoft claims to do just basic network monitoring and this is a v1.0 module, I think it's pretty darn full featured. Given their committment to support additional devices in their quartlerly CUs, that should help keep the module relevant and supporting the latest and greatest hardware.

For service desks, the out of the box functionality with a few custom dashaboards has the potential to eliminate the need for tools such as What's up Gold or SolarWinds. Of course network engineers still need their heavy duty tools, but seriously consider the cool Visio/SharePoint/Dashboard features of OpsMgr combined with network monitoring.

VIR305: Creating App-V Packages more efficiently with App-V 4.6 SP1 Sequencer

2011-05-18T18:44:00.000-07:00

Application virtualization is one of the "new" areas of virtualization I'm very excited about. App-V has been around for several years, in various forms, formerly known as SoftGrid. Server App-V is just around the corner, and I think more exciting than client virtualization, but that's a topic for another blog post. App-V basically wraps up an application into a self-contained package, which you can deploy via various means.

Unlike a traditional software package, the application is not directly installed into the client. There's a level of abstraction between the software and underlying operating system. This let's you, for example, have multiple versions of Microsoft Office or Java, on your computer without conflicts. App-V is one example of SaaS (Software as a Service). Other benefits include centralized servicing, centralized patching, tigh version conrol, and better software metering. You may of also heard of VMware ThinApp, which is similar in concept.

This session covers the enhancements made in SP1 of App-V 4.6. These enhancements include:

No 1990s era 8.3 file name restriction for directory paths. Yipppee!
New sequence diagnostics now proactively warn you of potential problems with a software package before you get to the end, start testing, and find out something is broke. Issues like having anti-virus running, software that includes device drivers, or other issues that make application virtualization harder.
An XML report is generated with each package showing all of the diagnostic data, so if you do run into issues, you have documentation about the packaging process to help you troubleshoot the issue. Report includes excluded files, drivers, COM+ objects, system differences, SxS conflicts and shell extensions.
Diagnostic alerts include pending reboots, VM not reverted, services enabled like defender or SMS, etc.
Dynamic Suite Composition (DSC) allows you to more flexibly package several applications or components (such as plug-ins or middleware) so the suite of software works properly. Office plug-ins are very common.
The major news about SP1 are package accelerators. Previously everyone that wanted to package up an application, say Office 2010, had to go through a somewhat lengthy and tedious process. No more! With package accelerators you point App-V to the install files, the accelerator files, click a few times, and viola, out the other end is a sequenced application.
Depending on the application, there may be a little more work required to use the accelerator, but it still eliminates a vast majority of the trial and error associated with sequencing. Microsoft, third party vendors, and the community can create and release accelerators. Ones for Office 2010 and some Adobe products already exist.
Project templates let you pre-populate sequencer GUI settings, or access settings that you can't via command line automation.
New CLI package optimization features let you launch all short cuts, control timeouts, and other features.

App-V 4.6 SP1 is already out, released back in March 2011. So you don't have to wait to use the new time saving features such as package accelerators. You can download a lot of them from here. If you haven't looked at application virtualization, you really should. Physically installing applications on clients these days is so yesterday. LOL. Now not all apps can be virtualized, but many, many can.

FYI, SCCM 2012 is tightly coupled with App-V, and has great in-depth support. So if you are a Microsoft shop, have SCCM, then you really need to look at App-V. Other application virtualization platforms just won't have the level of integration that you may want. See my SCCM 2012 posts for more details. Even if you don't use SCCM, App-V still can provide you some significant benefits. VDI environments almost require App-V, if you want to follow best practices and maintain very clean/minimal base images.

SIM361: System Center VMM 2012: OSD, OOB and Agent Management

2011-05-18T18:07:00.000-07:00

So this is the second part of SIM361, which actually covers the content in the official session title. The last half of the session focuses on the bare metal deployment and automatic cluster creation process for Hyper-V. So if you are a VMware or XenServer, you can skip the rest of the content. vSphere 5.0 will have a bare metal deployment appliance, so go read up on that. :) But for you Hyper-V users, keep reading.

Hyper-V host lifecycle management features include:

Full control of the bare metal using baseboard management controller (BMC) (e.g. DRAC, iLO, etc.). Supports discovery of basic hardware inventory such as SMBIOS GUID, model, asset tag, serial number, etc.). Also can control power states such as power on and power off.
Supports IPMI, DCMI, and WS-MAN interfaces to BMC devices. This interface is extensible.
Provision Hyper-V onto a bare metal machine.
Fully automated hyper-v cluster creation.
Leverages a VMM server, WDS server, and a library server. Co-exists very well with an existing WDS server (but requires Server 2008 R2 WDS). Dynamic driver injection support as well.
Deploys a VHD to the bare metal, meaning the server permanently boots off the VHD not a traditional disk partition.
Automates IPing, domain join, role/feature installation, computer naming, etc.

So there you have it.....deploy Hyper-V hosts directly from VMM 2012 with just a few clicks of a mouse.

SIM361: System Center VMM 2012: VMware and XenServer Support Features

2011-05-18T17:42:00.000-07:00

I've broken this SCVMM 2012 session into two blog posts, since half of the content wasn't directly related to Microsoft's session title so you might overlook this great info about VMware and XenServer support in VMM 2012.

It covered the enhanced integration of VMware and XenServer within VMM 2012 and the changes from the previous version (SCVMM 2008 R2 SP1). New to SCVMM 2012 is full support for ESX 4.1 and XenServer. The previous version has so little VMware support, that IMHO, it was practically useless. In fact the speaker asked the audience who uses VMM to manage their VMware environment and no one raised their hand. Ouch! No previous XenServer support.

VMM 2012 has a virtualization abstraction layer that allows VMM to use a common interface, yet interface with various hypervisors. For example, the same powershell command to live migrate a VM will work on Hyper-V, ESX 4.1, or XenServer. It is very likely datacenters will have more than one hypervisor, so this is a nice common point of administration for supported operations. VMM 2012 now has over 460 powershell commandlets, up from 160 in the previous release.

VMware support enhancements include:

Import ESX hosts and clusters and put them into any folder structure in VMM 2012, unlike previous versions that did a one time static import of your datacenter object tree.
Discovers standard and distributed port groups, and virtual switches.
VM templates are also discovered, and it imports the metadata about the VM template but does not touch nor ever delete the VM template (unlike previous versions).
VM workflow now uses vCenter to do the VM copy (which means it could leverage VAAI).
Thin provisioned VM templates are supported.
You can't create FT VMs (but who really uses those anyway given all of the limitations).
vMotion and Storage vMotion are supported.
Utilizes the native VMware HTTPS interface to vCenter (seems like a no brainer but the previous version did NOT.)
No requirement to enable root SSH on ESX servers (seriously, what was MS thinking when they required this?)

XenServer support includes:

No dependency on XenCenter. VMM 2012 directly talks to each XenServer.
Like VMware, you configure the XenServer host outside of VMM. When it's fully configured, then you add it to VMM.
It supports standalone and pooled hosts.
Can enable maintenance mode (like it can with VMware), and shutdown, power on, and restart the server.
Supports iSCSI, NFS, HBA, and StorageLink disk types, both shared and local.
Supports ISO repositories, although they must be read-write (not read-only).
Due to differences in how vSwitches work, VMM wrapps a single vSwitch around all of the XenServer vSwitches that get created when you use multiple VLANs.
Full VM support, both paravirtual and hardware VMs. Does leverage checkpoints (snapshots) and gues console access.
No Dynamic memory support.

It appears to me there are very few caveats regarding features not supporting in both hypervisors. That's not to say VMM will replace vCenter. It was clearly stated it will not, and vCenter will still be used to manage your ESX servers. However, many of the very common daily tasks that you do in vCenter can now be done in VMM 2012. There's no tie into VUM, for example, so any host patching and maintenance will still be vCenter only. But for the purposes of building and managing a private cloud, the level of support goes very deep. Bravo VMM team (just please rename it to something like Cloud Manager).

Now the million dollar question is, how soon will MS support vSphere 5.0 when it's GA?

SIM336: System Center VMM 2012: Configure Networking and Storage

2011-05-18T17:10:00.000-07:00

This session is the first installment of a three part series on how to configure your private cloud with VMM 2012. Clearly, this session covers networking and storage, two fundamental pieces of your cloud fabric. I was impressed with the depth of the integration of networking and storage. More amazingly, it's hypervisor agnostic for most all features so every vendor including arch rival VMware are treated fairly.

Tidbits from the session include:

Ability to define logical networks using VLANs and subnets per datacenter location. For example, you can configure separate pools for Boston, LA, DC, and London. When you deploy a VM to a location, it automatically uses the right pool and only presents to you the logical pools you can use. You can't accidentally assign a VM in Boston a London IP.
Address management for static IPs, load balancer VIPs, and MAC addresses (both VMware MAC address range and general MAC address range). VMM uses a check-in and check-out mechanism for static IPs. No more needing spreadsheets to keep track of your IPs. Select the proper pool, and it will use the next unused address. Delete the VM? That IP goes back into the pool. 100% automated static IP assignments. Sweet! Same thing for MAC addresses and HLB VIPs.
Automated provisioning of F5, Citrix NetScaler and Brocade (at RTM) hardware load balancers. F5 and Citrix both now have virtual LB appliances, BTW. The lack of Cisco support is a bit surprising.
You can define HLB VIP templates that define properties such as protocol, LB method, persistence, and health monitors. You assign a HLB to a site, so when you deploy an application it automatically uses the proper physical HLB, and check-out the proper IPs.
Storage component discovers storage arrays and pools of storage, let's you classify storage based on capabilities you dictate (throughput, availability, etc.), discover and configure LUNs and assign to Hyper-V hosts and clusters. You could have platinum storage, gold storage, silver storage, or bronze storage (or whatever names you want).
Storage capabilities include end-to-end storage device mapping, allocation and assignment of storage, provisioning a VM using SAN array hardware copy capabilities, and storage migration of a VM (.e.g. storage vMotion).
End-to-end mapping is truly end-to-end: service instance to VMs, to logical disks in the guest, to the guest volumes, to the physical logical disk, to the LUN, to the array disk pool, to the disk array, to the array provider. This information is fed to SCOM for event/performance correlation (very sweet).
Uses a standards based approach for discovery, SMI-S v1.4. Many vendors are working on providers if they don't already have them.
Supported storage types include Fibre Channel, iSCSI, and local storage. (Not sure about FCoE but I think it is supported.)
Supports configuring iSCSI masking/unmasking, and initiator logon/logoff parameters.
Supports Fibre Channel masking/unmasking, and NPIV vPort deletion/creation.
This is NOT a storage management tool, so you won't use VMM to create an entirely new LUN on your storage array. You will continue to use your array's tools. Likewise for the network, this will not create VLANs in your network, but will consume them. (Although you could use Orchestrator 2012 to automate the creation in the array/switch, then have VMM discover it.)

The speaker when through many demos, showing how VMM 2012 discovers networking and storage assets and brings them into the fabric. For VMware it supports both standard vSwitches and virtual distributed switches (didn't get clarification on the Nexus 1000v, but I'd hope that is covered by their vDS support). I was very impressed by the capabilities, and the attention to detail. As the speaker mentioned, MS really should have renamed VMM to something else, like Cloud Manager. It's easy to think VMM is just for managing VMs, and in the 2012 case that is so not the case.

SIM207: Systems Center Orchestrator 2012: Overview

2011-05-17T17:51:00.000-07:00

What is SC Orchestrator 2012? It's the new name for Opalis. What's Opalis you ask? It's a Microsoft acquisition made about 1.5 years ago to automate IT processes. Opalis brings integration, orchestration and automation to your IT silos (network, server, storage, security, service desk, etc.). It works with a huge variety of MS and non-Microsoft products and comes with a number of integration packs (IPs) and you can easily extend it yourself. CodePlex also has a community of IPs that you can draw from and find ones that MS doesn't officially support.

So what's new to Orchestrator 2012? Let's take a look:

Major investments in operator support (triggers, monitoring and troubleshooting). Deeper developer application integration, and more reporting features.
Radically simplified installer and out of the box support for Windows Server 2008 R2, SQL 2008 R2. The RunBook designer will run on Windows 7.
Java/JBOSS interface is now history, replaced by IIS and Silverlight.
Globalizing to run on non-EN_US OS, but still not localized.
Will feature a SCOM management pack.
Enhanced vSphere integration packs, as well as Active Directory, FTP and Tivoli Netcool Omnibus.
Like the rest of the Systems Center suite, RTM is expected 2H CY11.
Still just a 32-bit application, unfortunately.

The speaker did a ton of demos, so that's why I'm a bit light on the content. Frankly, to date I haven't had much interest in Opalis as we have some fundamental challenges that need to be dealt with first. But, I am very glad to see the 2012 release more "Microsoftized". Definitely worth checking out if you haven't already.

WCL307: How to Develop a Succesful Desktop Strategty

2011-05-17T17:30:00.000-07:00

This session was also very good, but a higher level session that didn't discuss any particular technical solutions. In fact the speaker said he's solution agnostic, as any given company has many scenarios and there's no one size fits all solution. The speaker is a Microsoft strategist, who talks to 6-8 different customers a week all over the world and also does extensive market research and reading. So the information he presented was not his personal view, but a view shaped by decades of customer interaction and industry analysts.

The basics of the session were:

The desktop is evolving from the concept of a physical PC located in a single office, to anywhere, anytime, any device access. This increases exposure of your data, for better or for worse.
User complaints are historically have been solved by throwing new device solutions at them (faster hardware, newer hardware, etc.). Complaints include slowness, app failure, complicated to manage, complicated to maintain, hard to update, etc. Performance, IT control and user satisfaction all deteriorate as time goes by.
The road map to nirvana which once was pretty clear, is now not so clear. VDI, tablets, mobile devices, green, thin computing, cloud computing, compliance, etc. These requirements compete and some are mutually exclusive.
For 20 years people immediately jump to a hardware solution to fix user problems. Throw new gadgets at users hoping they are happy. But what really needs to happen is an analysis of roles/functions, how will you enable that use case, combined with where.
We need to move from a desktop strategy to a flexible work strategy.
In 10 years (1999-2010), thin clients have only increased market share from 0.6% to 1%. VDI only has 1.5%. Clearly, this solution has not yet taken off like wild fire.
According to Gartner thin clients have almost no TCO advantage over a well managed fat client.
Virtualized applications can reduce desktop TCO by 5% to 7%, and reduce testing, packaging and app support by 60%.
Each architecture (VDI, thin client, fat client, phone, etc.) requires different expertise and has unique infrastructure requirements. Some like VDI are very demanding on storage performance, while others may require increased network bandwidth, or perimeter security.
Cloud based computing has a major impact on the datacenter. HA, storage, networking, configuration management and application lifecycle become critical to properly manage. Processes need to be rethought.
Any particular architecture will NOT cure your problems, it will make it harder. The biggest factor to reduce TCO is based on how managed are your profiles as unlocked users can cost up to 36% more.
VDI/thin clients are not appropriate for everyone. You must select the right tool for the right job.
The level of trust for a particular asset (phones, slates, tablets, netbooks, laptops, etc.) determines the access level. The more trusted, the wider access it has to corporate data.
To build a well managed desktop environment you need Control + Management + Security.
Application virtualization & streaming + data compliance and user personalization + image certification + user environment isolation + recovery and disaster readiness = more efficiency, lower desktop support costs.
Organization segmentation, silos of roles, processes and responsibilities are the biggest reasons stopping people from achieving this end.
The speaker then presented some very detailed slides on basic, standardized, rationalized and dynamic IT core infrastructure models. With this slides you could go down through a number of categories and determine your current state (.e.g. manual app deployment is basic, while self-service role based app install is dynamic) and pick your target end state. Very enlightening slides.
Step 1 of this entire process is to baseline your current capabilities. You have to know where you are at today, so you can develop a roadmap. Step 2 is to define an end point. Put users into different buckets (office workers, task workers, etc.). Step 3 is build a plan and execute with all joint virtual teams...across organizations and with full buy-in.
Foundation elements for a well managed desktop include compatibility analysis and migration, imaging and deployment strategy, deployment implementation and migration.

The speaker sited numerous sources for his data points, which were all included in the slides. I thought the session was pretty thought provoking, and makes you realize that must get out of the mind set of just throwing new hardware widgets at the problem. You need to fundamentally think about desktops in a different way today.

SIM352: Systems Center Configuration Manager 2012 Technical Overview

2011-05-17T09:59:00.000-07:00

Batting 100% today for killer sessions. This session was on SCCM 2012, which like VMM 2012, has undergone radical changes for the better. The speaker went over several demos and there was a ton of content, so I didn't catch everything. Plus, honestly, I've never been a big SCCM guy because it was such a complicated beast, hard to manage, and typically has lots of glitches. If you are a current SCCM user, or haven't used SCCM because of the reasons I just mentioned, take a hard look at SCCM 2012. Both under the covers and visually it's a radically different product designed to address these pain points, and address the era of desktop virtualization and mobile device computing.

Enhancements include:

Applications can now be targeted at users and delivered to users in different ways depending on the device they are on at the moment. You can configure multiple deployment options for the same software package (say Office 2010), so it can be automatically delivered via App-V, MSI installation, XenApp, or RDS, depending on the rules you configure. In essence you have roaming applications.
Fine grained administrative delegation, and if you can't access a particular object, it's hidden. Only a desktop administrator? You can't even see servers, server only packages, or deploy software to servers. No more accidentally having your desktop guys deploy Adobe Reader to your servers on accident.
All replication (packages, etc.) is now done via SQL replication. No more proprietary SMS/SCCM replication engine that always seemed to have problems. Big win!!
Agents can now auto repair themselves, and even re-install on their own. It can repair WMI, bits, and other components that commonly break and require manual fixing on clients. Huge!
You now deploy applications to a collection, and the client determines the deployment type (App-V, MSI install, etc.).
Web based app store that presents to the user the list of all applications they are authorized to use. You can also do a basic approval workflow (but no email notification, or routing of requests to a supervisor). Only routes requests to a SCCM admin.
Aware of pooled VDI desktops, such as XenDesktop, and each client maintains uniqueness in the SCCM console between reboots (when the VM is reset to a virgin state). Major feature for XenDesktop shops.
DCM (desired configuration manager) can now SET/remediate settings, not just monitor compliance. So you can enforce compliance for key security settings on clients. Console can alert you when a threshold of non-compliance is crossed.
Completely redesigned GUI with a ribbon interface. No more MMC or actions pane.
Global search within the GUI.
13 built-in RBAC roles, which you can copy, modify, or create your own.
Clients can now host distribution points.
Radically simplified topology that will reduce the need for additional sites.
"Light" and "Depth" management of phone devices, depending on the phone's operating system. iOS, Android, and Windows Phone 7 are all "light" at this time.

There was a lot more covered, but there's no way I can cover it all. In short, SCCM 2012 is a huge upgrade, easier to manage, and has a good migration story from previous versions. It's currently in beta 2 and on track to be released by the end of the calendar year. I'm really excited about this release, as it looks like the first version to me that has addressed many of the long standing issues with SMS/SCCM and is forward leaning into VDI management.

SIM211: Microsoft Systems Center Virtual Machine Manager 2012: Intro

2011-05-17T09:21:00.000-07:00

Wow! This was a really killer session on System Center VMM 2012. Frankly, since the shop I work in is 100% VMware for virtualization, I haven't seriously looked at VMM before. Yes I knew it could interface with VMware, but eh, I just didn't see much value in yet another product I need to learn and manage. VMM 2012 is really a game changer for private clouds, and I'm blown away by the capabilities. Unlike SCOM 2012 that seems like an incremental upgrade (personally I'd call it 2007 R3) , VMM 2012 is a radically different product in every way.

I talked to a MS guy yesterday, and all of the functionality in VMM 2012 will be available with VMware hypervisors minus the bare metal deployment of the hypervisor to the server. Same thing for XenServer as well. So get ready to build your private cloud on the hypervisor of your choice.

This was a high-level overview session, so here are some of the tidbits I learned:

VMM 2012 will allow you easily build private clouds based on Hyper-V, VMware vSphere, and XenServer. It abstracts the hypervisor, storage, compute, and networking resources and lets you build a scalable, elastic, self-service, and metered use private cloud.
Major investment areas include HA VMM server, deeper powershell integration, fabric management of hypervisors, networks, storage, power, and clusters. Delegation, quotas, application service templates, application deployment, image based servicing.
Network management includes defining VLANs, subnets, address management for static IPs, load balancer VIPs, and MAC addresses.
Automates provisioning of load balancers (F5, Citrix Netscaler, etc.). Adding a new web server to your SharePoint farm? No problem, VMM will automatically reconfigure your HLB.
Storage management using SMI-S v1.4. Discover disk arrays and pools, classifiy various storage types, discover and configure LUN, assign LUNs to clusters, configure LUN masking, and snap cloning of VMs.
Integration with WSUS for automated no downtime patching of Hyper-V hosts (like VMware VUM)
Dynamic optimization of VM workloads (like VMware DRS).
Power Optimization (like VMware DPM).
Enhanced VM placement via 100 checks/validations.
Abstracts all of these resources into objects that a cloud user can comsume (capacity, capabilities, libraries, logical networks, load balancers, storage classifications, storage capacity).
Fine-level delegation down to the user role, and ability to define quotas based on various resources like compute, memory, storage, and number of VMs.
You can shape VM limits by specifying ranges for VMs (processor range, memory range, disk range, NIC range, etc.). For example, you can limit the ability of a user to provision a VM with 1-2 vCPUs, upto 4GB of ram, and only one NIC.
Service templates to model multi-tier appplications, and automate their deployment. Support for Server App-V, SQL DAC, and web deploy. Have a multi-tiered application? Just model it, then click deploy, input some parameters, and voila..the VMs are created, configured, storage provisioned, network configured, and HLB setup.
Image-based composition separates OS from the apps, and the combination is composed during deployment. Drastically reduce the number of required VM templates. Limit the need for AD templates, SQL templates, web server templates, etc.
Change the template, then apply that change to deployed instances to ensure compliance. Roll-back changes as well. This ensures all production instances are based on your template and compliant.
Application owners can author service templates, then share that template with others to ensure it is deployed correctly. IT operations can now deploy multi-tiered applicatins in a consistent, tested, and very elastic manner.

The speaker also did a number of demos, and covers a lot more information as well. Needless to say, the capabilities in VMM 2012 are really amazing. I'm even more stoked that it's hypervisor independent, so most any shop can take advantage of all these private cloud features.

SIM355: Systems Center Operations Manager 2012: Overview

2011-05-16T18:17:00.000-07:00

If you are a SCOM user, you will have noticed that MS has been pretty mum about SCOM 2012. Last year at TechEd 2010 MS had sessions on configuration manager v.Next, but not a single one on SCOM v.Next. Well this year they finally have some SCOM 2012 sessions. First up was their overview session which hit a few key highlights of the next release. Areas of investment include:

Focus on how to reduce the mean time to resolve a problem, and expand IT's visibility into their environment.
Reduced TCO through a simpler topology (no more RMS server), reliable (no blackouts), and a consistent experience.
High-availability is now built in, and does not require MSCS. It features automatic fail over and what I would call load balancing.
Platform extensions and network device management are now handled by a pool of SCOM servers, not a single server, so an individual service outage will not result in the loss of management functionality.
Holistic view of application health: Adding native .Net, Java and robust network device monitoring.
Supports SNMPv3, SNMP community strings, and ICMP monitoring of network devices.
Simple and powerful visualizations. It's easy to create stunning dashboard views, personalize them, and consistently access them through the SCOM console, web browser, or SharePoint.
MPs can contain/define 'widgets' which are visualization options such as bar chart, line graph, etc.

Most of the session was demos, which were quite impressive. They really focused on network monitoring, and the level of detail you can drill down into is amazing. For example, for a Cisco switch you can go down to port level I/O stats, latency, switch CPU utilization, and see which servers are connected to which ports. It collects and displays historical data, and lets the server admin troubleshoot performance issues involving the network. A vicinity view lets you view network devices two hops away and maps them out, so you can spot performance issues upstream.

They also spent a lot of time on .Net monitoring with their AviCode acquisition, and their native Java monitoring. Both enhancements looked very impressive. Microsoft is targeting Q4 of 2011 for a major release for most all their Systems Center products. However, one of the MS guys I was talking to afterwards said SCOM 2012 really isn't fully baked, and the release would probably slip to Q1 of 2012.

It seems to me MS has spent a lot more dev dollars on SCCM, and has not invested as heavily in SCOM. SCCM 2012 is already in beta 2, yet SCOM 2012 is still in CTP. While SCCM 2012 has gotten the ribbon interface treatment, it didn't appear the early CTP release of SCOM got the same makeover. It's a shame MS can't get major release versions of sister products the same GUI make overs. For years SCOM was ahead of SCCM's MMC, and now SCCM will be ahead of SCOM with the ribbon. Why oh why can't they be on parity?

WCL324: Thin Clients, from Building to Deployment and Management

2011-05-16T13:32:00.000-07:00

You know, sometimes Microsoft drops the ball on some TechEd sessions. WCL324 is one that the ball got dropped on. While the presented content was useful, it in no way reflected the course title or description and to make it worse, only took 25 of the 75 minutes. So disregarding the course title, here are some of the tidbits I gleaned in the very short session:

Windows ThinPC is now GA, and is Windows 7 stripped down that lets you turn a legacy PC into a thin client. Great for re-using existing PCs as thin clients without buying new thin client hardware.
Windows Embedded Standard SP1 was released in March and supports RemoteFX.
Windows Embedded Standard 7 (WES7) is the full featured thin client operation system. You can also chose from Windows Embedded Compact, which is more stripped down. Or, yet to be released, is Windows Embedded Compact 7T which is even more stripped down and lower maintenance.
WES7 is for a rich desktop experience including multi-media redirection. Compact edition is for deskless workers like nurses or factory floor workers. The "T" edition is for task workers, like a call center operator that only requires very basic functionality.
WES7 supports security features like bitlocker and AppLocker.
SCCM 2007 and even 2012 are NOT Windows embedded aware and make for very poor management tools of WES. So Microsoft has a new product called Windows Embedded Device Manager (WEDM) 2011 that augments SCCM 2007/2012 with WES management tools. It is a separately licensed product, and is NOT built-in to SCCM 2012 (which is a big bummer. What is MS thinking?!?!)

Unfortunately that was all of the meat in this session. It was not a walk through of the lifecycle of a WES7 client in the enterprise as the description had indicated. WEDM 2011 was new to me, so I'm glad to see that MS now has a good embedded management story, but the bolt-on nature to the 2012 releases is really unforgivable given SCCM 2012's deep VDI integration.

SIM214: Client Management and Security Roadmap

2011-05-16T11:59:00.000-07:00

This was a great session covering Systems Center Configuration Manager 2012 and ForeFront EndPoint Protection (FEP) 2012, which are both in the beta phases. Both products have undergone major changes from their previous releases. The major highlights of this session are:

The products now integrate management, security, and compliance into a single pane of glass.
SCCM 2012 is now in beta 2, and will be publicly available shortly.
SCCM now has a full ribbon UI and deeply integrated search. You will not recognize the console if you are a SCCM 2007 user. It is now modern and much more usable.
SCCM brings a personalized app store to your IT users via a web site. It can deliver applications via App-V, XenApp, or many other deployment types. Workflow and approval are built-in, so users can only select apps they are authorized to use.
VDI is a first class citizen and unique deployment options for situations such as pooled XenDesktop VMs.
SCCM is aware of the desktop type (physical, virtual, pooled virtual, etc.) and you can easily create collections based on these desktop types.
SCCM has built-in randomization features so that you don't get VDI storms with software updates, A/V updates, reboots, etc.
You can easily setup exclusion rules for pooled VDI desktops, so that you don't deploy patches or software updates to them, since you update the master image, not the cloned VMs.
FEP 2012 is now in beta 1.
FEP 2012 is deeply integrated with SCCM 2012, and can now natively deploy A/V signature updates without the use of a WSUS server.
FEP 2012 A/V updates are also randomized and VDI aware, so don't create update storms.
End point compliance is now integrated into a single console so you can manage items such as the Windows firewall, IE settings, office security, A/V settings, etc.). No more multiple products or windows to ensure endpoint compliance.
For internet connected systems, there are now automatic A/V deployment rules so you can automatically push signature updates without manually creating or approving new packages.
You can easily report on malware infections on a per-user basis, not just per-device basis. So you can track WHO is impacted or targeted by malware infections, in addition to what device needs remediation.
Microsoft and Citrix have worked closely to properly manage XenDesktop/XenApp servers in conjunction with SCCM/FEP 2012. The native integration of VDI scenarios with both products is really impressive.

Bottom line, the 2012 wave of products is a major upgrade. If you are going down the VDI road, the deep integration and awareness of VDI unique problem sets is a big win. Things that you couldn't easily do to, or even do at all, are just a few clicks away. By the end of 2011 the entire Systems Center suite of products are due to RTM. If you are a current SCCM/FEP user, I would urge you to get the betas and start using them in a lab environment.

PRC13: Group Policy in Win 7/2008R2

2011-05-15T17:26:00.000-07:00

So this week I'm at once again making the trek to Microsoft TechEd 2011 in Atlanta, so I'll be posting lots of session summaries and information that I want to pass along. First up is an all day pre-conference session by Jeremy Moskowitz on Group Policy. Here are some of the tips and highlights:

You don't need Server 2008 R2 domain controllers to take advantage of Windows 7 or Server 2008 R2 group policy enhancements. You can use any version of domain controller, including Windows 2000. GPOs are client based technology, not server based.
The following policies are only applied at startup/login: software installation, folder redirection, disk quotas, drive mappings.
Use Windows 7 or Server 2008 R2 to manage all GPOs, even those for 2003 or XP. Do NOT use older GPMC versions to manage newer policies.
It is no longer recommended to tweak the "GPO status" (enabling or disabling user/computer sections). Just leave the entire GPO enabled.
Recommend you configure "Always wait for the network at computer startup and logon" policy located at Computer\Admin Templates\System\Logon for client operating systems. This forces synchronous GPO processing on clients, not the default of async. Provides a more consistent user experience.
The ADMX central store can work with any DC type (2000, 2003, etc.).
Comment your GPO policy settings and GPO to help document your settings.
Install the group policy preferences client for XP/Vista so you can use GP preferences. WSUS can deploy as well (optional software).
Group policy preferences can cover most of the tasks previously handled by logon scripts (mapping drives/printers, copying files to the client, configure shortcuts, etc.). Very powerful! You can also easily disable devices or device classes, such as CD-ROMs or USB sticks with a couple of clicks.
Group policy preferences are NOT a good place to store passwords, as you can reverse the encryption. So don't use GP preferences to set local account passwords. MS published the encryption, and all computers use the same key.
GP preferences extensions only for IE 5,6,7, and 8. MS hasn't released IE9 settings yet.
Use the F5, F6, F7, and F8 keys to 'red' and 'green' individual GP preference settings. All green settings get delivered to the client, so set to Red items that you don't want delivered.
Applocker service (Application Identity) takes 2 minutes to initialize after it starts, so don't immediately try your rules until it's fully started. Applocker is a VERY flexible way to limit what users can run on their computers. Way cool auto rule generation and other ways to make life easier for configuring rules.

Overall, it was a good session. Group policy preferences are really a powerful tool, and can be used on Windows XP and higher systems. If you aren't using GPPs, take a serious look at them.

XenDesktop USB Filtering the easy way!

2011-03-27T10:58:00.000-07:00

In XenDesktop 5.0 you can configure HDX policies to block or allow certain types of USB devices. For example, you could block flash drives but allow USB printers or webcams. Unfortunately, Citrix doesn't give you an easy to to discover class IDs, vendor IDs, or other identifiers that can be used in their policies. Citrix has a good article here on USB filtering in XenDesktop 5.0.

Instead of digging through the registry to discover this critical USB data, I found a great tool that makes it a snap. Nirsoft has a free USB viewer you can download here.

To create the appropriate rules I did the following process:

1. In Citrix Desktop Studio open the Users HDX policy and navigate to USB Devices\Client USB device redirection. Edit the policy and change the value to Allowed.

2. Using the Citrix receiver connect to a virtual desktop, then from the menu bar click on the USB button.

3. In my case I have a flash drive connected to my physical computer, so I selected that from the drop down menu. I then heard the Windows USB disconnect/connect sounds and saw my flash drive ready to use in the VM.

4. Download the USB viewer tool and run it inside the VM. In the list of USB devices, locate your connected device and double click on it. Here's what comes up for my USB stick:

5. Take note of the USB class ID and USB subClass IDs, as you will need these for the HDX rules.

6. Back in Citrix Desktop Studio open the Users HDX policy and navigate to USB Devices\Client USB device redirection rules. Edit the policy and create a new rule, for example:

7. Accept the rule, then log out of your virtual desktop then log back in. If you try and connect your thumb drive now, nothing happens. Unfortunately XD5 doesn't provide the user any feedback why you can't connect the device. It would be most useful if a warning popped up saying that device was administratively prohibited, so the user didn't call the help desk wondering why it wasn't working.

You can use the same basic procedure to build up allow or deny device lists as required. Some devices can be tricky, such as multi-function USB printers/scanners/fax machines. So a single composite device might need a few allow entries to make it properly function. But using USB device view, you can pretty easily figure out what you need to do.

Immediately Activate Windows 7 and Office 2010 in a VDI Environment

2011-03-20T16:20:00.000-07:00

Lately I've been working on a XenDesktop 5.0 Proof of Concept with Windows 7, and I wanted to make sure Windows 7 and Office 2010 were activated immediately when a VDI VM booted up. Normally after some period of time Windows 7 and Office 2010 will activate themselves, but this can take several minutes or longer. So I wanted a solution that would activate both products shortly after boot time, so I didn't have to worry about any activation messages.

In my scenario I'm using a KMS server with DNS SRV records, so Windows and Office can automatically find the KMS server on the network. Windows 7 has a nifty new task scheduler, so I thought I'd see if I could make a boot time task that activated Windows 7 and Office 2010. Sure enough, it was pretty easy and works like a charm.

Here's how to create the scheduled task:

1. Launch the Task Scheduler and find a good place to put your new task. I chose Microsoft\Windows\Windows Activation Technologies.

2. Create a new task and configure the General properties as shown below:

3. Configure the trigger settings as shown below:

4. Configure the actions as shown below (Note that I configured cscript as the 'program' and put the rest of the command line as the arguments):

5. I cleared all of the condition settings, as they were not relevant for this task.
6. Finally, I configured the following settings. These settings are not critical, so you can tweak them as needed.

At this point you now have a configured task the runs once right after a computer boots to activate Windows 7 and Office 2010, on both 32-bit and 64-bit platforms. I wouldn't run this on a physical computer, as activation is automatic and only is required every 180 days. But in a VDI environment where the VM's state is reset after every reboot, I like making sure it's immediately activated.

To validate that Office 2010 is activated you can run:

32-Bit: cscript "c:\Program Files\Microsoft Office\Office14\ospp.vbs" /dstatus
64-Bit: cscript "c:\Program Files (x86)\Microsoft Office\Office14\ospp.vbs" /dstatus

To validate Windows 7 is activated you can run:

cscript c:\windows\system32\slmgr.vbs /dli

P.S. Microsoft: Would it really be too much to ask that the Windows and Office teams collaborate on the command line switches for activation? Clearly there was no coordination as the switches are totally different between the products.

Building a Sandy Bridge ESX Server

2011-03-13T15:00:00.000-07:00

A few months ago I posted the parts list for sub-$1000 ESX server. At that time Intel had not released their Sandy Bridge CPUs or chipset. Now that manufacturers are shipping chipsets without the SATA bugs, I thought I'd post an updated list of components for a screaming home ESX server. You can buy everything you need except the CPU cooler from TigerDirect.

The same criteria apply to this server as my previous post: compact form factor (HTPC case), quiet, power efficient, and dual 1Gb NICs. The CPU has a built-in GPU, so no separate graphics card is required. If you already have external shared storage (like a QNAP), then you don't need an internal HD and could use the 8Gb tiny flash drive listed below to install ESXi on. The NICs on the motherboard are not supported by ESXi, which is why you need the SuperMicro card.

Intel also ships a lower power i5-2400 CPU (65w max vs 95w max), but it runs at a lower clock speed. Depending on your requirements, this may be a better option if your electricity bill is a concern. The WD hard drive is a screamer, but is a bit noisey. So if the noise will disturb you, going with their green line would be a much quieter option, but is slower.

Antec MicroATX Minuet 350 case $116
Asus P8H67-M Pro $130
Intel Core i5-2400 $200
Patriot 8GB DDR3 PC3-12800 2x $85
SuperMicro Dual GiGE NIC AOC-SG-I2 $86
Scythe Big Shuriken CPU Cooler $35

Total: $737 + Shipping

Optional parts:
Asus DVD Burner $20
LaCie 8Gb ultra-small USB flash drive $25
Western Digital 1TB 6Gbps Caviar Black SATA HD $85
Intel Core i5-2400S Low-power CPU $200

Update 1: Got all of the parts and the server works great. I would advise that you install the memory before you put on the CPU cooler, or you will find yourself taking the CPU cooler off. Also, if you don't get the memory I specified above, make sure the memory you get is low profile and doesn't have any heatsinks sticking up above the DIMM PCB. I really like the new ASUS EFI BIOS..very graphical, easy to use, and slick. I also decided to get a tiny 4GB USB memory stick from Best Buy for $15, located here.

Update 2: I did some power measurements, and idiling the server uses 47 watts, which is more than 20 watts less than my previous ESX server. The integrated graphics card and more power efficient CPU sure make a difference!

Update 3: vSphere 5.0 recognizes the onboard RealTek NIC. So if you are satisfied with a single NIC you can skip the SuperMicro card in my parts list. If you want the very latest motherboard the Asus P8Z68-M Pro works extremely well, and its onboard NIC is also recognized. The Z68 chipset has the advantage of overclocking the CPU while letting you use GPU equipped CPUs, so you don't need an add-in PCIe video card like you do with the H67 chipset.

Hashes to Ashes - Root your domain in seconds

2011-03-13T13:26:00.000-07:00

Protecting privileged passwords is always extremely important, and you may think just because you use a very long password that is highly complex that it may take a hacker days, weeks, or even years to break into your system. Or you may also think that giving regular users local admin rights on their workstations or laptops, while not ideal, really isn't THAT bad. After all, the user's elevated rights would not extend beyond their local computer, right? Or how about giving low-level help desk staff administrator rights on certain servers or computers in the domain? Fairly low risk, right?

Think again, as these scenarios may not seem overly dangerous to your average administrator or IT manager, but to a person with a couple of free tools, they could potentially 0wn your entire Windows forest in just a few short seconds...without having to decrypt long complex passwords, download large rainbow tables, or use large computing resources.

How? Very simple really...it's called passing the hash. This is not a new attack, and if you are a security professional then I'd sincerely hope you are aware of this attack vector because it's been around for a long time. But there are a couple of freely available tools that allow the attack to be accomplished in just a few seconds, even on the most modern Microsoft operating systems like Windows Server 2008 R2 SP1 and Windows 7 SP1.

To demonstrate this attack, I have two computers. The first is a Windows Server 2008 R2 SP1 domain controller (fully patched), called N001DC01. The second server is called N001Lync01, and is also Windows Server 2008 R2 SP1 (fully patched) and running Microsoft Lync Server 2010. Both have additional security lockdowns applied to them, including many of those required by the Federal Government.

In a typical corporate environment you would have a limited number of domain administrators, and then many delegated administrators with fewer rights. For this example, the sysadmin account is the domain admin, and on N001Lync01 there is a local administrator account called LocalAdmin. The domain admins would likely use their account to logon to most servers and clients in the environment to do their daily job such as fixing system problems, installing patches, etc. All accounts have long, strong passwords, that are 20 characters or longer. The domain name is contoso.net.

To completely 0wn the domain and elevate my rights from local administrator on one server to being a domain admin, I'll follow this quick, pain free, process:

1. I'll logon to N001Lync01 as the local administrator, localadmin. I'll do a whoami to verify I'm who I say I am.

2. Next, I'll try to map to the C drive on a domain controller, just to prove that I don't have permissions since I'm a local administrator only on N001Lync01.

3. Since I have local admin rights on N001Lync01, I'll dump the hashes stored in memory and see if there's anything useful. By George, yes, I do believe we have a winner. I can see that the password hash for the domain administrator account, contoso\sysadmin, is present and was captured.

4. Now that we have the domain admin hash (after just a few seconds of work), let's now use the hash to impersonate sysadmin and root the domain.

5. Ok, after that little trick a new command prompt opens up and once again check to see who I am. Ok, it looks like I'm still just the localadmin, but maybe I have covert super powers waiting to be used?

6. So let's see if I can access the domain controller, shall we? Yes sir, the command prompt has the hash of sysadmin injected, so when I do a directory listing of my domain controller it now returns the results I'm expecting.

7. Directory listings are a bit boring, how about we try and launch Active Directory Users and Computers and create a new user account? Bingo, success! Now we could also remotely launch a hash dumper on the domain controller and extract the password hashes for pretty much every account account in AD.

To recap, I was a lowly delegated administrator and only had local admin rights on a single server in the domain. But because more privileged accounts logged onto the system, I could dump those hashes and escalate my privileges to a full blow domain admin. At this point I could create back doors, add new accounts, impersonate other user accounts, or download practically anything off the network. Total time to 0wn could be less than 30 seconds.

A few lessons can be learned to help mitigate this attack vector:

1. Do not give regular users local administrator rights on their domain joined computer. It's pretty trivial for them to escalate their rights to full domain admin, given the right circumstances.

2. Limit lower level help desk staff to non-administrator roles on servers and clients. Don't think giving them admin rights on everything but domain controllers is safe or low-risk. They can be a domain admin in just a few seconds.

3. Only use your domain admin credentials to logon to domain controllers. Use another delegated administrator account to access member servers or client computers. This at least makes escalation a little bit harder. Remember, don't use privileged credentials on less trusted computers.

4. Limit service account rights to least privilege, and NOT domain admin. If the service account, say for auditing software, accesses every computer in the domain then any local admin on any computer can impersonate anyone else.

5. Always have anti-virus software on your computer that may recognize the tools that I used in this demo and block them. Or, you could implement some type of white list/black list of applications on clients, but this has its limits.

6. Use Authentication Mechanism Assurance, a new feature in Windows Server 2008 R2. Marcus Murray has a great blog on how to implement this in the real world.

If you want to try this out for yourself, in a test environment, you can download runhash x86 here, runhash x64 here, lslsass x86 here and lslsass x64 here. Remember, these tools could get you fired or worse if you use them in an unauthorized manner.

Lync Server 2010 Install failure on Server 2008 R2 SP1

2011-02-21T14:55:00.000-08:00

During my free time I thought I'd install Lync Server 2010 and check out the new features. If you want a great guide for installing Lync Server 2010, check out Jeff Schertz's blog here. I used Windows Server 2008 R2 SP1 slipstream media, since it was just released to the public a few days ago. Why not use the latest and greatest?

Everything was going fine until I got:

Error: Prerequisite installation failed: Wmf2008R2

After a few minutes of digging I spotted the problem, and it's directly related to Server 2008 R2 SP1. The issue is that the Windows Media format package name was changed in SP1 so the installer bombs since it can't find the RTM name. The old version was 6.1.7600.16385 and the new version is 6.1.7601.17514. Houston we have a problem!

To fix the problem you can manually install the component using the following command line:

C:\Windows\system32\dism.exe /online /norestart /add-package
/packagepath:c:\Windows\servicing\Packages\Microsoft-Windows-Media-Format-Pack
age~31bf3856ad364e35~amd64~~6.1.7601.17514.mum /ignorecheck

REBOOT the computer, then you can continue with the Lync Server 2010 installation.

Configure custom Default Profile in VMware VM Templates

2011-02-20T13:37:00.000-08:00

Over the past year I've been developing Windows Server 2008 R2 and Windows 7 VM templates for my VMware environment. However, the process has not been without its challenges. One of the features I wanted was a customized default user profile so that things like WMP, IE, and other settings were configured to our standards.

However, using the VMware customization specifications you can't easily change the XML file it feeds sysprep to perform a profile copy operation. So how do you get a customized default profile with VMware vCenter?

Here's the process that I use which works like a charm:

1. Install Windows into a VM using whatever method you wish (autounattend or manual).
2. Customize the Administrator's profile however you wish (desktop icons, launch WMP, modify the toolbar, etc.).
3. Install any additional software or other tweaks you want to make to your image.
4. Install User Profile Manager then use the Copy To feature to copy the settings to the default profile, as shown below.

5. De-install User Profile Manager.
6. Shutdown your VM and turn it into a VM template.
7. Use vCenter to provision a new VM from the template, and use the customization specification to modify the VM such as hostname, administrator password, etc.

When vCenter provisions your VM and you log into it for the first time the default profile contains your customizations, and is configured how you wanted it to be. Easy as pie!

Windows Recovery Environment mass storage driver injection

2011-02-18T16:47:00.000-08:00

In a previous blog here I described how to inject VMware pvscsi and VMNET3 mass storage drivers into your Windows Server 2008 or Windows 7 image. However, that did not cover injecting the same drivers into the Windows Recovery Environment, which is a separate WIM within the install.wim, thus requiring extra work. Here's how to inject drivers into the winRE.wim file and repackage the install.wim with the updated recovery environment.

1. Follow the first five steps at my blog here to install WAIK, find the right drivers, and create a scratch directory.

2. Mount the install.wim file from your Windows installation DVD:
dism /Mount-Wim /WimFile:D:\install.wim /Index:1 /MountDir:D:\mount

3. Copy the winRE.WIM to a working folder:
copy D:\mount\windows\system32\recovery\winre.wim D:\

4. Create another mount directory, D:\Mount2, then run this command:
dism /Mount-Wim /WimFile:D:\winre.wim /Index:1 /MountDir:D:\mount2

5. Inject the pvscsi and VMXNET3 drivers:
dism /image:D:\mount2 /Add-Driver /driver:d:\drivers\pvscsi.inf
dism /image:D:\mount2 /Add-Driver /driver:d:\drivers\vmxnet3ndis6.inf

6. Unmount the winRE image:
dism /unmount-wim /mountdir:d:\mount2 /commit

7. Copy the modified winRE.wim file to:
D:\mount\windows\system32\recovery\

8. Unmount and commit the changes to the install.wim:
dism /unmount-wim /mountdir:d:\mount /commit

At this point you now have a modified install.wim file that you can copy back into your Windows OS ISO. Depending on your install.wim file, you may have multiple operating systems that need to be modified. To do this you would serially mount all OS indexes, inject the new winRE.wim then unmount the image. For a typical Windows Server 2008 R2 DVD, you could have 8 images that need to be modified to cover all of your bases. So this process can be a bit tedious and would lend itself to scripting.

vCenter 4.1 U1 and FIPS encryption: Verify your IE Settings

2011-02-12T12:22:00.000-08:00

During my regression testing of my vCenter 4.1 U1 installation instructions on Windows Server 2008 R2, I came across a problem that made me scratch my head. I was updating the vCenter SSL certificates, per my blog here. However, when I opened IE and tried to connect to the vCenter default home page would not come up. I got Internet Explorer cannot display the webpage.

OK I thought, maybe I goofed up the SSL certificates. I regenerated them, and nope, no good! The Windows Server 2008 R2 template that I'm using is locked down and has many security features enabled, including FIPS compliant encryption.

You can connect to vCenter with the vSphere client, but it appears the web services on port 443 are broken. For example, as I mentioned, the vCenter home page would not come up, the vCenter Service Status screen would not open, and performance graphs were also broken.

After additional research since my original post, the root cause appears to be the combination of two security settings: FIPS compliance, AND restricting what encryption algorithms IE is allowed to use.

The IE settings that cause the problem is the unchecking of TLS 1.0, as shown below.

This in combination with enabling FIPS on the server, as shown below, create a situation that doesn't allow the TLS handshake to complete, so web based services that rely on IE settings break.

The lesson here is that if you have FIPS encryption enabled on the computer that you are accessing vCenter from, ensure your IE settings allow TLS 1.0. Normally TLS 1.0 is checked, so this won't be a problem for most people. But if you are trying to enhance security by only allowing TLS 1.1 or higher, then you will run into issues.

VMware VUM 4.1 U1 SSL Certificate Replacement

2011-02-12T09:12:00.000-08:00

One of the continuing pain points with VMware vSphere is the unnecessarily complicated procedure to install trusted SSL certificates in ESXi, vCenter and VUM. Up until 4.1 Update 1 (released 2/10/11), VMware had no public procedures to update the VUM SSL certificate, over 1.5 years after vSphere 4.0 hit the streets. Plus I've found that even the published procedures for ESX(i) and vCenter were convoluted and incomplete.

So over the last couple of years I've written several blogs about how to replace your ESXi, vCenter and VUM certificates. VMware made a little progress with VUM 4.1 Update 1, in that they now have a GUI utility that performs the behind-the-scenes reconfiguration of VUM to use a new SSL certificate. This new tool is called VMware Update Manager utility and does more than just update your VUM SSL certificates. You are still left with a painful process for ESXi and vCenter, so maybe in vSphere 5.0 VMware will wake up and provide a more streamlined procedure.

Even with the new tool in 4.1 U1, I found the associated KB article less than helpful and even tells you to leverage openssl on an ESX (not ESXi) host to generate new self-signed certificates. The second half of their article has instructions for using a trusted commercial CA certificate, but they still have you leveraging ESX to generate the certificate requests. This boggles my mind for several reasons:

1) ESX 4.1 is the last release and is a dying code branch. VMware has stated ESXi is the future and the only option in vSphere 5.0.

2) OpenSSL is not included in ESXi so you can't follow the KB article if you only run ESXi . Why publish a KB article that doesn't apply to organizations that only use ESXi?

3) OpenSSL is open source and widely available for free for many platforms including Windows. vCenter and VUM only run on Windows, so it makes a lot more sense to have customers download Windows OpenSSL and generate the certificates on a Windows computer.

4) Even if you have ESX, VMware always lags in incorporating the latest version of OpenSSL into ESX, so you could be using a version with known vulnerabilities.

I'm not trying to bash VMware, but come on guys, please get with the program. As a testiment to the SSL problems VMware has not addressed, my SSL blog posts get alot of hits. Until VMware "gets it right" I'll continue to help the community at large. So to that end, let's get on with how to update VUM SSL certificates in VUM 4.1 U1.

1. Download OpenSSL Windows binaries here. I recommend the full v1.0.0c package. Install OpenSSL using all default values on any Windows computer. I put OpenSSL on my vCenter server since I need it for ESXi and vCenter SSL certificate generation.

2. Generate a 2048-bit RSA private key (you could use 1024 bit as well, but I like stronger keys):
openssl genrsa 2048 > rui.key

3. Create a certificate request based on the previously generated private key:
openssl req -new -key rui.key > rui.csr

For the certificate request parameters (in green) use the values appropriate for your organization. The critical parameter, the common name (in red), should be the FQDN of your VUM server. Do not use a challenge password.

4. At this point you have a valid certificate request and you can submit it to a commercial CA, or your internal trusted CA. For the purposes of this article I will leverage a 2008 R2 Microsoft CA, so some steps may vary if you use a commercial cert.

5. Use NotePad and copy the contents of rui.csr to the clipboard.

6. Navigate to your Microsoft CA, click on Request a certificate, click advanced certificate request, click Submit a certificate request by using a base-64-encoded CMC....

7. On the Saved Request screen paste the contents of the clipboard, and change the certificate template to Web Server (or your organization's web server template name).

8. Submit the certificate request and download it as base-64 encoded WITHOUT the certificate chain, and save it with a filename of rui.cer.

9. Type the following command (and use a blank password when prompted):
openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -out rui.pfx

10. Stop the VMware vCenter Update Manager service.

11. Backup the existing VUM certificates located your VUM directory, by default it's :
C:\Program Files (x86)\VMware\Infrastructure\Update Manager\SSL.

12. Copy your new rui.crt, rui.key and rui.pfx files to the SSL directory above, replacing the existing files.

13. Navigate to C:\Program Files (x86)\VMware\Infrastructure\Update Manager and launch VMwareUpdateMangerUtility.exe. Login with your vCenter administrator credentials.

14. Click on SSL certificate then check the box under the instructions and finally click Apply.

15. Restart the VMware vCenter Update Manager Service and pray it starts.

16. If you've left any of your certificate files laying around the file system, except in the VUM SSL directory, you should back them up to a secure location then delete them. You need to protect the private keys, so don't leave them laying around just anywhere.

17. Launch the vSphere client and connect to vCenter. Verify that the VUM tab appears and that you can access VUM without any errors. It would also be smart to check the vCenter Service Status from the vCenter home page to ensure everything looks healthy.

The gotchas of backup software 'capacity' licensing in virtual environments

2011-02-03T20:15:00.000-08:00

Some backup software manufacturers are offering a capacity based licensing model, instead of a agent based model. Depending on your situation, this may be a dramatically easier and cheaper licensing model. Traditionally you had to buy per-server licenses, per-agent licenses, per-library licenses, and maybe other options as well. Backup software licensing could get very complex and very expensive. But it's extremely important to understand how their model works, for both physical and virtual servers or you may be in for sticker shock.

CommVault, Symantec, and others have introduced capacity based models. With this model, you have unlimited number of agents and servers, but the total amount of data you want to be backed up must be licensed. Depending on the vendor, they may have slight variations on this model. Tivoli capacity licensing is nearly as complex as their agent based model.

When can this model be more cost effective? Generally if you have a lot of servers with minimal data on them, this model works well. Per-server and per-application agents can get expensive. If you have a small number of servers with huge data stores, then a per-server/agent method could be more cost effective.

But, you need to be VERY careful and understand how the backup product measures capacity, if you go with that model. In a physical environment, it's generally very simple. If your pizza box server has 4TB of storage but you've only written 500GB of data, 500GB would count towards your capacity license. The remaining 3.5TB is 'free', until you start to physically use it.

In a virtual environment, this can get more complicated, and you could be in for some nasty surprises. Let's say you do a P2V migration of that same pizza box server, but you downsize the virtual disks to just 1.5TB. Now the million dollar question is, how much capacity counts towards your backup license? 500GB or 1.5TB, or something in between?

With CommVault Simpana 9.0, the answer is 'it depends'. CommVault counts the VMware VMDK disk size against your capacity license, regardless of how much physical space the VM is using. If you use VMware thin provisioned VMDKs, then at least 500GB comes out of your capacity license. If you use thick VMDKs and rely on your storage array to do the thin provisioning (which nearly all modern arrays support), the full 1.5TB counts against your license!! This is because Simpana is not intelligent enough to look inside the VM for actual disk usage and just charge you for the allocated amount. It looks at the VMDKs as a big blob, and charges you accordingly.

As a result, licensing for a virtual server can be significantly more expensive per GB than a physical environment, at least with Simpana 9.0. I don't know how NetBackup or other capacity based products count VMDK storage. I'd hope they are more consistent and use the physical server logic. If my VM has a 2TB hardware thin provisioned disk, but I've only written 500GB of data, only 500GB should count.

Using VMware thin provisioned disks doesn't fully solve the problem. Why? Let's say you have 2TB software thin provisioned disk, and the allocated VMDK space is 500GB. If you copy 1TB of data into the VM, then delete it, the VMDK is now 1.5TB but only contains 500GB of data. So you pay 3x the price for the same 500GB of data, unless you somehow shrink the VMDK.

Finally, if you leverage VMware fault tolerance, you simply can't use VMware thin provisioned disks. You must use EZT (eager zeroed thick) disks. So regardless of how much or how little disk space you use, the total VMDK disk size counts against your license.

This can be very confusing. CommVault has a two methods for counting capacity that you need to be aware of. One model for physical servers and another for virtual servers that can catch you off guard depending on how your VMDKs are configured.

Authentication Denied: Unable to logon to an ESXi 4.1 console

2011-02-01T20:45:00.000-08:00

Over the last couple of days I've been performing ESXi 4.0 to 4.1 build 320137 upgrades. During the upgrade process today one of my servers had a hiccup and the web services was not responding. After a couple of reboots and non-responsive web services I iLO'd in (it's an HP server) so I could get to the ESXi DCUI (direct console user interface), AKA the yellow screen. To my shock when I pressed F2 I got:

Authentication Denied: Direct console access has been disabled by the administrator for contoso.net.

At first I thought OK, maybe someone enabled lockdown mode and I didn't know it. Checked a few things, nope, no lockdown mode. After more poking and prodding, I found the root cause of the problem. But it's a mystery to me why this is occurring. The only consistent theme is that the server was on ESXi 4.0 and they were upgraded to ESXi 4.1 build 320137.

So what was the problem? New to ESXi 4.1 is the Security Profile configuration screen. Here you can stop/start several low-level system services. On 25% of my upgraded boxes the "Direct Console UI" service was in the stopped state as shown below.

The solution is to reconfigure the service to Start and Stop with host, which is the ESXi 4.1 default configuration. After I started the service, viola, DCUI access was restored!

Since this happened on several boxes, but not all, I'll chalk it up to another VMware bug. So in my upgrade procedures I'm adding a check to verify the service status before we bless an upgrade as being complete.

XenDesktop 5 Machine Creation failure: VMware PVSCSI driver fix!

2011-01-29T13:50:00.000-08:00

During my testing of Citrix XenDesktop 5 I ran across yet another bug, which set me back in my testing. Apparently if the VM template that you want machine creation services (MCS) to use has been configured with the VMware pvscsi controller, creating the VMs will fail when you generate a catalog.

The error that XenDesktop Desktop Studio will give you is:

The specified master VM snapshot could not be found. No machines have been created.

If you look in the Windows application log you see:

Provisioning scheme creation workflow operation failed : System.InvalidOperationException: VM not Found ---> Citrix.HypervisorCommunicationsLibrary.InvalidVmConfigurationException: No disk controller found

As mentioned in my previous blogs, I always use the VMware pvscsi controller since it's more efficient than the emulated legacy SCSI controllers. But, it seems that Citrix didn't test this use case, since it fails miserably. The fix is to not use the VMware pvscsi controller, and use something like the LSI Logic SAS controller. But what if you already have a VM template built with the pvscsi controller, like me, and you don't want to rebuild it because of a Citrix bug?

There's an easy fix! While your VM template is running and using the pvscsi controller, open an elevated powershell and type the following:

Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\services\LSI_SAS" -name "Start" -Value 0 -type "DWORD"

I then rebooted the VM (still with the pvscsi controller), shutdown the VM, then in vCenter changed the SCSI controller type to LSI Logic SAS. Next time the VM boots the LSI Logic SAS driver will be active at boot time and your VM won't blue screen.

I hope Citrix can fix this bug in their next update for XenDesktop 5. It's a bit disturbing that this scenario, just like the FIPS bug, wasn't tested prior to GA.

Update: This issue is now fixed in XD5 SP1. Check out my post here.