Sunday, March 13, 2011

Building a Sandy Bridge ESX Server

A few months ago I posted the parts list for sub-$1000 ESX server. At that time Intel had not released their Sandy Bridge CPUs or chipset. Now that manufacturers are shipping chipsets without the SATA bugs, I thought I'd post an updated list of components for a screaming home ESX server. You can buy everything you need except the CPU cooler from TigerDirect.

The same criteria apply to this server as my previous post: compact form factor (HTPC case), quiet, power efficient, and dual 1Gb NICs. The CPU has a built-in GPU, so no separate graphics card is required. If you already have external shared storage (like a QNAP), then you don't need an internal HD and could use the 8Gb tiny flash drive listed below to install ESXi on. The NICs on the motherboard are not supported by ESXi, which is why you need the SuperMicro card.

Intel also ships a lower power i5-2400 CPU (65w max vs 95w max), but it runs at a lower clock speed. Depending on your requirements, this may be a better option if your electricity bill is a concern. The WD hard drive is a screamer, but is a bit noisey. So if the noise will disturb you, going with their green line would be a much quieter option, but is slower.

Antec MicroATX Minuet 350 case  $116
Asus P8H67-M Pro $130
Intel Core i5-2400 $200
Patriot 8GB DDR3 PC3-12800 2x $85
SuperMicro Dual GiGE NIC AOC-SG-I2 $86
Scythe Big Shuriken CPU Cooler $35

Total: $737 + Shipping

Optional parts:
Asus DVD Burner $20
LaCie 8Gb ultra-small USB flash drive $25
Western Digital 1TB 6Gbps Caviar Black SATA HD $85
Intel Core i5-2400S Low-power CPU $200
Update 1: Got all of the parts and the server works great. I would advise that you install the memory before you put on the CPU cooler, or you will find yourself taking the CPU cooler off. Also, if you don't get the memory I specified above, make sure the memory you get is low profile and doesn't have any heatsinks sticking up above the DIMM PCB. I really like the new ASUS EFI BIOS..very graphical, easy to use, and slick. I also decided to get a tiny 4GB USB memory stick from Best Buy for $15, located here.

Update 2: I did some power measurements, and idiling the server uses 47 watts, which is more than 20 watts less than my previous ESX server. The integrated graphics card and more power efficient CPU sure make a difference!

Update 3: vSphere 5.0 recognizes the onboard RealTek NIC. So if you are satisfied with a single NIC you can skip the SuperMicro card in my parts list. If you want the very latest motherboard the Asus P8Z68-M Pro works extremely well, and its onboard NIC is also recognized. The Z68 chipset has the advantage of overclocking the CPU while letting you use GPU equipped CPUs, so you don't need an add-in PCIe video card like you do with the H67 chipset.
at

40 comments:

  1. UnknownMarch 27, 2011 at 10:18 AM

    Hello Derek,
    First, excuse me for my bad English. I'm French, and have a serious problem with foreign languages​​. I am interested in your experience with ESX and Sandy Bridge. I bought an ASUS P68H67 Pro and an intel i5 2400 in January to create a WhiteBox ESXi. A Intel Sandy Bridge Bug's , and also by lack of time, I stopped my tests. After reading your post, I'll try to find NIC based on Intel chipset and I will restart my test on ESXi . If you have other recommendations for me, I'll be happy. I'll try to keep you informed of my progress.
    Tank you to share your experience

    ReplyDelete
  2. NoneMarch 27, 2011 at 10:20 AM

    Works great..like you mentioned, just make sure you get a NIC that works with ESXi. The SuperMicro one I linked to above works perfectly.

    ReplyDelete
  3. AnonymousMarch 29, 2011 at 9:57 AM

    Very nice build for a low price. I am going to build a similar system soon. I will probably follow your build except I will go with an i7 2600, do you recommend the same motherboard or should I go with a different one? My server will be racked in a colo so power/noise isnt a problem.

    ReplyDelete
  4. AnonymousMarch 29, 2011 at 10:02 AM

    also, are there any known sandy bridge motherboards that have nic's that work in esx4.1? I'll need a perc4 card in my server, so I am unsure if I'd have room for the extra nic.

    ReplyDelete
  5. NoneMarch 29, 2011 at 8:00 PM

    Unless you are on a very tight budget, I would not recommend building your own server for production usage. I'd go with a low-end HP or Dell server, since it would be more reliable and certified for VMware ESXi.

    ReplyDelete
  6. NoneMarch 29, 2011 at 8:02 PM

    There may be a SB MB with NICs that ESXi would support, but I haven't seen one (haven't looked either). Most of the MBs these days use RealTek which is unsupported by VMware. That's why I'd get a low-end server from a major vendor since it will be fully supported.

    ReplyDelete
  7. NoneMarch 29, 2011 at 8:09 PM

    Oh, it looks like there is underground support for the RealTek NIC drivers. I haven't tried this myself, yet:

    http://www.vm-help.com/forum/viewtopic.php?f=12&t=1272 for support Realtek 8111e

    ReplyDelete
  8. electronMarch 30, 2011 at 4:03 AM

    Thats a asume hardwarelist. I will build a simular system. I need more memory, but I can´t find Momory with 8GB DIMM per slot. Do you know where I can find 8GB DIMMs. The memory compatibility list will not work by the asus homepage.

    ReplyDelete
  9. AnonymousMarch 30, 2011 at 9:37 AM

    Thanks for the info!

    It would actually be a personal server, not production with customers or anything. I'm looking to spend around 1k.. but what price would I be looking at if I was to build a similar HP or Dell system? I am up to speed on homebuilt machines, but I really dont know what a pre-built system models or cost would be..

    ReplyDelete
  10. NoneMarch 30, 2011 at 5:46 PM

    You'd need to contact Asus or whoever makes your MB to get a list of certified memory. Kingston makes 16GB kits, but I don't know if they work in the Sandy Bridge MBs.

    ReplyDelete
  11. NoneMarch 30, 2011 at 5:52 PM

    If it's a personal server, then I would be find with home built server. Unfortunately the low-end HP servers that are on the VMware HCL are more than $1K. The cheapest is a DL160 G6, which is going to cost you more than $2K if you want a good amount of memory and storage.

    http://h10010.www1.hp.com/wwpc/us/en/sm/WF25b/15351-15351-3328412-241644-3328421-3884343.html

    ReplyDelete
  12. electronApril 12, 2011 at 8:10 AM

    Hi Derek,

    thanks for the Info. I have another question:

    It is possible to configure up to 2 Raid arrays for the Motherboard? In advance can I add another Raid array to the esx as a datastore when I realize that the first datastore is full?

    Thanks a lot,
    Nico

    ReplyDelete
  13. NoneApril 12, 2011 at 6:13 PM

    The onboard RAID controller is not supported by VMware. So you would need a third party SATA RAID controller that is on the VMware HCL. You can add additional datastores to ESXi at anytime, and move VMs to the new datastore. Whether you can do it 'live' or 'cold' depends on the type of ESXi license you have.

    ReplyDelete
  14. AnonymousApril 14, 2011 at 7:55 AM

    I decided to give this a go and I am running into some issues. Did you have to enable/disable any BIOS settings other than virtualization support? Any special oem file used? Doesn't sound like you did but just in case I missed it. I can get the ESXi 4.1 CD to boot up but it fails to start the install with a system image count not be mounted or unsupported CD Rom (I think that is the error).

    ReplyDelete
  15. NoneApril 14, 2011 at 6:23 PM

    Try hanging the HD and DVD drive off the two 6Gb SATA connectors. I had a similar issue when the DVD drive was on one of the 3Gb connectors.

    ReplyDelete
  16. UnknownApril 14, 2011 at 7:10 PM

    Thanks Derek.. I am using an IDE DVD and trying to install onto a USB stick. I am thinking it is the IDE that controller that it doesn't like for some reason? Strange it will boot up the installer though. Going to try 4.0.0 to see if that works - I read one forum post that said this worked for them so figured why not give it a shot.

    ReplyDelete
  17. NoneApril 14, 2011 at 7:12 PM

    Ah, I haven't used IDE with ESXi 4.1, so can't comment on that.

    ReplyDelete
  18. UnknownApril 16, 2011 at 7:59 AM

    Looks like the IDE CDROM was the problem. Hooked up a SATA drive from another system and I could install, bootup and am now creating a Win2K8 VM without any issues! I was worried about my purchase for a while there...but now I am feeling much better.

    ReplyDelete
  19. NoneApril 18, 2011 at 6:43 PM

    I know for sure the 2 6Gb/s ports work in ESXi 4.1 U1. I have not connected devices to the 3Gb/s ports.

    ReplyDelete
  20. electronApril 20, 2011 at 6:07 AM

    Hello Derek,

    I will not built a business ESX-Server. So it will not so criticaly. But be right, I will use a RAID Controller. Can you tell me if the Paththrough function will work with the mainbord?

    Thanks!

    ReplyDelete
  21. NoneApril 20, 2011 at 3:53 PM

    The onboard NICs are not supported by ESXi. That is why I have the SuperMicro/Intel dual-port NIC card. Be sure to check that the RAID card you want to buy is on the ESXi HCL.

    ReplyDelete
  22. electronApril 21, 2011 at 2:04 AM

    When you say not supported, do you mean that the NIC will not work? To come back to paththrough again:

    Did the PCI Paththrough will work? I know DL380 G5 server, the PCI Paththrough will not work. When you look at the part in the vsphere client, it this function available?

    Thanks!

    ReplyDelete
  23. NoneApril 21, 2011 at 4:43 PM

    VMware does not support most (all?) RealTek NICs, which most consumer MBs use. So no, ESXi does not support the two onboard NICs, which is why I have the SuperMicro card on the purchase list. If you are referring to VMDirectPath, no the low-end Intel NICs do not support that feature. You would need to buy a higher end NIC card.

    ReplyDelete
  24. GiovanniMay 15, 2011 at 3:51 PM

    Hi

    Does your mobo the Asus P8H67-M Pro support VMdirectPath? (check on vSphere client) does it allow you to configure passthrough?

    Thanks!!

    ReplyDelete
  25. NoneMay 15, 2011 at 3:52 PM

    No, since the on-board NICs are NOT supported by ESXi. You'd need a third-part PCIe card that supports VMDirectPath.

    ReplyDelete
  26. IgroAugust 8, 2011 at 11:07 PM

    It's a great news for me that Core i5-2400 works with ESXi! By the way, are you using version 4.1 Update 1? I'm going to install VMware Workstation first and then deploy VSphere infrastructure in it, so the only concern I have is the CPU support. Hope the Core i5-2400 will work fine. Did you have any problems related to CPU? Thanks!

    ReplyDelete
  27. NoneAugust 10, 2011 at 7:36 PM

    Yes I'm using 4.1 U1 and it works like a charm. No problems with CPU.

    ReplyDelete
  28. Jase McCartyAugust 11, 2011 at 11:57 AM

    Good info Derek.

    Nice setup at a good price.

    I went a little different route, with server based boards.

    E3-1230s, 16GB RAM, iKVM, 5 nics, local SSD, etc for $1650
    http://www.jasemccarty.com/blog/?p=1516

    Also some as well as a cheaper configs based on the same setup here:
    http://www.jasemccarty.com/blog/?p=1571

    Cheers,
    Jase

    ReplyDelete
  29. FredOctober 7, 2011 at 12:24 PM

    Derek,

    Great information on the Website! You mentioned that Asus P8Z68-M Pro works extremely well with vSphere 5. Have you tried it or know of someone who has? I am looking to replace my current setup with a pair of servers to put in a cluster.

    Thanks Again!
    Fred

    ReplyDelete
  30. NoneOctober 14, 2011 at 4:35 PM

    Ya the P8Z68-M Pro works great..have an ESXi 5.0 machine running with it.

    ReplyDelete
  31. AnonymousOctober 19, 2011 at 9:16 PM

    I have the P8Z68-M LE board. Trying to use the onboard SATA RAID capabilities, but ESXi 5.0 still sees each individual drive, even though I setup up a RAID group of the 4 drives in the system.
    It's hard to find info on this RAID functionality. I am wondering if this is more of a software RAID assist, and requires drivers on a supported OS.
    Anyone have any knowledge here?

    ReplyDelete
  32. NoneOctober 20, 2011 at 4:55 PM

    You can't use the cheapo Intel "RAID" controller on these consumer MBs with VMware ESX. To utilize RAID you would need to look at the VMware HCL and find a RAID controller that is listed.

    ReplyDelete
  33. AnonymousOctober 23, 2011 at 4:24 PM

    i really wanted to do something like that itx build there, but being limited to 8Gb (2x4) total, because of the poor price of 8GB dimm sticks put me right off. I went on the scrounge and assembled an i7 965 system with 24GB ram, from ebay, and buying the rest of the components locally.

    ReplyDelete
  34. AnonymousNovember 30, 2011 at 7:38 AM

    Nice article, but does the P8Z68-M support vt-d on your esxi5 setup?

    ReplyDelete
  35. UnknownDecember 6, 2011 at 9:47 AM

    Excellent post Derek. Am planning such a lab for VCAP prep and now more confident about it. Would be going ahead with i5 2400S. As per my understanding, FT and VM DirectPath IO will not work, but rest of the fetus should be fine. Is my understanding right?

    ReplyDelete
  36. NoneDecember 6, 2011 at 4:04 PM

    Actually FT should work, but DirectPath IO will not.

    ReplyDelete
  37. AnonymousDecember 31, 2011 at 4:58 PM

    Derek I want to make your config, but curious how many simultaneous VM's can you run at the same time on a 16GB setup? Right now I have a dual Xeon E5340 w/ 16GB's as my main ESXi Server but want something up in the living room to show off and maybe purchase a full license for testing. But curious how many VM's I can setup and run w/ at least 15% utilization. I'm curious w/ this motherboard if it has a built in graphics controller as I would like to install a higher end i7 if possible in that same box. Do you think that will be possible?

    ReplyDelete
  38. NoneDecember 31, 2011 at 9:38 PM

    Anonymous it would really depends on what those VMs were doing and what kind of performance they needed. The CPU has a built-in GPU, so you don't need a video card. The MB supports a variety of CPUs, so just check the MB specs for what it supports and get something higher end if you want.

    ReplyDelete
  39. AnonymousJanuary 20, 2012 at 4:08 AM

    Hi Derek,

    I have read your article and I build an ESXi 5 server with an Intel Core i7 2600S (65W) an ASUS P8Z68-V LX and 16GB DDR3 1333 RAM.

    My problem is, after some time, the ESXi kernel crash with a host purple diagnostic screen and "Lost Heartbeat" from CPU0. I think its a enery saving thing in the Bios, so I have deselect the C1E states and all other enery saving things, but this doesnt work. The time is longer, but the ESXi host crashes anyway.

    Its horrible, I cant localize the problem, an Memtest/CPU-Test works successfully, so the hardware isnt defect.

    Do you have any ideas? Thank you very much.

    Regards

    Mireux Chopard

    ReplyDelete
  40. UnknownFebruary 1, 2012 at 3:13 AM

    Hi Derek, got i5 2400 + p8z68-m-pro + 16gb assembled and works fine with vsphere v5 and 4.1 (custom esxi). Been able to create nested esxi, that too works fine and the assembled system totally rocks. Thanks for the info. Was wondering if pushing the RAM to 32gb would be good idea? Any comments?

    ReplyDelete