If you aren't familiar with thin provisioning, it's a technology being widely adopted by storage vendors that only allocates back-end disk space to a LUN when data is written. For example, if you present a 500GB LUN to a server and only write 100GB of data, the array only uses 100GB of disk space. The remaining 400GB can be used by another server, or the same server when more data is written.
But what happens when you delete the data? With many arrays the space is still allocated to that LUN and not put back into the 'scratch' pool for future re-use. Depending on your data access patterns, you may end up with a lot of trapped capacity. Thin volumes can gain weight and become fat..like much of America!
Several vendors such as EMC, Hitachi and 3PAR have updated their arrays to reclaim some of this trapped capacity. They do this by detecting chunks of zeros and tossing those chunks back into an unallocated storage pool. Since we use a 3PAR array at work, I decided to put their Thin Persistence feature to the test.
Unfortunately I can't post screen shots since I don't have an array at home and this is my personal blog. So you will have to take my word for the results. Listed below are the high-level steps that I took to perform the test.
1. Provisioned a new thin-provisioned virtual volume of 100GB in the 3PAR T400.
2. Enabled zero-detection on the volume (simple GUI check-box).
3. Exported that virtual volume as a LUN to a Windows Server 2003 R2 physical server.
4. Formatted the new LUN with the NTFS file system and assigned it a drive letter.
5. Copied 90GB of data into the new volume and verified in the 3PAR InForm Management Console that it showed approximately 90GB capacity utilization for that volume. Check, it did!
6. Deleted 40GB worth of data from the LUN, so 50GB was remaining.
7. Executed the MS Sysinternal tool "sdelete" with the -c option to zeroize all unused disk space.
8. I then monitored the volume's utilization in the 3PAR InForm management console, and it slowly dropped as the sdelete command was running.
9. By the time the sdelete command finished, the volume utilization was down to 50%, matching the 50GB of still used disk space. 40GB of space has been reclaimed.
This test proved that the 3PAR zero reclaim feature worked as advertised, happens in real time, and take very little effort to use. The same process would work for a virtual machine as well. If I was using the Veritas Storage Foundation I would not have to use the sdelete command and it would be fully automated. Hopefully they will work with Microsoft and VMware to support a fully automatic and native method to reclaim the deleted space. Until then, you can run sdelete from time to time to drop those extra pounds from your fat LUNs.
Stay thin with no gym time for your array..yippee!
Hi Derek, Marc Farley from 3PAR here. Thanks for posting this. We are constantly working with others to implement thinning technology. For instance, our support for automated thinning with OPracle ASM and implementing VMware's VAAI Block Zeroing command, which essentially "thins" a thick volume when it is being created. The work goes on...
ReplyDeleteI was able to achieve similar results a couple of days ago by seeding a 3PAR VLUN with data from /dev/urandom on a RHEL box, deleting the random data, and "replacing" it on the filesystem with a stream from /dev/zero.
ReplyDeleteAwesome stuff.
I've got a 3par F400 running InForm 2.3.1, hosting volumes for a 3-host VMware cluster. One of the primary vmfs volumes, 1.5TB in size, is running at 90% capacity. I freed up and deleted a couple vmdks on the volume, and VMware showed an extra 600GB freed up. On one of the ESX hosts, I used dd to write 250GB of zeros to a file on the filesystem, but didn't see any of the space reclaimed on the provisioned volume in InForm. In fact, it appeared that the reserved usage went UP instead. The volume was configured as thin-provisioned, zero-fill on partially written pages, and zero detect.
ReplyDeleteI'm just going off memory here, but I think there was a command you run on the nodes to globally turn on the thin reclimation feature, in addition to the LUN configuration in the IMC. If you did an upgrade from previous InServe versions, look at the various release notes or call 3PAR. I've seen very good results with zeroizing LUNs within a guest OS.
ReplyDeleteDoes this work on InForm OS version 2.3.1 MU3 already or do I need to upgrade to the latest InForm OS 3.1.1 to make use of the space reclamation feature?
ReplyDeleteSpace reclamation works fine with 2.3.1 MU3.
ReplyDeleteHey Derek. i like your work. Keep it up. I have a question- I'm working with a 3Par P400, and I have provisioned full virtual volumes to several host. Wanting to take advantage of thin provisioning...is there away for me to convert these fully provisioned volumes (that were already exported and are in use) to thin volumes? If so, what ate the risks and what are the steps to achieve this? Thanks.
ReplyDelete@Danjehflexx: I don't have the IMC handy, but as I recall you can take a thick volume and make it thin, online without any disruption. You may need a software license such as dynamic optimization to do that though. So if you don't have the option, you may not be licensed. As of 3.1.2 you can also take a thin volume and make it fully provisioned, all online. If you are using VMware, you can also just storage vMotion the VMs to thin datastores and not require any 3PAR license beyond the thin suite.
ReplyDeleteI just created a automatic proces for determine the used size on the 3par versus the size on a Windows Server 2003 R2 server. I did it with system center orchestrator. The plan is that when the difference is more then 100 GB I want to start a sdelete proces. I am almost done but after the sdelete I see that the vv on the 3PAR F400 still using the space by "Reserved User Space", the space named under "Used User Space" seems to be good. My question is how can I reclaim the "Reserved User Space". I hope you can help me.
ReplyDelete@Pieter: You might want to look at the CompactCPG command.
Delete