First, refer to Patrick O’Rourke’s blog entry, which does a great job of summarizing the need for application mobility licensing. Clearly, customers needed the ability to move applications freely between physical servers, and Microsoft themselves needed to allow customers to do this now that they have a more robust virtualization solution in place (Hyper-V and SCVMM 2008). While the licensing changes do benefit all virtualization vendors, it’s important to note that Microsoft needed these changes for themselves as well.

Patrick’s post also brings to light that while VMware has joined the SVVP, cooperative support is not yet in place. That won’t come until validation via SVVP is completed, which may take some time. The joining of SVVP was necessary, as it is merely one step toward a larger goal.

However, there’s more here than perhaps many people are realizing. Fortunately, there are a number of sites out there pointing out important caveats to the new licensing changes:

• Rich at VM /ETC correctly points out that the new licensing does not apply to the Windows Server OS itself. So you are still going to have problems with VMware HA and VMware DRS automatically moving VMs from server to server unless you use Windows Server Datacenter Edition (see below).
• Chris Wolf points out (both on his personal blog as well as the Data Center Strategies blog) that the lift on the 90-day license transfer does not apply to licenses purchased outside of a volume license agreement. Using OEM licenses? Then you’re out of luck; those licenses still fall under the old restrictions.
• eWeek’s Joe Wilcox points out that because the Windows Server OS isn’t included in the 90-day license relief, some customers will simply license Windows Server Datacenter Edition for every CPU in their data center. Of course, the fact that you now get Hyper-V for free with Windows puts Microsoft…ahem, ahead of the game, shall we say? Read Joe’s full report here.

So, while Microsoft’s licensing changes are a good first step, there’s still more work to be done. Let’s applaud the changes, which were necessary, but let’s continue to press Microsoft to fix the issues that remain.

• A Quick Note to VMware
• TechEd Schedule
• Hyper-V Released
• Microsoft Virtual Server Now Free Also
• VIR253: Microsoft System Center VMM 2008, Part 1 of 2

Tuesday, September 16, 2008

9:30 AM to 10:30 AM - BC2621 - Fault Tolerant VMs in VMware Infrastructure: Operation and Best Practices
1:00 PM to 2:00 PM - TA2668 - VMware ESX Architectural Directions
2:30 PM to 3:30 PM - BC1693 - Architecting DR Solutions with VMware Site Recovery Manager
4:30 PM to 5:30 PM - PO1694 - Datacenter Migrations Using VMware Site Recovery Manager

Wednesday, September 17, 2008

11:00 AM to 12:00 PM - VD2422 - Offline VDI
1:30 PM to 2:30 PM - TA2659 - Managing ESX in a COS-less world
3:00 PM to 4:00 PM - TA2275 - Tech Preview: VMware Infrastructure Virtual Networking - Future Directions
4:00 PM to 5:00 PM - BC2215 - Top Tips for VMware Consolidated Backup

Thursday, September 18, 2008

11:00 AM to 12:00 PM - TA2441 - VMware Infrastructure 3.5 - Networking Concepts and Best Practices
1:00 PM to 2:00 PM - PO1644 - VMware Update Manager Performance, Best Practices
2:30 PM to 3:30 PM - PO2061 - VMware VirtualCenter 2.5 Database Best Practices
4:00 PM to 5:00 PM - BC2214 - Advanced HA Troubleshooting

As you can see, there are some gaps in the schedule. First, I’ve specifically blocked off time to go to the vendor pavilion and talk to some vendors about their products. This is something that I always say I’ll get around to doing, but never actually make it. This year I decided to fix that by scheduling time specifically for that. Second, I’m working with SearchVMware.com again this year on some stuff, so there’s time set aside for that. Finally, my boss has scheduled some meetings with various people while I’m there, so I had to set aside time for that as well.

If you’re interested in catching up with me this year, shoot me an e-mail and let me know. I’ll do my best to meet with you while I’m there. In addition, I’m getting in early before the conference and not leaving until the Friday afternoon, so I might be able to meet with someone outside the conference days themselves.

If you’re attending and have a session suggestion for me, feel free to shout out in the comments below. Thanks!

• My VMworld Schedule
• Spouses at VMworld
• TechEd Schedule
• Liveblogging at VMworld 2008
• A Few SRM Discussion Points

If I had to choose only one thing that I wanted VMware to be sure to do in the next few months, here is what it is: join the Microsoft Server Virtualization Validation Program and get VMware Infrastructure validated. No, I don’t want you to finish VI4. No, I don’t want you to concentrate on Continuous Availability. I want you to join SVVP.

Why? I can answer that question with only one word: support.

As fully expected now that Microsoft has a more robust virtualization solution, there has been a sudden about-face on the use of server virtualization in conjunction with their products. Consider this announcement regarding support for Exchange Server 2007 SP1 (quoting from this blog entry):

Microsoft now supports Exchange Server 2007 SP1 running Hyper-V or hypervisors validated under the Microsoft Server Virtualization Validation Program (SVVP).

I can hear you at VMware now: “We don’t need to validate our technology by signing up for some program run by Microsoft! Our hypervisor is the best! We have the best feature set! 100% of the Fortune 100 use our products—what do we need SVVP for?”

Relax, I understand where you’re coming from. But, honestly, let’s get real about this. VMware, do you want to remove the #1 obstacle to customers adopting your technology? That #1 obstacle is support, and whether you like it or not the introduction of the SVVP by Microsoft gives you a vehicle whereby you can remove that obstacle. A refusal to participate in the SVVP—even if you disagree with the program in principle—merely locks VMware out of many opportunities where Microsoft will be able to walk right in. That’s right! Your superior technology is going to lose out to technology that is just “good enough” simply because of support. Is that what you want?

So go sign up for SVVP now. If you don’t, you’re giving the competition a great big weapon to use against you. The power to remove that weapon is yours.

UPDATE: Looks like I waited a bit too long to chide VMware on their lack of inclusion in SVVP. In any case, I’m glad to see the news. Good job, VMware! (My thanks to Dave, who pointed this out to me.)

• Thinking on Microsoft’s Licensing Changes
• The Practicality Perspective
• Citrix Access Essentials Unsupported Features
• Virtualization Sea Change?
• Rootkit Prevention Coming to Windows

The movie’s web site is here, and there’s also a Fireproof blog available as well.

Marriage resources based on the movie are available at fireproofmymarriage.com.

Lord willing, I will be there opening night, paying full price for tickets to see the movie. These kinds of movies are worth supporting.

• Another Take on Life(song)
• A New Direction
• Weblog Moved to a New Host
• Seeing the Stars
• NetApp OmniGraffle Stencils

defaults write com.apple.mail PreferPlainText -bool TRUE

So that’s all well and good, but what I’ve noticed is that Mail.app seems to “ignore” some of the line endings in my message. It primarily only happens in signatures; I haven’t noticed it happening in the body of the message. At the same time that I adopted plain text messages, I also adopted the “standard” signature delimiter of two dashes and a space, so my signature will typically look something like this:

-- (hidden space at the end here)
Scott

What happens is that Mail.app turns it into this:

-- Scott

What in the world? Why is Mail.app playing with my signature? I’ve also noticed that in my longer signature—where I include my official title, phone numbers, company name, etc.—that Mail.app plays with the line endings there as well.

It also seems that this may be somehow related to Exchange Server 2007, as it only seems to happen to messages sent through my corporate Exchange infrastructure (I use IMAP and SMTP for connectivity to Exchange). I can’t find a single instance of an e-mail message where this has happened with any of my other non-Exchange e-mail accounts. But this doesn’t really make much sense, because the message I’m seeing is the local copy after it is submitted via SMTP. Perhaps the way in which Mail.app interacts with the SMTP server affects how the message in the Sent mailbox looks? I don’t know.

This is really irritating. If I type something, Mail.app (or Exchange Server) should NOT be going back and changing what I type. Anyone have any clue what could be going on here, or how I might fix it?

• STARTTLS and IMAP in Mail.app
• Erroneous Mail Relay Error with Exchange
• Preserving Nickname Cache in Exchange Migrations
• Suggestions for Processing Mail
• IMAP, VersaMail, and Mail.app

In my opinion, there are three key skills that professionals really need:

1. Being more effective (doing the right things) and more efficient (doing things the right way); and
2. Collecting and managing information effectively and efficiently; and
3. Brainstorming and thinking creatively.

In my mind, these three skills are kind of like the legs on a three-legged stool. You need all three to be really solid.

I adopted a “Getting Things Done” (GTD) approach for the first skill listed above, as described first here and then in this follow-up. I think that the use of OmniFocus and the GTD methodology has helped me to improve myself in this key area.

To help address the second key skill, I need to find and start using an information aggregator, like EagleFiler, Yojimbo, or Together. The problem is that I can’t find one that I really like. Each of these has some features that I like. Each of them also has some stuff I don’t like. I know that no application will be an exact fit, but I’m still having a problem getting over the stuff I don’t like. If anyone has any suggestions for other applications of this type that I should evaluate, please let me know.

The addition of MindNode helps me with the brainstorming/creative thinking skill. Until now, I hadn’t really been able to find an application that helped with this. Having settled upon TextMate as my text editor of choice, I can use TextMate for text-based brainstorming, but sometimes a graphical mind map is the best tool. The problem that I have with brainstorming and mind mapping is that I tend to pre-judge the ideas. The real purpose behind brainstorming is just getting all the ideas out of your head and onto the paper or mind map, and not to evaluate them. Evaluating them is supposed to come later. I’m hoping that the more I use mind mapping the better I’ll get about freely listing the ideas.

Anyone else have any other suggestions to helping to improve brainstorming and creative thinking skills?

• TextMate Revisited
• What I Want To Do When I Grow Up
• TextMate
• Optimizing iSCSI Traffic with ESX
• Plain Text Messages with Mail.app

• Cancel or Allow?
• del.icio.us API Change
• This is Hilarious
• Matthew West’s New Album
• I Love It, But It’s Not Available

I won’t go into all the sordid details here—you can read the white paper yourself—but the end results look something like this:

• Fibre Channel provided the highest throughput and the lowest processor utilization of all the storage protocols.
• Software iSCSI provided only slightly lower throughput than Fibre Channel (not more than 9% or 10% less than Fibre Channel depending upon the specific tests being run). However, software iSCSI consistently showed the highest CPU utilization on the ESX hosts.
• NFS showed throughput on the same levels as software iSCSI (again, not more than about 9% or 10% less than Fibre Channel depending upon the tests being run) and had higher CPU utilization than Fibre Channel. However, the CPU utilization was lower than with software iSCSI.

While overall performance was roughly comparable between all three storage protocols, depending upon the tests being run, the host CPU utilization was a different story entirely. In some cases, software iSCSI’s CPU utilization was as much as 80%—that’s right, almost double—that of Fibre Channel. In no cases did the CPU utilization drop below 40% higher than Fibre Channel. Keep in mind these numbers are relative to Fibre Channel. So if Fibre Channel used 200MHz of host CPU power and software iSCSI used 360MHz of host CPU power, that’s an 80% relative increase. We don’t know, unfortunately, how this translates into actual host CPU usage; in my mind, that’s a key piece of information that really should have been included. I’m puzzled as to why it’s not included.

NFS fared better; at its worst, the tests showed NFS running CPU overhead 40% greater than Fibre Channel. At its best, NFS looked like it was only requiring about 15% more CPU overhead than Fibre Channel (keep in mind the comments made above regarding relative utilization). Of course, NetApp loves to push the NFS; the document adds the extra sell for NFS:

While NFS does not quite achieve the performance of FC and has a slightly higher CPU utilization, it does have some advantages over FC that should be considered when deciding which protocol to deploy. Running on a standard TCP/IP network, NFS does not require the expensive Fibre Channel switches, host bus adapters, and Fibre Channel cabling that FC requires, making NFS a lower cost alternative of the two protocols. Additionally, operational costs are low with no specialized staffing or training needed in order to maintain the environment. Also, NFS provides further storage efficiencies by allowing on-demand resizing of data stores and increasing storage saving efficiencies gained when using deduplication. Both of these advantages provide additional operational savings as a result of this storage simplification.

I suppose I can’t blame them; NFS is one of their strong points, so they’ll naturally lean that direction.

There are a few key things that I need to say about this document, though:

1. Benchmark tests can be made to say just about anything. It’s all in the types of tests that you run and the parameters of those tests. I’m not saying that NetApp specifically skewed the tests in any way; what I am saying, though, is that users need to take these types of benchmark tests as a general guideline and not the definitive word.
2. While NetApp does highlight the “operational savings” of NFS, what they fail to mention is the added complexity of scaling NFS traffic as the environment grows. Fibre Channel multipathing in a VMware environment is very robust, and I expect that the Round Robin pathing policy will move from “experimentally supported” to fully supported rather quickly. This makes it quite easy to scale the FC connection, although to be honest that probably won’t be necessary. However, to scale the NFS connection, you need multiple NFS exports with multiple IP addresses, link aggregation via LACP/802.3ad/EtherChannel and switches that support cross-switch link aggregation, and possibly multiple VMkernel ports on different IP subnets. This is described, by the way, in the latest revision of TR-3428, also from NetApp. (As a side note, I believe that these scaling issues would affect any NFS storage vendor and are not specific to NetApp in any way.)
3. If you look at VMware’s development, you will see that Fibre Channel gets the goods the earliest. iSCSI and NFS were only added in VMware Infrastructure 3, whereas Fibre Channel support has been around in ESX for much longer. Storage VMotion support went to Fibre Channel first. VCB support went to Fibre Channel first. SRM support went to both iSCSI and Fibre Channel, but not NFS. Fibre Channel multipathing is, as I mentioned already, quite robust; iSCSI multipathing and NFS multipathing aren’t quite so robust. All these things considered, there could be a sound business case to use Fibre Channel in spite of cost savings from iSCSI (especially software iSCSI, given the added CPU overhead) or NFS. That’s something that each individual organization will need to decide for themselves.

By the way, I know the gentleman that wrote this technical report and he’s a straight-up guy. I respect him. So, don’t take any of my comments or thoughts to imply anything beyond the fact that I’m simply presenting my thoughts around the data contained in this document. You should also know that I am a fan of using NFS for VMware, but I don’t necessarily believe that it is the “slam dunk” that it’s often presented to be.

UPDATE: I’ve made some corrections to the interpretations of the CPU utilization numbers in response to some of the comments below.

• Proving VMware Over NFS
• Fibre Channel to Software iSCSI Failover Failures
• NFS for VMware Storage
• Using NetApp Deduplication with Block Storage
• Another Round with iSCSI and ESX Server 3

The problem is that as soon as the date hits August 12, 2008, all VMs refuse to power on and an error is logged indicating that the product’s license has expired. If VMs are already running, they should be fine; the problem, as I understand, only prevents users from powering on new VMs or performing VMotion operations.

If you are being affected by this bug, then your only real option at this point is to disable NTP (if it is being used) and set your date back a couple of days.

There’s no firm word yet on when a fix will be available from VMware.

Rince, thanks for the heads-up!

Update as of 9AM ET August 12, 2008

Updated ISOs and TARs for Update 2 (let’s call them Update 2 v2) will be available by noon PT tomorrow, August 13. In the meantime, the following workaround should help:

1. Disable NTPd on the ESX hosts.
2. Set the date on the ESX hosts back to some point prior to August 12, 2008. Some people have suggested using the same day and month but a completely different year in the past; this will make it easier to repair ESX log files using search/replace.
3. Set the VM to boot into the BIOS.
4. Boot the VM. In the BIOS, set the date and time properly. This will be incorrect because it gets inherited from the host. This is independent of the time sync functionality in VMware Tools. If you don’t fix this, VMs will boot up with the wrong date and/or time and that will cause additional problems (like Kerberos authentication within Active Directory).
5. After the VM has booted, ensure that time sync within VMware Tools has been disabled. Configure time sync to a reliable source. Active Directory usually handles this for Windows-based systems in a domain.

This workaround is only necessary to power on a VM, resume a suspended VM, or perform a VMotion operation. If all your VMs are currently up and running, just disable DRS.

Some have also suggested disabling HA. If you disable HA, then you will lose the ability for the VMs to be registered on another host if a host in the cluster fails. However, if you leave HA enabled, then without this workaround the VMs won’t boot anyway (but they will be registered). If you choose to leave HA enabled, just be aware that you will need to use the workaround in order for the VMs to actually power up after a host failure.

Hopefully this information will help. I’ll post more information as it becomes available.

Update as of 3PM ET August 12, 2008

VMware will apparently release an express patch to specifically address this issue by 6PM PT (9PM ET) today, August 12, 2008. As soon as I get word that the patch has been released, I will post an update here.

Update as of 11:15PM ET August 12, 2008

An express patch specifically to correct this issue has been released. It is available for download from VMware’s site. Also available there are links to new KB articles for ESX and ESXi that describe deployment and installation considerations.

Update as of 9AM ET August 13, 2008

I received word from VMware that the express patch released last night is fully compatible with VMware Update Manager. This should help simplify the deployment of the patch to affected servers.

• Security Fixes for ESX
• It’s Patch Tuesday Again
• Potential Problem with ESX 3.0.x and VirtualCenter 2.5
• Word Zero-Day Attack Alert
• Troubleshooting VMware HA Isolation Response

• The VI Team Blog posted a list of interesting things in Update 2. They identify things like Enhanced VMotion Compatibility (EVC) and monitoring and availability enhancements. For example, did you know that VM failure monitoring is now fully supported and not experimental? One thing that caught my eye when I first announced Update 2 was VSS support. What I hadn’t really noted about this was the fact that this extends into the application layer, meaning that VSS-aware applications will be quiesced for VCB-based backups. This brings application data consistency to VCB-based backups, and this is a big deal.
• And while we’re discussing the VI Team Blog, have a quick look at part 1 and part 2 of their tips for deploying VI. The discussion of Fibre Channel path selection policy with active/passive storage arrays is particularly helpful.
• Rich over at VM /ETC brought some interesting facts to my attention regarding the free version of ESXi in this post. I was not aware, for example, that the Remote CLI was read-only with free ESXi. Very interesting, and quite useful.
• Duncan at Yellow Bricks has also been pumping out some very helpful information, with an update on HA advanced options and a pointer to a document describing what happens if VirtualCenter crashes. One of the HA options that caught my eye was an option that allows VMotion interfaces to be used for HA. This means that we can easily provide additional HA redundancy simply by configuring VMotion interfaces. Handy!
• Chris Wolf brings to light information on PXE booting ESXi, and in so doing discusses the idea of the “stateless hypervisor.” This is an idea that is gaining a lot of ground, also being the subject of the release of a utility designed specifically around that very idea (but for ESX, not ESXi). When I design a VI environment, I’m already treating the hypervisor as mostly stateless; all the data and “important” information is stored on the SAN. To fully embrace the idea of a stateless hypervisor, we also need to incorporate auto-configuration.

That’s it for now. If any readers have any interesting links they’d like to share, please do so in the comments below. Thanks!

• Virtualization Short Take #15
• VMware Lab Manager 3.0 Released
• Apparent Date/Time Issue With Update 2
• Virtualization Short Take #12
• Virtualization Short Take #7