Recently, I’ve become aware of a rise in the number of technical industry / vendor certifications stressing the importance of communication with consumers of IT resources — call them customers if that sounds less clinical and makes you more comfortable. This got me thinking. Two such certifications currently live in my repository, so I believe I can speak from experience:

• VMware Certified Design Expert (VCDX)
• HP Master Accredited System Engineer – Converged Infrastructure Architect (MASE-CI)

Talk with the customers? That’s crazy talk! Honestly, I find myself trying to understand why this is such a revelation. I am not saying that I disagree with this direction; on the contrary, I see this as a critical component of any responsible design process. My challenge today is understanding what is all of a sudden driving the industry to emphasize a seemingly intuitive concept.

Background

In my background I have experience in both IT infrastructure support and software development. Because of the software development component, characteristics of a workload have always been near and dear to me: I was that crazy developer who needed to understand the footprint of my application on the infrastructure. In fact, I believed that I was unable to effectively develop for a platform unless I understood how that platform handled resources like CPU, memory, disk and network.

Remember when we had a 64K envelope in which to develop our applications? Imagine how much efficiency had to be coded into the applications to balance functionality with size. This created a serious learning curve for me whenever I was asked to develop for new platform. Sure, I could have just taken my existing code and “search and replace” ported it, but that wouldn’t have been very efficient and I was not about to compromise. Apparently, I can be kind of a pain to work with, but I like to think that I produced more efficient solutions tailored to specific platforms.

Efficiency

Speaking of efficiency, most of us in the IT industry are familiar with the “over-size” engineering concept: build a box big enough that whatever we try to put into it will fit. The potential for waste is pretty obvious, and became very evident during the x86 server sprawl of the late 90′s/early 2000′s. Virtualization (server, storage, network…) was the hammer brought to bear here to collapse silos of trapped capacity and allow sharing of resources without the need to change the over-sized boxes we put our stuff into.

Unfortunately, virtualization can only do so much to drive inefficiency out of an environment. At some point, the components which have been virtualized must be “right-sized” to accurately reflect the needs of the workloads running within them. Doing that requires understanding of what those workloads are doing, how they are used, and any cycles associated with that use: monthly batch jobs, specialized backup processes, or seasonal demand.

A Fundamental Difference

I don’t think anyone would argue that infrastructure teams have evolved with a different focus and mindset than development groups. Most infrastructure teams have grown up from “IT support” teams whose job was to keep the existing computer systems up and running. That’s it: make sure the file/print/mail/database server doesn’t go down… and make sure the PCs can talk to them. In my experience, a vast majority of teams retain this focus and understanding how the systems are being used is barely on their radar. I’ve heard loud and clear from a lot of “old school” IT support personnel that they really don’t (and shouldn’t need to!) care about how the platform they provide is being used:

“That’s not my job. I’m just here to provide infrastructure that is resilient. As long as it’s running, the users should be happy.”

I may be in a consulting position now, but I’ve been there and I can’t disagree enough. Sure, it is possible to build IT infrastructure without talking to consumers about actual requirements. I see it all the time: we’re building the network with a 10Gb core because we don’t want that to be the bottleneck. Or, similarly: we’re building a SAN and we need 16Gb FC and 75TB capacity so we don’t run out of space.

Maybe you can design and build the best network/storage/virtualization infrastructure on the planet, but does it meet the needs of your consumers? What if your consumers’ workloads are compute heavy and barely touch the existing 100Mb network and 2Gb SAN connections? You just spent your money in the wrong place and way over-built and overpaid for your amazing solution… in the wrong areas.

In development, we have a parallel concept: spend your time optimizing the code that will be executed the most. If I have code that is executed once a day and code that is executed several times a minute, where do you think I should spend my time optimizing? You got it. The typical challenge is in determining where to focus our energies. For that, you need requirements. From consumers.

KEY POINT: Don’t build something in a vacuum and just assume it will be useful.

As a developer, I’ve got a challenge that forces me down a certain path: how can I even begin to develop anything without first capturing some requirements? Sure, I can build an application or server shell, but what if the actual requirements for the project turn out to involve building a plugin for an existing system rather than a standalone solution? Yup, back to the drawing board.

Cloud?

When I think about it, this is a fundamental challenge of IaaS cloud solutions, aside from the more common ones: connectivity, security, portability, etc. Cloud models are all about sharing, agility and cost efficiency. In order to most effectively utilize resources and drive costs down in an IaaS cloud, understanding of the workloads within the compute units (VMs) is required. For many, though, that level of transparency is in direct violation of the whole ‘cloud’ model. How do we strike a balance between providers and consumers to ensure adequate performance for minimal cost while maintaining enough translucency to provide security?

Thought

Working with resource consumers to both gather requirements and convey what is possible using a given technology or platform is a key workflow for any software development activity. It should likewise be a critical path component for any infrastructure implementation, including specification of cloud solutions. Fortunately, it seems that the industry is catching on: becoming a better IT guy (or gal) requires skills in gathering requirements from the consumer. This is true whether you are a salesperson, consultant, or internal IT department resource.

Obtaining the required information from the workload owners is a whole different story and one I’d like to discuss in a future post.

If you’re like me, you are annoyed by the message warning that the certificate being presented by your vCenter machine is “untrusted.” Yeah, you all know the one I mean:

A little bit of OCD?

For me, I don’t like to just check the box at the bottom and pretend it doesn’t exist.  Aside from the fact that the checkbox doesn’t always make the messages cease, I like to have a real solution instead of sweeping it under the rug.  When you think about it, the team at VMware implemented certificates for a reason.  I’m sure it had mostly to do with encrypting the connection between vSphere clients, vCenter and the hosts, but those scary certificate things can do more, too!

Prerequisites

In this posting, I will focus on the VMware vCenter Management Appliance (VCMA) since the process for getting certificates registered on that appliance has been difficult to find.  If you want to know how to do it for ‘regular’ (i.e. Windows-based) vCenter, or the ESXi hosts, I think the processes are documented in the vSphere Security Guide – check Chapter 5 for more information.  (The actual generation of the key and CSR files and retrieval of the certificates follows the same process I outline here, but you push the files to the ESXi hosts to replace the rui.key and rui.crt files in the /etc/vmware/ssl directory.)

Also, for the sake of this posting, I will assume that you have some form of public key infrastructure (PKI) in place – that would be a certificate infrastructure used at your company – and the capability to have a server authentication-type certificate request fulfilled.  These certificates are commonly used by web servers, so being able to do that is probably a good indication that you can get the appropriate certificates created.  Creating a PKI just to make the warnings go away is a little crazy and actually does your environment a disservice: the CA needs to be a trusted authority within your organization and should be designed accordingly rather than stood up in an hour.  If you pay for all of your certificates to be issued from a public CA, that’s fine as well.

Getting off my soapbox and back to the task at hand, let’s make sure you have the following in your environment:

1. A vCenter Management appliance deployed on top of an ESXi host – it will run under Workstation 8. I haven’t tried previous versions, but this works if you want to play around.
2. puTTY, or your favorite SSH client – I like accessing the VCMA via SSH so I can paste the long commands into the console.
3. Some way to transfer the CSR and CER files to/from the VCMA.  Technically, you can copy and paste the text using puTTY, but I use FileZilla in my example.
4. A vSphere client installed
5. A compatible web browser
6. A properly functioning PKI, public key (certificate) infrastructure, is currently in place

High Level Overview

1. Deploy the VCMA from the OVF
2. Configure some initial parameters within the VCMA’s Linux OS
3. Create a certificate request (CSR) on the VCMA
4. Submit the CSR to the certificate authority (CA)
5. Apply the resulting certificate to the VCMA

This should be simple, right?

Arrival

I am assuming that step #1 has been completed.  It is best if you start with a fresh, clean VCMA in order to ensure that the process works.  Once your database has been populated, I think the process works, but I don’t know if there are any snags – caveat emptor.

2. Configure some initial parameters within the VCMA’s Linux OS

Set Hostname

In my testing, I’ve found that it is helpful to set the hostname and ensure name resolution on the VCMA prior to messing with anything else.  From the console of the VCMA, select the Login option and login as the root user (default username is root and the password is vmware – at least as of build 472350).

This is a good time to present my hostname and IP address — I’ll use these throughout the example.

• VCMA Hostname: vcma01
• VCMA Domain: itplab.local
• VCMA FQDN: vcma01.itplab.local
• VCMA IP Address: 192.168.199.128

Once logged in on the console as root, set the hostname and activate the change

# echo vcma01.itplab.local >/etc/HOSTNAME

# hostname –file /etc/HOSTNAME

Once that is done, logout and back in to make sure the prompt changes appropriately.  On the console, this looks something like the following:

I make a KEYS directory in the root user’s home directory to contain the files I’ll be using here. Feel free to do so, or not, but my examples assume this setup.

# mkdir KEYS


Name resolution

I always forget to do this, but if you don’t have DNS resolution configured, either do that or append a line to your /etc/hosts file to provide name resolution.

# echo 192.168.199.128 vcma01.itplab.local vcma01 >>/etc/hosts


I like to test pinging by name to ensure that I didn’t make a mistake.

3. Creating a Certificate Signing Request (CSR) on the VCMA

Now back to our regularly scheduled program. Switch to the KEYS directory and create the CSR.

# cd KEYS
# openssl req -out vcma01.csr -new -newkey rsa:2048 -nodes -keyout vcma01.key


Walk through the process, answering the questions as they apply to you.  Be careful to answer the Common Name (eg, YOUR name) prompt with the FQDN of the VCMA, not YOUR name.

..continued

Transfer the Certificate Signing Request (CSR) to the Certificate Authority (CA)

Once the CSR has been generated according to the process above, there will be two new files in your directory: a .csr file and a .key file.

You need to get the CSR file’s contents to your CA to request and create the certificate. FileZilla is a free way to do this using the already-running SSH process on the VCMA.  Provide the IP address or hostname, credentials, and port number for the connection (22):

Accept the SSH key for the connection — if you don’t check the box, it will only accept the key for this session.

If you like,  you can verify the fingerprint from the command line of the VCMA:

# ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub

Just check that the resulting fingerprint matches the one in the FileZilla box above and you’re good.

FileZilla presents a simple two-pane file manager – the VCMA is on the right and your local machine is on the left.  Navigate to the KEYS directory on the VCMA and pull the vcma01.csr file to your machine: drag it from the right pane into the left pane and wait for the transfer.

In the example, I got a little crazy with the mouse and copied both files – you only need the CSR.

4. Requesting the Certificate

In my environment, we have a Microsoft Windows Certificate Server deployed, so this process uses that infrastructure.

Browse to the web portal for the CA by using the proper URL.

This displays the following Welcome page. Note that you may need to authenticate before it gets there.

Click on the Request a certificate link in the Select a task list.

Depending on your access to the CA and the types of certificates it has been enabled to deploy, you may see different options here.  Regardless, you want to click on the advanced certificate request link.

Since we have already generated the certificate request (CSR), we want the second option here.  Don’t worry, even though the link text is longer, it means fewer boxes for us to fill out. Here, you want to select the Web Server certificate type from the Certificate Template drop-down.  If that certificate type is not in the list, your CA Administrator may have disabled it, or you may not be authorized to create that type – you’ll need to contact the appropriate person to get that resolved.

Open the CSR file from the VCMA in Wordpad, Notepad, or your favorite text editor, copy the entire contents – be sure to include the

-----BEGIN CERTIFICATE REQUEST-----

and

-----END CERTIFICATE REQUEST-----

lines.

…and paste it into the Saved Request box:

When you click Submit, the certificate request process implemented in your environment is set in motion.  If certificates require approval, you’ll have to get that process completed and come back once it is done.  Otherwise, the next screen will be presented.

Be sure to change the radio button from DER encoded to Base 64 encoded before continuing.  It just makes the process simpler.  Once that is done, click the Download certificate link and save the file somewhere that makes sense to you, and with a name that is easy to type.  I tend to use the VCMA’s hostname to keep things simple.

NOTE: In our lab environment, the root CA actually issues our certificates.  If you’ve got a mature or large PKI, there is normally an intermediate issuing CA that you’re accessing, and you may need to download the whole chain.  My assumption here is that your root and any intermediate CAs are already trusted by your client – i.e., your PKI is properly in place.

Once the certificate has been created and saved, use FileZilla to push the file onto the VCMA using the reverse of the process you used to download the CSR.

Checking the Certificate

From the VCMA CLI, verify that the files are where you think they are.

I always check the fingerprint of my certificates so I know which ones I’m replacing and which ones I’m installing.  An added benefit is that it ensures a proper transfer occurred:

# /usr/bin/openssl x509 -fingerprint -noout -in vcma01.cer

Aside: DOS line endings

One thing I noticed was that CER files from DOS machines came across with CR/LF line endings that tend to muck things up in Linux – when it comes to certificate files and keys, at least.  I realize that you can take care of this in a variety of ways – use your favorite method.

To see what I mean, use the vi editor to open the certificate

If you look at the bottom line (highlighted in red in the above image), it contains the filename in double quotes followed by [dos].  This means that the file contains CR/LF line endings used by DOS and Windows operating systems.  It may not seem like a big deal, but those hidden extras can bite you.

For fixing things up, I like awk.  With a simple command, you can replace all that DOS/Windows nastiness with *nix goodness:

# awk '{sub (/\r$/,"");print}' vcma01.cer >vcma01.crt

Check the new file with vi and notice that the [dos] is gone. Notice that I fixed the file into a new file rather than clobbering the original.  We’ll deal with the .crt file from this point forward.

Configure the VCMA

Accept the EULA

Before you can do anything useful with the VCMA, you must accept the EULA.  You can do that via the web interface of the VCMA https://YOUR_VCMA_IP_ADDRESS:5480, or from the command line.  We’re already there, so I use that.  If you prefer to click rather than type, go nuts.  I’ll wait while your web browser loads the UI and you login.

# vpxd_servicecfg eula accept

Accepting the EULA does all kinds of things for us, but the most important part is the VC_CFG_RESULT=0 line at the end.  That indicates that our vpxd_servicecfg command was successful.  Don’t worry about any of the other red or yellow stuff at this point.

Initialize the Database

Why do you have to configure the database now?  Well, if you don’t, the certificate swapping function in vpxd_servicecfg fails.  Beyond that, I have no idea. In this example, I’m using the embedded database.  You can use an external database and configure it via the command line as well, but I don’t have an external Oracle DB in my test environment.

# vpxd_servicecfg db write embedded

See those Host name lookup failure lines? As I mentioned before, I’d forgotten to configure name resolution during this run. Don’t worry, the database initialized fine. Be warned, however, that this can take a long time if you’re doing it against slow storage.  As before, the important line is the VC_CFG_RESULT=0.

As I mentioned before, I’m big into checking certificate thumbprints. The first two keys’ fingerprints should match, but will be different from the one you want to install (the third one):

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware/etc/lighttpd/server.pem

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware-vpx/ssl/rui.crt

# /usr/bin/openssl x509 –fingerprint –noout –in ~/KEYS/vcma01.crt

Making the certificate swap is accomplished with a single command

# vpxd_servicecfg certificate change vcma01.crt vcma01.key

Note that you will get an error, VC_CFG_RESULT=653, if you were jumping ahead (as in the following picture) and got the service started before we got the certificates swapped.

The following command will take care of that so you can get back on track:

# service vmware-vpxd stop

When all goes as planned, you get the VC_CFG_RESULT=0

Re-checking the fingerprints of the same files as before, we see that they all match, indicating that the certificate swap was a success.

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware/etc/lighttpd/server.pem

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware-vpx/ssl/rui.crt

# /usr/bin/openssl x509 –fingerprint –noout –in ~/KEYS/vcma01.crt

Go ahead and login to the VCMA with your vSphere client and you should no longer receive a certificate warning.

NOTE: I have noticed that the vSphere Web Client on the VCMA (https://YOUR_VCMA_IP_ADDRESS:9443/vsphere-client/) appears to use a different certificate.

That will have to wait for another blog post…

I’ve been meaning to write this post for a long time.  That’s nothing special since there are several posts that I’ve been meaning to write for quite a while.  As a quick aside, I have to highlight that I have a great deal of respect for @DuncanYB, @FrankDenneman, @scott_lowe, and the countless other bloggers who manage to post quality technical content just about every day – consistently and continuously.  Honestly, I don’t know how they do it.

Ever since the Windows Powershell cmdlets for VI3 were released, in the VMware community, there has been a friendly rivalry between the Perl and Powershell camps — I seem to recall an epic VMworld 2010 air hockey battle between @lamw (Perl) and @alanrenouf (Powershell):

Perl vs. Powershell - Epic Air Hockey Battle - Photo from @mattliebowitz

I realize that there are other ways to access the vSphere API these days, including Java, C# (via the vSphere Web Services SDK), and even Ruby (and a VMware Labs fling here).  Unfortunately, I lack experience with those versions, so I will restrict this post to Perl and Powershell.  If you’re interested about the SDKs in general, more information is available on VMware’s support site.

My Experience

Before I begin, I’d like to tell you a little about me so you know where I’m coming from.

Perl

I have been using Perl for various tasks since 1993 when I picked up a copy of Learning Perl (“the llama book”) by Randal Schwartz  so I could more easily manage a website I was working on at the time.  I had a slightly more scenic road than your average Perl novice since I was learning Perl on a Macintosh. Before they were UNIX-based or even had a command line. Fun? If you’ve ever embedded Hypercard XCMDs in a Perl script, you know what I’m talking about. I published something in 1998 when the AutoStart 9805 worm was running rampant on Macs around the world.  My code was not pretty, but the first version was written in an afternoon and did what I needed it to do.

Powershell

I did not encounter Powershell until 2008 when the VMworld scripting lab team offered me the opportunity to assist in developing the material.  Originally, I agreed because I thought my Perl experience would be helpful and I wanted to spend some time digging around the VI API.  As it turned out, there were already some Perl ninjas on the team and more help was needed with the Powershell material.  I am no @alanrenouf, but I spent many late night hours playing with both plain Powershell and the VMware add-ons so I could understand how the language was put together.  I learned a lot during those days, and even more helping attendees in the labs.

Bias?

I don’t feel that I have a language bias either way, and I tend to develop in whichever language is supported on the target platform(s).  Sure, I can run Perl on a Windows machine, but it still feels unnatural when Powershell is available natively.  Likewise, if it makes more sense to run a certain script from the vMA or from my Mac, I’ll use Perl.  I’ve spent some time on the VMware Community Forums translating from Perl into Powershell and vice versa.  You can do it!

Maybe it is just my perception because I learned Perl first, but I discovered that any significantly complex Powershell script looks a heck of a lot like Perl.  I’ll demonstrate this later.

Perl vs.(?) Powershell

As a team for the scripting lab, we came up with the (crazy?) idea to fork the single 2008 scripting lab into Perl and Powershell options that worked through the same exercises.  The cool thing was that I was able to experience both methods of accessing the virtual infrastructure to accomplish our defined tasks — attendees could go through the lab twice, or switch between languages as well.

While coding and testing the solutions to our lab exercises, it became clear that either language could be used to perform whatever action we required, and the complexity varied based on availability of Powershell cmdlets or pre-existing scripts and understanding of the API.  Some tasks were dead easy in Powershell but quite painful in Perl. Believe it or not, some tasks were easier in Perl because navigating the API’s object structure is common practice in Perl and just felt wrong in Powershell.  At least, until you got the hang of it.

Cmdlets and Scripts

From a Powershell perspective, the cmdlets included with the VMware Infrastructure Toolkit for Windows (now known as vSphere PowerCLI) provide convenient shortcuts for common actions.  The old VMware Infrastructure Perl Toolkit (now vSphere SDK for Perl), includes some pre-written scripts for performing common actions.  The added benefit of the Perl SDK is that the code for the scripts is available so you can see how they work, copy them, tweak, and repurpose as needed.

Interactivity

Hands down, PowerCLI provides superior interactivity.  Before you think I’m bashing Perl, let me explain.  Powershell was designed differently than Perl: it is an interactive shell that allows batch execution of commands while Perl is an interpreted language.  Think of Powershell more like your traditional command interpreters like csh, bash, ksh, or even (dare I say it?) DOS.

When using the Perl SDK, a per-script binding is created with the virtual infrastructure (login), the code is executed, and the binding is destroyed (logout).  Because Powershell is an interactive shell and PowerCLI extends that, a binding can persist beyond the lifetime of a single script’s execution.  In fact, it is possible to interact with the environment via that binding by executing cmdlets against the objects within the environment, accessed via the persistent binding or bindings.  Don’t misunderstand, it is possible to do some of this using the Perl SDK, but that’s not how it is normally used. Perl is like a swiss army knife or duct tape: if you can dream it, you can probably make Perl do it — thank you Larry Wall!

Along these lines, I noticed that VMware has grouped their various APIs and SDK-related tools according to the intended audience:

A Simple Example

All of that talk about theory and technical differences… but no code yet.  Who’s bored?  Okay, here comes my first set of examples.  In this one, based on exercises from the VMworld US 2008 Scripting lab, we compare the task of getting a list of all development VMs registered in a vCenter and report their current power state. To make things manageable, each development VM has the string “DEV” in its name:

Powershell

$vc = Connect-VIServer -Server MYVC
Get-VM *DEV*
Disconnect-VIServer -Server $vc -Confirm:$false

Perl

#!/usr/bin/perl -w
# Import runtime libraries
use strict;
use warnings;
use VMware::VIRuntime;
use VMware::VILib;

# Read and validate command-line parameters
Opts::parse();
Opts::validate();

# Connect to the server and login
Util::connect();

# Perform the requested action
my $List = GetVmByName('DEV');
PrintVmPowerState($List);

# Close server connection
Util::disconnect();

##############################################################################

sub GetVmByName {
  my $vmname  = shift;
  return Vim::find_entity_views(
  view_type => 'VirtualMachine',
  filter => {
    name => qr/$vmname/i
  });
}

sub PrintVmPowerState {
  my $vms = shift;
  foreach my $vm (@$vms) {
    print "Virtual machine " .
    $vm->name . " power state is: " .
    $vm->runtime->powerState->val . "\n";
  }
}

To execute this script, it needs to be saved into a file and called:
perl GetVmByName.pl --server MYVC --username MYUSER

That may look very complicated, but, to be fair, the only code that needs to be written is the two subroutines (under the line of hash marks, pound signs, capital 3′s … whatever you want to call them) and their calls from the main program.  The rest is a template created to handle the setup and teardown of the vSphere API connection.  I’ve also taken care to make this script readable – Perl ninjas can shrink this thing down to a few lines if that sort of thing is important.

So, what does that mean?

In the labs, we had several people come in without any scripting experience.  I was totally impressed – it was fantastic to see so much interest in scripting among the VI administrator community.  When we asked whether they were interested in Perl or Powershell scripting, we were asked most of the time for a suggestion.  We came up with one simple question:

1. What operating system(s) are you most comfortable with and which do you intend to use for running your scripts?

If the answer was Linux or MacOS we’d suggest Perl, otherwise we suggested Powershell.  We found that Powershell was much more accessible to people who had never written scripts before.  Sure, writing “hello world” in either Perl or Powershell is just as trivial, but accessing the vSphere API can be more daunting in the Perl world than Powershell’s simple Connect-VIServer cmdlet followed by Get-VM.  More success faster = happier new users.

What do I do?

I don’t keep the vSphere API in my head – I swear that @lamw does! I would probably be able to knock out a Perl script to do whatever I wanted pretty quickly if I spent a lot of my time writing that kind of code, but I forget things if I don’t use them regularly.  At this point in my life, I have few development responsibilities, so I may have short periods of scripting with LONG periods of other work in between.  Like many of you, I am typically looking for a quick, simple script to report on or change some aspect of my environment.  I need to be able to dust off my scripting brain and crank out the report. For that, PowerCLI works very well.

Better Together?

Looking back on my recent work, I noticed something interesting:  when I need to understand how an object “looks” in the API, I regularly jump to the PowerCLI console, even if I am going to write the final script in Perl.  Why would I do that?  The answer goes back to the interactivity of Powershell.  Sure, I could fire up a web browser and go to the vSphere API Reference or even go to the Managed Object Browser (MOB), but there is something about the command line that just feels better to me: I can see how my script will see the objects, minus any translation or filtering.

Examples

Reporting

In this case, I want to take a look at an Advanced Setting on my VMs to ensure that it is set properly. Setting it properly would be nice, too. Once I understand how the object is constructed and how I can access that data, I can construct either Perl or Powershell code to access the object.  Here, the vSphere API Reference or MOB are good for getting an idea of where I should look, but I like to poke around on my own sometimes just to see what’s out there.

My example VM is called “MYVM” and we’re looking at the “isolation.tools.paste.disable” setting. Here’s how I find it:

Powershell

PowerCLI C:\> $vmView = Get-VM MYVM | Get-View
PowerCLI C:\> $vmView

Capability           : VMware.Vim.VirtualMachineCapability
Config               : VMware.Vim.VirtualMachineConfigInfo
Layout               : VMware.Vim.VirtualMachineFileLayout
…

PowerCLI C:\> $vmView.Config

ChangeVersion                : 2011-12-20T16:10:43.434877Z
Modified                     : 1/1/1970 12:00:00 AM
Name                         : wintest2
GuestFullName                : Microsoft Windows Server 2008 R2 (64-bit)
Version                      : vmx-07
…
ExtraConfig                  : {nvram, virtualHW.productCompatibility, pci…
CpuFeatureMask               :
…

PowerCLI C:\> $vmView.Config.ExtraConfig

Key                           Value
---                           -----
nvram                         wintest2.nvram
virtualHW.productCompatibi... hosted
pciBridge0.present            true
sched.scsi0:0.throughputCap   off
disk.EnableUUID               true
pciBridge4.present            true
snapshot.action               keep
deploymentPlatform            windows
… (that's a LONG list... let's filter it!)

PowerCLI C:\> $vmView.Config.ExtraConfig | ? {$_.Key -like "isolation*"} | ft -a

Key                                  Value
---                                  -----
isolation.device.connectable.disable true
isolation.tools.connectable.disable  true
isolation.tools.copy.disable         true
isolation.tools.diskShrink.disable   true
isolation.tools.diskWiper.disable    true
isolation.tools.dnd.disable          true
isolation.tools.paste.disable        true
isolation.tools.setGUIOptions.enable false
isolation.tools.setinfo.disable      true

I now know that the value of the “isolation.tools.paste.disable” Advanced Setting is accessible on my VM at the following location: VM->Config.ExtraConfig[isolation.tools.paste.disable]  For the sake of this example, we’ll assume that the key is present and set to something (true).

Perl

I know I may take some heat from the Perl purists who know that they can achieve the same result using Perl SDK and some simple print/printf or Data::Dumper commands, but the above process is how I tend to work.  (For more information on using Data::Dumper with vSphere, you can go here.) Armed with an understanding of where the data that interests me lives, I can create some Perl code to perform the appropriate query in the same manner.  Note the process is pretty much the same as in Powershell:

1. Obtain a View of the VM
2. Grab the Config->extraConfig property
3. Find the key(s) in question
4. Print the value(s) associated with the key(s)

# Connect to the server and login
Util::connect();

my $vm_name = 'MYVM';
my $check_key = 'isolation.tools.paste.disable';
my $vm_view = Vim::find_entity_view(
view_type => 'VirtualMachine',
filter => {'name' => qr/$vm_name/i}
);

my $extraConf = $vm_view->config->extraConfig;

foreach(@$extraConf) {
  my $vm_key = lc($_->key);
  if($vm_key =~ m/$check_key/) {
    print "$vm_key\t" . lc($_->value) ."\n";
  }
}

# Close server connection
Util::disconnect();

Making Changes

Making changes is a little more difficult, but looks even more similar between languages:

Perl

my $vm_name = 'MYVM';
my $vm_view = Vim::find_entity_view(
  view_type => 'VirtualMachine',
  filter => {'name' => qr/$vm_name/i}
);

my $virtualMachineConfigSpec =
  VirtualMachineConfigSpec->new (
    extraConfig => [OptionValue->new(
      key => 'isolation.tools.paste.disable',
      value => 'true' ),] );

$vm_view->ReconfigVM( spec => $virtualMachineConfigSpec);

Powershell

$vmView = Get-VM MYVM | Get-View

$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.extraConfig += New-Object VMware.Vim.optionvalue
$vmConfigSpec.extraConfig[-1].Key = "isolation.tools.paste.disable"
$vmConfigSpec.extraConfig[-1].Value = "true"

$vmView.ReconfigVM($vmConfigSpec)

I hope this is useful, or at least interesting for people to see. If you made it this far, thanks for reading!

This is a continuation of Storage Migration with (and without) RDMs, part I

As of the last posting, we had identified the need to migrate a batch of vSphere 4.1 VMs configured with physical mode RDMs (pRDMs) from one storage array to another.  During this process, we decided that it would be nice to convert the (legacy) pRDMs to VMDKs in order to conform to the customer’s new standard configuration.

Doing them one at a time can be a lot of work, is error-prone, and I’m lazy when it comes to tasks that can be scripted.  There had to be some way PowerCLI could be brought to bear. What follows is our solution. Note that relocating and converting the pRDM to a VMDK must be done with the VM powered off. The script will check for that and let you know.

I based the following function (heavily) on Set-ThinDisk from the VMware vSphere PowerCLI Reference: Automating vSphere Administration.  You can get it on Amazon(US) here. I probably could have put this together without their framework, but it would have taken much longer and been nowhere near as elegant.

function Convert-RdmToVmdk { 
<#
.VERSION
   1.1 - 12/19/2011
.SYNOPSIS
   Converts an physical mode RDM to a thick provisioned VMDK hard disk
.DESCRIPTION
   Makes a thick provisioned copy of an RDM disk on a new
   datastore and configures the virtual machine to use the thick
   provisioned copy
.NOTES
   Author: Doug Baer
   Inspiration: Automating vSphere Administration by Luc Dekens,
             Arnim van Lieshout, Jonathan Medd,
             Alan Renouf, Glenn Sizemore
.PARAMETER HardDisk
   Specify the hard disks you want to convert
.PARAMETER datastoreName
   Specify the name of the datastore which will host the VMDK(s)
.PARAMETER credential
   A PSCredential object used to authenticate the VMHost server
.PARAMETER user
   The user account used to authenticate the VMHost server
.PARAMETER password
   The password for the account specified by the -User parameter
.PARAMETER replace
   Optional parameter to delete the original thick file
.EXAMPLE PS> Get-VM VM001 | Get-HardDisk | Convert-RdmToVmdk -Credential `
                  $hostCred -datastoreName "MyNewDatastore"
.EXAMPLE PS> $hd = Get-VM VM001 | Get-HardDisk | `
                 ?{$_.Name -eq "Hard disk 2"}
         PS> Convert-RdmToVmdk -hardDisk $hd -user "root" -password `
                "password" -replace  -datastoreName "MyNewDatastore"
#>

   Param ( 
      [parameter(valuefrompipeline = $true, Mandatory = $true, HelpMessage = "Enter a hard disk entity")] 
         [VMware.VimAutomation.ViCore.Types.V1.VirtualDevice.HardDisk]$hardDisk, 
      [Parameter(Mandatory = $true, HelpMessage = "Enter the target datastore name")] 
      [ValidateNotNullOrEmpty()] 
         [string]$datastoreName,
      [Parameter(Mandatory = $true, ParameterSetName = "cred", HelpMessage = "Enter a PSCredential object")] 
         [System.Management.Automation.PSCredential]$credential, 
      [Parameter(ParameterSetName = "user")] 
      [ValidateNotNullOrEmpty()] 
         [string]$user = "root",
      [Parameter(Mandatory = $true, ParameterSetName = "user", HelpMessage = "Enter the root account password")] 
         [string]$password, 
         [switch]$replace) 
   
   process { 
      if ($hardDisk.Parent.PowerState -eq "PoweredOff") { 
         if ($hardDisk.DiskType -eq "RawPhysical") { 
            Write-Host "...Connecting to host $esxhost"
            if ($credential) { 
               $esxHost = Connect-VIServer -Server $hardDisk.Parent.host.name -Credential $credential -NotDefault
            } else { 
               $esxHost = Connect-VIServer -Server $hardDisk.Parent.host.name -User $user -Password $password -NotDefault 
            }
            $vmdkFile = $hardDisk.Filename -replace ('\[.*\]',"[$datastoreName]")
            $datastore = $hardDisk.Filename.split('[')[1].split(']')[0] 
            $esxHardDisk = Get-HardDisk -server $esxHost `
               -Datastore $datastore `
               -DatastorePath $hardDisk.Filename
            Write-Host ("Copying RDM@ " + $hardDisk.Filename + "`n`t to VMDK " + $vmdkFile)
            Copy-HardDisk -HardDisk $esxHardDisk `
               -DestinationPath $vmdkFile `
               -DestinationStorageFormat "thick" | Out-Null 
            Write-Host "...Disconnecting from host $esxhost"
            Disconnect-VIServer $esxHost -Confirm:$false
               
         # Change the VM to use the VMDK instead of the pRDM
         $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
         $spec.deviceChange = New-Object `
            VMware.Vim.VirtualDeviceConfigSpec[](2)
         $spec.deviceChange[0] = New-Object `
            VMware.Vim.VirtualDeviceConfigSpec 
         $spec.deviceChange[0].operation = "remove" 
   
         if ($replace) { 
            $spec.deviceChange[0].fileOperation = "destroy" 
         } 
   
         $spec.deviceChange[0].device = $hardDisk.ExtensionData 
         $spec.deviceChange[1] = New-Object VMware.Vim.VirtualDeviceConfigSpec 
         $spec.deviceChange[1].operation = "add" 
         $spec.deviceChange[1].device = New-Object VMware.Vim.VirtualDisk
         $spec.deviceChange[1].device.key = -100 
         $spec.deviceChange[1].device.backing = New-Object VMware.Vim.VirtualDiskFlatVer2BackingInfo 
         $spec.deviceChange[1].device.backing.fileName = $vmdkFile
         $spec.deviceChange[1].device.backing.diskMode = "persistent" 
         $spec.deviceChange[1].device.backing.thinProvisioned = $false
         $spec.deviceChange[1].device.controllerKey = $hardDisk.ExtensionData.ControllerKey 
         $spec.deviceChange[1].device.unitNumber = $hardDisk.ExtensionData.UnitNumber 
         
         $vm = Get-View -Id $hardDisk.ParentID 
         Write-Host "...Reconfiguring VM to use VMDK"
         $vm.ReconfigVM_Task($spec) | Out-Null 
         } else { 
            Write-Host ("Virtual disk " + $hardDisk.Name + " is " + $hardDisk.DiskType + ", not a physical mode RDM")
         }
      } else {
            Write-Error "Virtual machine must be powered off" 
      }
   } #process
} #function

###

Note that the Copy-HardDisk cmdlet requires acting against a specific host, in vSphere 4.x, at least. I believe in vSphere 5.0, this requirement is gone and only the vCenter binding is required.

Once we created the host credential object (using Get-Credential) and connected to vCenter (using Connect-VIserver), we were able to use the function as follows:
Get-VM MyVM | Get-HardDisk | `
Convert-RdmToVmdk -Credential $hostCred -datastoreName "MyVMFSDatastore"
… as long as MyVM is currently powered off. This will walk through the Hard Disks attached to the VM and convert all of the encountered pRDMs to VMDKs on the datastore called “MyVMFSDatastore”.

To move different pRDMs to separate datastores, use something like the following:
$hd = Get-VM MyVM | Get-HardDisk | ?{$_.Name -eq "Hard disk 2"}
Convert-RdmToVmdk -hardDisk $hd -Credential $hostCred `
-datastoreName "MyOtherDatastore"


DISCLAIMER: As with any code, it is your responsibility to test the code’s suitability in your environment. I have successfully tested this function in our lab and have used it to migrate production workloads in a vSphere 4.1 environment — it works for me, but YMMV.

The Result
After running this script for each VM and powering them back up in the production environment, we manually cleaned up the old mapping files and un-presented the LUNs from the old array. So far, so good.

Issues?
When testing in my lab against a Windows 2008 R2 VM, I encountered an odd issue. Note that we did not encounter this in any of our production moves, and I have had a difficult time recreating it in the lab. However, I include it here in the event that someone runs into it — maybe it’ll save someone a little time.

The problem looked like this: Windows did not assign the drive letter back to the partition on the migrated disk and Disk Management console wouldn’t touch it. The error reported was, “The operation failed to complete because the Disk Management console view is not up to date.” I even tried using diskpart, but it wouldn’t even recognize that there was a partition on the disk. In my research of this issue, I turned up a lot of confused people on various support forums asking the same question, mostly with respect to removable drives.

The bottom line appears to be that Windows recognized a change in the device that confused the driver (kind of like the OS saying, “Hey, that’s not the same disk! What are you trying to pull here?” I came up with a workaround and what I think is a solution. Whatever you do, DO NOT FORMAT THE DRIVE!! — I don’t care what the articles on the forums say. (For the record, diskpart‘s clean [all] command is destructive) There were many, many people who formatted their disks while trying to figure this one out. I believe that ‘fixes’ the problem, but in a rather destructive way (i.e. you’ve got a working partition but no data).

Workaround
The workaround is to shut down the VM and change the SCSI ID of the VMDK — just bump it from SCSI0:2 to SCSI0:3, power up and Windows will recognize it properly (it won’t think you’re trying to pull something), mount the partition and assign the drive letter. Unfortunately, moving the VMDK back to the original SCSI ID resurfaced the problem.

Solution?
Digging around in Windows Device Manager when the VM exhibited the problem (I don’t love doing that) led me to what I think is a solution. In this instance, Windows apparently detected enough of a change in the disk to create another instance of a storage object, but only a partially functional one. Deleting this object and rescanning for new hardware (a good old-fashioned reboot would be best, of course) gave me my drive back.

Unfortunately (fortunately?) I was unable to duplicate the issue on demand in order to get a screenshot and record the ghost device’s name. In my various trials, the only ‘problem’ I could consistently create had to do with the VMDK-based drive needing to be ‘Onlined’ in Disk Management after the swap. Just be aware that it could show up — If I am able to re-create it, I will be sure to post screenshots here.

First, I’ve got to apologize for my lack of attention to this blog in the past months.  I have been working with a customer to integrate new storage into their environment and prepare for disaster-recovery enhancement across the board.  In fact, that engagement led me to the topic of this posting.

The Setup

So, as I mentioned above, over the past few weeks, I have been working with a customer to migrate data between two storage arrays.  In this case, they’re both HP EVA-class arrays: an older 6000 and a new P6500.  Staying within the same class of arrays tends to make the migration of certain data easier because array-to-array replication options exist.  In this phase of the engagement, the data was a mixture of physical Windows hosts and Windows virtual machines hosted on vSphere 4.1.

Windows 2003 and 2008

For the physical Windows hosts, we used array replication technology — HP Continuous Access EVA (CA EVA) — to replicate the data by nailing up a DR Group and allowing the primary and replica to synchronize while the hosts remained online servicing user requests.  Once the initial replication was complete and the arrays kept the Vdisks/LUNs in sync, we shut down each Windows box, flipped the DR groups to activate the replicas and suspended the original.  Once the copies on the new array were active, we brought each Windows host back online.  This is a pretty standard and well-documented process, and tends to run without issue as long as the correct steps are followed.  In this case, we didn’t even have to mess with reassigning drive letters or volume IDs on the Windows 2003 cluster machine!

vSphere – Two Flavors

For the vSphere-based VMs, it turns out that we had two distinct flavors of VMs: plain and pRDM.  For the plain VMs, we simply created new (matching) datastores on the target array and used Storage vMotion to migrate between arrays.  There was no downtime required for those VMs, and PowerCLI was used to automate the process somewhat:

Get-Datastore OLD-DSNAME | Get-VM | Move-VM -Datastore (Get-Datastore NEW-DSNAME)

For the VMs with physical mode RDMs (pRDMs), we explored a few different options:

1. Treat the Windows VMs with RDMs like physical Windows machines and migrate using array replication
2. Convert the RDMs to VMDKs and be done with it

The initial plan was to use CA to replicate the RDMs because that method was consistent with the rest of the machines and simplified our planning: fewer variables due to fewer variations in the process.  During our testing and POC, we ran into some interesting behavior using this process:

• HP CA maintains the WWN of the Vdisk when transferring it between arrays.  This results in the same NAA ID being assigned to the activated copy in vSphere, so the VM does not require a reconfiguration to function:
  • Shut down VM
  • Flip CA DR group (activate the copy as primary)
  • Rescan storage at the host level
  • Power up the VM

On the surface, this looks great and the VM powers up fine.  Unfortunately, something in the process caused vMotion to break.  This didn’t manifest itself right away and we probably wouldn’t have caught it if we weren’t simultaneously walking through the hosts in the cluster to upgrade RAM.

What Happened?

Everything seemed to work properly.  We even vMotioned the VM with the newly- migrated pRDM to another host in the cluster so we could shut the host down for upgrade. When the host was rebooted and we tried to put the VM back, it was no longer possible to vMotion the VM onto that host.  What? All we did was reboot!

The error we were seeing (red in the Migrate Virtual Machine window’s Compatibility pane) was this:

Virtual Disk ‘hard disk #’ is a mapped direct access LUN and its not accessible

As a test, we upgraded another host’s RAM and rebooted it.  Sure enough, that host was removed from the pool of vMotion compatible hosts for our VM.  Following standard vMotion compatibility troubleshooting methodology, we powered off the VM and cold-migrated it to one of the rebooted hosts.  We were able to power the VM up on that host without any problems.  When we attempted to use vMotion, the pool of possible hosts was the inverse of before: any host that had been rebooted was a possible target while the others were not.

After some digging using our trusty Google, we came upon the VMKB article # 1016210, which explains a related stuation:

vMotion compatibility for RDM LUNs is dependent on the vml identifier instead of the unique identifier (such as the NAA, EUI, or T10) for the LUN…

So, as it turns out, the vml ID of the pRDM hosted on the new array was slightly different from the original, but it wasn’t updated until the host was rebooted.  Typically, HP CA likes the host to be powered off when the Vdisk’s DR personalities are swapped and we thought that shutting down the VM would be good enough.  Apparently, it is not.

How did we proceed?

Troubleshooting this looked to be time-consuming, so we took a step back and looked at the reasons that these VMs were using pRDMs to determine where we should most effectively spend our time.  Unfortunately, the administrators who had originally created these VMs were no longer with the company, but there were none of the usual compelling reasons to implement pRDMs (SAN integration, MSCS clusters).

Our conclusion was that the machines were configured in this manner due to the perception that RDMs provided better performance than VMFS.  As it turns out, the current staff was considering a future migration away from the RDM configuration in order to simplify the environment, so we altered our migration process a little to accommodate that and eliminate the need for another outage in the future.

In my experience, there was one way to do this: use vmkfstools with the -i option to copy the pRDM through its VMDK mapping file into a new VMDK on a VMFS datastore.  This works fine, but requires logging into the ESX/i console to perform the action, and each RDM must be handled individually:

1. Shut down the VM
2. For each pRDM associated with the VM
   1. Document the current SCSI configuration (controller and address)
   2. Execute vmkfstools to copy the data
   3. Disconnect the pRDM mapping from the VM
   4. Connect the VMDK copy
3. Power up the VM

While we didn’t have hundreds of these things, there were some VMs that had several drives, and even a mix of VMDKs and pRDMs.  I’m big into automation and bigger into having a repeatable process that requires less thinking on my part during outage windows.  So, I tossed together a PowerCLI function to help here.  Fortunately, I have a copy of the excellent VMware vSphere PowerCLI Reference: Automating vSphere Administration written by Luc Dekens, Alan Renouf, Glenn Sizemore, Arnim van Lieshout, and Jonathan Medd, and was able to leverage some of their examples to accelerate my work.

In part II, I will present the Convert-RdmToVmdk function and explain how we used it to migrate our VMs with pRDMs.

Issue

I was having an issue with a newly-deployed VMware vSphere ESXi v5.0 host.  Everything looked fine, but when I pulled it into vCenter, it disconnected.

I’ve seen that happen before when there was “network weirdness” — you know, firewalls between vCenter and the host that were blocking some ports, bad DNS, broadcast storms — and I simply reconnected.  The host stayed connected, so I didn’t worry too much — must be a glitch of some sort.  However, when I went to configure NTP on the host, it disconnected again. Weird. I pulled a second host into the environment and IT DID THE SAME THING!  Okay, now I had to figure out what was going on.

I remembered that one of our customers (Thanks Scott!) mentioned something about this and they had talked to VMware and HP support about the issue.  I had deployed my host from HP’s custom ESXi 5.0 image — available by going to http://www.hp.com/go/vmware. The file I used was called 5.0_Aug_2011_ESXi_HD_USB_SDImgeInstlr_Z7550_00204.iso For the record, I find it crazy that the ESXi build number is not included in that filename — why not tack it on the end? Would that make the filename too long?

Logging into the host directly using the vSphere Client revealed that there was a problem writing to the “/etc/vmware/vmware.lic” file:

Write failed file:/etc/vmware/vmware.lic when adding license to host

Using the CLI on the ESXi host, I saw that the file was owned by root and permissions were Read Only (octal 444: r– r– r–).  I messed a little with those and tried to get the host to stay in vCenter, but that didn’t work: vpxuser wouldn’t touch the license file…

I saw possibly related error in vCenter on the host object:
Agent unable to save configuration to disk: Error syncing firmware configuration: vim.fault.TooManyWrites

That sounds REALLY bad!

Resolution

When all else fails, RTFM, right?  After reading the release notes (available as a separate download on the HP ESXi image page), I discovered that the HP-specific ESXi image has a license pre-installed so you don’t need to run in Evaluation mode.  Now, this is one of the ‘freebie’ vSphere Hypervisor license, so it can’t be managed by a vCenter that has a non-Evaluation license.

There’s an entry in the “Important notes and recommendations” section of the Release Notes:
“When the HP custom image for VMware ESXi 5.0 is downloaded and installed, problems will be encountered when applying a temporary or permanent license to the host server”

Yes, that sounds about right.  Ultimately, it appears that the license file integrated in the ESXi 5.0 image is improperly formatted and causes some problems.  I read the notes, which indicate how to REPLACE that license with a good one.  My fix was to yank the bad license since I have my own licenses to apply:

Either enable SSH and login to the host or connect to the host’s console, enable ESXi Tech Support Mode and remove the HP-installed license:

~ #
~ # esxcli software vib remove -n hp-esx-license --no-live-install
Removal Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true
VIBs Installed:
VIBs Removed: Hewlett-Packard_bootbank_hp-esx-license_1.0-03
VIBs Skipped:
~ #


Then reboot…

Sure enough, following a reboot the /etc/vmware/vmware.lic file had the correct permissions (octal 600: rw- — —) and the host remains connected in vCenter.

NOTE: There is an HP Customer Advisory on this topic, although it is past the “mid-September” date identified for a fixed image to be posted.

So, what’s cool about the HP Image of ESXi?
I get this question a lot.  Normally, right after, “What do you mean there is a custom HP image of ESXi?”

The HP ESXi 5.0 Customized Image includes the following:

• HP Management Tools:
  • HP CIM Providers
  • HP NMI Driver
  • HP iLO Driver
  • HP CRU Driver
  • HPONCONFG Utility
  • HPBOOTCFG Utility
• VMware IOVP Certified Device Drivers added to HP ESXi 5.0 installation images for HP device enablement.
• VMware’s Standard ESXi 5.0 image

The drivers are probably the biggest benefit that most users see immediately — no need to slipstream them into the install.

I’ve read several articles and blog postings about the ‘top VMworld takeaways,’ product announcements, or information garnered from the conference.  At this point, it seems almost redundant for me to toss my thoughts out to the world, but that’s not going to stop me.  As I mentioned in previous postings, I have attended a majority of the VMworld events, so maybe I do have a perspective worth sharing.

For starters, this was the largest VMworld ever. There were almost 20,000 people this year, up from 17,021 in 2010 — it would have been larger if not for the impact of Hurricane/Tropical Storm Irene on the eastern US.

Before I begin, I want to get it out there that I think it was very wise of VMware to make the vSphere 5 licensing change announcements prior to VMworld.  If they had waited to drop that bomb at VMworld, I think the atmosphere would have been very different — in a bad way.  As it was, I don’t think I heard anyone talking about the licensing changes during the conference.  Nice job to whomever architected that announcement.

For me, these conferences are more about reconnecting with colleagues and hearing VMware’s vision than the technology itself.  Believe me, I never thought I would say that, but I am more interested now in how we’re going to apply the technology than how the technology works. Don’t get me wrong, I still care a lot about how it works, but I also realize that amazing technology by itself doesn’t provide value.  This year, I spent a lot more time in the Solution Exchange than I had in the past.  I noticed a few trends this year:

• There are a lot of products designed to provide better storage performance to virtualized environments.  If you’ve got VMs, you know that they place unique demands on storage systems, and there are many ways to skin that cat.
• Many of the larger booths were the hardware OEMs, but just as many seemed to be companies providing management, monitoring, and automation software — that’s a good sign.
• A lot fewer companies with “cloud” in their names, but a lot more providing something as a service (XaaS).  Has the industry realized that ‘cloud’ is essentially providing services and that the word ‘cloud’ is terribly overused?

From a product perspective, there is a lot of excitement about End User Computing (EUC) this year, specifically around Desktop enablement.  My opinion is that virtual desktops are simply a bridge between individual user desktop machines and an application-centric user computing environment.  Along those lines, it was nice to hear VMware execs indicate that perhaps the desktop metaphor is no longer as relevant as it was back in the 80′s.  Obviously, some kind of desktop-based environment will remain as long as useful applications have not been rewritten to take advantage of the new desktop-less paradigm, but I think we’re moving in the right direction.  Should the IT department really care what device I am using? Do they need to be responsible for securing an entire OS if I’m not using all of those components?

<soapbox>Arguably, bringing my own device implies that I have accepted a lot of the responsibility for securing my device, and I should work with the IT folks to achieve the common goal of a secure environment. </soapbox>

With technologies like Project Octopus, Horizon Mobile, ThinApp Factory, and AppBlast, it is apparent that VMware is charging into the EUC market full force.  Before you get too excited, remember that most of these are enabling technologies, technology previews, or pre-announcement demonstrations meant to showcase what VMware is currently working on.  Who knows, several of these may be productized and ready for proper announcement by VMworld Europe (October 18-20, 2011), but most of them are not available for purchase now.

For me, the labs are always a highlight, and technologies that support fundamental changes in both infrastructure use management are more interesting than the end user stuff.  I’m mostly an infrastructure guy, but I was particularly impressed with vFabric Data Director and SQLFire:

• vFabric Data Director introduces Data as a Service (DaaS?) to the VMware stack and allows DBAs to provision their own databases without requiring involvement from the server teams — the OS underlying the database has become a non-issue in this implementation.  If you’ve ever spent time tuning, patching, and troubleshooting OSes that support database servers, you’ll comprehend how much operational savings is possible here.  Today, DD supports a tuned and vSphere-optimized version of PostgreSQL, but plans are in the works to support Microsoft SQL and Oracle in a similar manner.  You can get more information about Data Director here.
• vFabric SQLFire is an in-memory SQL implementation based on the GemFire technology that VMware acquired last year.  The goal is to provide extremely fast access to data for highly-transactional workloads without the need to rewrite an application to support a specialized database coding.  Additionally, it supports continuous availability within or across data centers via a horizontal scaling model. From an infrastructure perspective, this is very attractive since the application can handle its own availability requirements internally. You can get more information about SQLFire here.

These two were the gems of the show for me, although the vFabric Cloud Application Platform is also very intriguing. I’ve been waiting for something like that since VMware acquired SpringSource, but I ran out of time during the show and did not have the chance to dig into it.  I will definitely need to pull the session slides and see what is going on there.

From a local perspective, we have Tech Talks later this month that will specifically cover the conference, sessions and announcements in greater detail.  If you’re interested in hearing more, contact your IT Partners account executive and we’ll get you the details.

Finally, it was great to reconnect with people from my past, the community in general (forums, blogs, twitter, past presenters) and my fellow VCDXes.  For those I missed, I look forward to catching up next year in San Francisco.

The following year, 2008, I worked on a VMware Professional Services (PSO) engagement with Richard Lebedeff and Jeff Baylor. I worked on the VCB portion of a large VI3 P&D engagement and it was interesting to witness how VMware PSO worked; these guys are professionals

Later that year, I was connected to David Deeths of VMware who was the ‘lab captain’ in charge of the team responsible for the scripting lab at VMworld 2008. That meant the Perl SDK and the freshly-released VMware PowerShell Toolkit. Since I had both VMware Infrastructure experience and a software development background, joining that team made sense. I was lucky to work with a fantastic team made up of VMware employees Shridhar Deuskar, Lisa Guinn, Terry Lyons, Aaron Miller, Alket Memushaj, Brian Watrous, and Alton Yu in addition to Owen Thomas from New Age Technologies. I had the privilege of meeting and working with the team for a couple of days at VMware’s Promontory campus and we created two complete labs: one in Perl and one in PowerShell. Our goal was to work through the same exercises in both languages so that attendees could compare the toolkits in an apples-to-apples manner. I learned a lot of PowerShell and had a tremendous amount of fun working with the VMworld Lab Staff and assisting students in our sessions. I think our lab was one of the top-rated sessions that year and you can still download the manuals on the VMworld.com website.

In 2009, we got (most of) the band back together and revised the scripting labs. We added Phil Anthony, Chirag Patel, and Josh Thomas – we even had Carter Shanklin as a guest presenter and Yavor Boychev of Project Onyx fame around for additional support. With the expansion of VMware’s product line, I don’t think there was enough space for two labs on scripting, so we combined, updated, and extended the Perl and PowerCLI labs from 2008 and allowed students to select a language when they attended the lab. Logistics were a little funny since we had to pair up Perl and PowerShell attendees, but it mostly worked out. Not cloud, but getting there.

This was also big year for me because I submitted a design and application for the VMware Certified Design Expert (VCDX) defense. I arrived in San Francisco a few days prior to VMworld to go through the defense and to get the lab environment prepared for the show. My defense took place at the Promontory campus, so at least I knew how to get there. None of the candidates really knew what to expect – there weren’t any blog posts back then to help us out and we had to wing it. Builds character, right? It was quite an experience. The following week at VMworld, I received an email, stopped by the onsite testing center, and was told that I’d passed… Whew!

With the focus on “cloud” in general and self-service in particular, the VMworld 2010 labs were reformatted and no longer required dedicated presentation staff for each one – or, it was more cost effective to have VMware employees handle the tasks rather than bringing in outside (Partner) resources. I was seriously bummed not to be part of the labs, but we had worked out a partnership with a small company that was coming out of “stealth mode” during VMworld, so I had booth responsibilities. Unfortunately, that company dissolved just before the conference and I was left as “just an attendee” for the first time ever.

To be honest, I didn’t know at first what to do about my free time, but I found all kinds of sessions that interested me and had been asked by the folks at VIBriefing.com if I would blog about the conference. We set up a feed using an Evernote notebook and Mariah West took care of making sense of my ramblings and posting them to the web. I hear they got pretty good traffic, so that was nice. I stopped by the massive VMware lab room to see how my VMware-badged lab veterans were doing, to catch up, and to snap some pictures of the environment. Also at this show, I attended a lunch with Paul Maritz (I got to sit at the same table!), met most of the current VCDXes and Susan Gudenkauf (VCP #1!), and received a jacket for being one of the first 50 VCDX-certified individuals. It was especially cool to finally meet Duncan Epping in person and talk a little with John Arrasjid, whose sessions I had attended since the beginning of my VMworld journey.

This year, I looked forward to seeing a lot of my past coworkers, current customers, and VMware-badged friends at the show. I ran into Jim Rast who was a member of the original Phoenix VMUG — it was nice to discuss current technology applications and customer challenges. I tracked down Frank Denneman in person, having worked with him a little this past year on the vSphere 5 Clustering Technical Deep Dive book that he and Duncan published. Those guys gave me a signed copy of the book; very nice. It was interesting to catch up with Andreas Groth via Twitter and realize that we had presented together at another VMworld so many years ago. As an unexpected bonus, I caught up with Paul Strong, VMware’s CTO for Global Customer & Field Initiatives. He’s quite possibly the smartest guy I’ve ever talked to and I look forward to doing it again soon.

I’m certain that I haven’t mentioned all of the people I’ve encountered over the years, and I apologize to those whom I have not mentioned – there have been so many and this post is already a lot longer than I thought it would be. As for the guys from the local Phoenix VMware team that I work with most often: Jared Byrd, George Peck, I look forward to seeing you again soon, too.

Wow, I guess this qualifies me as the ultimate VMware fan boy, doesn’t it? Looking forward to next year…

I was unaware of VMworld at this point, but Christine Holland got me in touch with Rob Smoot, who I believe is still with VMware. Rob was organizing a panel discussion at this VMworld conference in Las Vegas and wanted someone to talk about using their products for testing and development. I was excited to be a part of the group that included some much bigger implementations, and I hoped that I could add some value to the panel discussion and Q&A. That year, I also received an award for my part in the Core Customer program. I believe I had the second highest number of ‘points,’ losing out to Steve Beaver, a legend in my mind, at least on the VMTN Forums. Unfortunately, he was not present, so I never got to meet him. However, I was privileged enough to sit next to Diane Greene at the recognition luncheon. Only my first VMworld and I’d already gotten to meet the CEO. How cool was that!

I decided to pursue certification (VCP2), and attended the ESX2 and VirtualCenter Installation and Configuration class. Based on my level of experience, the class was a lot of review, but did a lot to help solidify my understanding of the fundamentals. My instructor, Damian Wraa, was excellent, and I believe he is still with VMware today. I think it says a lot that I remember his name since I only saw the guy for a week during a class over 6 years ago!

VMware was growing like crazy (doubling in size every month, it seemed), and a sales team was built in the southwest region where I was working. Kenon’s responsibilities were directed elsewhere and Shak Malik was assigned as my new VMware SE, along with Eric Rakotz as a local sales contact. Both Shak, who works for HP now, and Eric, now VCE, took good care of me, although I don’t think I required too much care and feeding. I continued my relationship with Chris Holland and the Core Customer Program, became more involved on the VMTN forums (I got my chess piece!) and our little VMUG. The VMUG here in Phoenix was originally run by Jason Ambrose and Josh Wright from Agilysis, and we had meetings in a small conference room at the HP office. Back then, we had a presentation from any vendor who would buy us sodas and sandwiches (for 10 people or so).

By 2006, I discovered that I enjoyed talking to people about the benefits of server virtualization and guiding them through the process of justification, design and implementation much better than my operational IT responsibilities. Through VMware, I worked with the team at OutCast Communications on a few media activities – telling my story to reporters, discussing VMware’s upcoming features, and how I would implement them. Here, I met Andrew Schmitt (OutCast), Sarah Bresee (OutCast), Amber Rowland (VMware), and Karthik Rau (VMware) – all of them are great people to work with and I had a lot of fun promoting VMware’s products. It was a tough decision, but I turned in my Customer status at the end of 2006 to become a Partner. The decision was made so I could focus my efforts, further develop my skills, and help more people by spreading the VMware message even more. That year, I was also asked to step in and present a session at VMworld in Los Angeles on upgrading to VI3. It was a last minute thing and I scrambled to come up with the content. I ended up with twice the amount of content than could be presented in a 45 minute period, and a few of my customers and coworkers were there to heckle me. Overall, a great time again.

I had such a great time that I submitted my own session idea in 2007 regarding different infrastructure layers where virtualization technologies could be implemented. Apparently, a few others had a similar idea and a session called “Virtualization Architectures, Options and Approaches” was born. I was honored to be in the company of Scott Davis (VMware), Ram Rao (HP) and Andreas Groth (IBM). That session was incredibly popular: standing room only and a Fire Marshall’s nightmare. We ran long and had to be kicked out of the room – it rocked! Each of those guys are amazing in their own way and we could have done a 2 hour session with ease. That year, I randomly had the opportunity to grab a few minutes with Dr. Mendel Rosenblum while he was waiting for Diane Greene on their way out of town and back home. Diane was like a rockstar or the President: she needed people to flank her and extract her from conversations so that she could make it to her appointments on time.

With that kind of start, how could I NOT want to continue to be a part of this community?

Sitting here on the eve of my 7th VMworld conference (yeah, I missed the first one in 2004), I think back on VMware journey up to this point and the people who have influenced me along the way.  I’m a fairly inconspicuous guy, so I don’t get recognized a lot, even though I’ve been around the VMware scene for quite some time.

In the beginning, I used VMware Workstation (2.x) when I was a software developer.  None of us ever wrote buggy code and had to reload Windows (3.1!) onto our development machines, but Workstation was there as a testing environment, just in case.  What a timesaver!  To be honest, my history with VMs goes back before that with RealPC, VirtualPC, SoftPC, and the other emulators that ran x86 machines on Macintoshes (it worked, but was NOWHERE near fast).  I always wanted something like that when I was developing software on my Mac – but that never happened. At least, not during my developer days: vSphere 5 allows it if all of the prerequisites are met!

In 2002, I continued my use of Workstation (3.x) during project planning and new software testing when I moved from the app/dev side of the house to infrastructure.  In those days, I did a lot of directory services work, and testing integration plugins or new Netware client builds/patches without nuking my production machine’s configuration was the primary use case.  Moving beyond my own machine, we licensed GSX Server for Windows in 2004 and collapsed two racks of lab hardware onto a pair of servers.  Our lab room immediately became much quieter, cooler, and more available.  Those who understand the ‘waterfall’ model of lab gear acquisition get that labs are regularly built from the oldest junk in an organization — and hardware support is not even within the realm of possibility.

After such great success in our lab, and with the 2.0 release of VMware ESX Server, we decided to take a look at what that could buy us beyond our GSX Server – we’d heard good things about less overhead and higher densities.  Back in those days, you were required to engage VMware resources in order to even try out ESX Server.  Enter Melissa Ercoli and Kenon Owens – these people were my first experience with corporate VMware, and helped me get started.  I believe Melissa got married and is still with VMware while Kenon has since moved on to work at Microsoft.  Both of these individuals invested the time with me to ensure that I understood this new ESX Server thing by answering all of my questions, both technical and licensing-related.  For that, I am very grateful.

In late 2004, armed with an understanding of the ESX Server architecture, and a trial implementation in the lab, I submitted a proposal to our management to stand up a POC environment in production – 2 SAN-attached ESX Server 2.5 hosts.  I was immediately met with the “all of our eggs in one basket” argument.  Fortunately for me, VirtualCenter 1.0 had been released and I was on top of the amazing VMotion (notice the capital “V”!) technology enabled therein.  My POC environment was kind of sad – we basically took two of our standard x86 servers (HP DL360 G3), bumped the RAM a bit, and shoved an FC HBA into it.  There was pretty much no redundancy (single power supply, single SAN path, single NIC port for Service Console, one for VMotion, and two for VM traffic), but the thing worked.  My first demo of VMotion blew everyone’s mind: “there’s NO WAY that just worked!”

In August of 2004, a hardware failure on a production server expedited the migration of our VMware POC into production.  We were presented with the option of blowing away the POC environment in order to repurpose its hardware as a replacement for the failed box, or repurposing a VM currently running on that environment.  The path of least resistance, and quicker TTR, was obvious, and we became production ESX Server and VirtualCenter users.  From there, the thing spread like wildfire, just like the stories we hear today — you know, the Legend of VMware.

This stuff was getting to be fairly complicated and I had invested a lot of time into understanding how the pieces fit together.  I registered on the VMware Community forums in October of 2004 in order to get answers to my more complicated questions and to share my experiences with others.  First off, I was amazed at the sense of community — people helping people without being compensated beyond the satisfaction of sharing knowledge.  This was a group I immediately liked quite a bit.  It was there that I ran across the VMware Core Customer program, which ultimately led me to Christine Holland.  Chris was interested in my environment and how we were using VMware’s technologies in both production and testing, so she introduced me to the PR people and they quoted me as part of the ESX Server 2.5 release.  We had a case study published in early 2005 and a related article as well.

From there, Charles Babcock (Network Computing) and Jennifer Mears (Network World) talked to me a bit about the ESX 3.0 and VirtualCenter 2.0 launch, and the much-anticipated HA and DRS features. I had some coverage there, including a quickly-snapped photo of me at VMworld. I’ve got to say it was pretty cool to get a magazine with my (goofy) picture in it!

To be continued…