IT Partners Blog

If you’re like me, you are annoyed by the message warning that the certificate being presented by your vCenter machine is “untrusted.” Yeah, you all know the one I mean:

A little bit of OCD?

For me, I don’t like to just check the box at the bottom and pretend it doesn’t exist.  Aside from the fact that the checkbox doesn’t always make the messages cease, I like to have a real solution instead of sweeping it under the rug.  When you think about it, the team at VMware implemented certificates for a reason.  I’m sure it had mostly to do with encrypting the connection between vSphere clients, vCenter and the hosts, but those scary certificate things can do more, too!

Prerequisites

In this posting, I will focus on the VMware vCenter Management Appliance (VCMA) since the process for getting certificates registered on that appliance has been difficult to find.  If you want to know how to do it for ‘regular’ (i.e. Windows-based) vCenter, or the ESXi hosts, I think the processes are documented in the vSphere Security Guide – check Chapter 5 for more information.  (The actual generation of the key and CSR files and retrieval of the certificates follows the same process I outline here, but you push the files to the ESXi hosts to replace the rui.key and rui.crt files in the /etc/vmware/ssl directory.)

Also, for the sake of this posting, I will assume that you have some form of public key infrastructure (PKI) in place – that would be a certificate infrastructure used at your company – and the capability to have a server authentication-type certificate request fulfilled.  These certificates are commonly used by web servers, so being able to do that is probably a good indication that you can get the appropriate certificates created.  Creating a PKI just to make the warnings go away is a little crazy and actually does your environment a disservice: the CA needs to be a trusted authority within your organization and should be designed accordingly rather than stood up in an hour.  If you pay for all of your certificates to be issued from a public CA, that’s fine as well.

Getting off my soapbox and back to the task at hand, let’s make sure you have the following in your environment:

1. A vCenter Management appliance deployed on top of an ESXi host – it will run under Workstation 8. I haven’t tried previous versions, but this works if you want to play around.
2. puTTY, or your favorite SSH client – I like accessing the VCMA via SSH so I can paste the long commands into the console.
3. Some way to transfer the CSR and CER files to/from the VCMA.  Technically, you can copy and paste the text using puTTY, but I use FileZilla in my example.
4. A vSphere client installed
5. A compatible web browser
6. A properly functioning PKI, public key (certificate) infrastructure, is currently in place

High Level Overview

1. Deploy the VCMA from the OVF
2. Configure some initial parameters within the VCMA’s Linux OS
3. Create a certificate request (CSR) on the VCMA
4. Submit the CSR to the certificate authority (CA)
5. Apply the resulting certificate to the VCMA

This should be simple, right?

Arrival

I am assuming that step #1 has been completed.  It is best if you start with a fresh, clean VCMA in order to ensure that the process works.  Once your database has been populated, I think the process works, but I don’t know if there are any snags – caveat emptor.

2. Configure some initial parameters within the VCMA’s Linux OS

Set Hostname

In my testing, I’ve found that it is helpful to set the hostname and ensure name resolution on the VCMA prior to messing with anything else.  From the console of the VCMA, select the Login option and login as the root user (default username is root and the password is vmware – at least as of build 472350).

This is a good time to present my hostname and IP address — I’ll use these throughout the example.

• VCMA Hostname: vcma01
• VCMA Domain: itplab.local
• VCMA FQDN: vcma01.itplab.local
• VCMA IP Address: 192.168.199.128

Once logged in on the console as root, set the hostname and activate the change

# echo vcma01.itplab.local >/etc/HOSTNAME

# hostname –file /etc/HOSTNAME

Once that is done, logout and back in to make sure the prompt changes appropriately.  On the console, this looks something like the following:

I make a KEYS directory in the root user’s home directory to contain the files I’ll be using here. Feel free to do so, or not, but my examples assume this setup.

# mkdir KEYS


Name resolution

I always forget to do this, but if you don’t have DNS resolution configured, either do that or append a line to your /etc/hosts file to provide name resolution.

# echo 192.168.199.128 vcma01.itplab.local vcma01 >>/etc/hosts


I like to test pinging by name to ensure that I didn’t make a mistake.

3. Creating a Certificate Signing Request (CSR) on the VCMA

Now back to our regularly scheduled program. Switch to the KEYS directory and create the CSR.

# cd KEYS
# openssl req -out vcma01.csr -new -newkey rsa:2048 -nodes -keyout vcma01.key


Walk through the process, answering the questions as they apply to you.  Be careful to answer the Common Name (eg, YOUR name) prompt with the FQDN of the VCMA, not YOUR name.

..continued

Transfer the Certificate Signing Request (CSR) to the Certificate Authority (CA)

Once the CSR has been generated according to the process above, there will be two new files in your directory: a .csr file and a .key file.

You need to get the CSR file’s contents to your CA to request and create the certificate. FileZilla is a free way to do this using the already-running SSH process on the VCMA.  Provide the IP address or hostname, credentials, and port number for the connection (22):

Accept the SSH key for the connection — if you don’t check the box, it will only accept the key for this session.

If you like,  you can verify the fingerprint from the command line of the VCMA:

# ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub

Just check that the resulting fingerprint matches the one in the FileZilla box above and you’re good.

FileZilla presents a simple two-pane file manager – the VCMA is on the right and your local machine is on the left.  Navigate to the KEYS directory on the VCMA and pull the vcma01.csr file to your machine: drag it from the right pane into the left pane and wait for the transfer.

In the example, I got a little crazy with the mouse and copied both files – you only need the CSR.

4. Requesting the Certificate

In my environment, we have a Microsoft Windows Certificate Server deployed, so this process uses that infrastructure.

Browse to the web portal for the CA by using the proper URL.

This displays the following Welcome page. Note that you may need to authenticate before it gets there.

Click on the Request a certificate link in the Select a task list.

Depending on your access to the CA and the types of certificates it has been enabled to deploy, you may see different options here.  Regardless, you want to click on the advanced certificate request link.

Since we have already generated the certificate request (CSR), we want the second option here.  Don’t worry, even though the link text is longer, it means fewer boxes for us to fill out. Here, you want to select the Web Server certificate type from the Certificate Template drop-down.  If that certificate type is not in the list, your CA Administrator may have disabled it, or you may not be authorized to create that type – you’ll need to contact the appropriate person to get that resolved.

Open the CSR file from the VCMA in Wordpad, Notepad, or your favorite text editor, copy the entire contents – be sure to include the

-----BEGIN CERTIFICATE REQUEST-----

and

-----END CERTIFICATE REQUEST-----

lines.

…and paste it into the Saved Request box:

When you click Submit, the certificate request process implemented in your environment is set in motion.  If certificates require approval, you’ll have to get that process completed and come back once it is done.  Otherwise, the next screen will be presented.

Be sure to change the radio button from DER encoded to Base 64 encoded before continuing.  It just makes the process simpler.  Once that is done, click the Download certificate link and save the file somewhere that makes sense to you, and with a name that is easy to type.  I tend to use the VCMA’s hostname to keep things simple.

NOTE: In our lab environment, the root CA actually issues our certificates.  If you’ve got a mature or large PKI, there is normally an intermediate issuing CA that you’re accessing, and you may need to download the whole chain.  My assumption here is that your root and any intermediate CAs are already trusted by your client – i.e., your PKI is properly in place.

Once the certificate has been created and saved, use FileZilla to push the file onto the VCMA using the reverse of the process you used to download the CSR.

Checking the Certificate

From the VCMA CLI, verify that the files are where you think they are.

I always check the fingerprint of my certificates so I know which ones I’m replacing and which ones I’m installing.  An added benefit is that it ensures a proper transfer occurred:

# /usr/bin/openssl x509 -fingerprint -noout -in vcma01.cer

Aside: DOS line endings

One thing I noticed was that CER files from DOS machines came across with CR/LF line endings that tend to muck things up in Linux – when it comes to certificate files and keys, at least.  I realize that you can take care of this in a variety of ways – use your favorite method.

To see what I mean, use the vi editor to open the certificate

If you look at the bottom line (highlighted in red in the above image), it contains the filename in double quotes followed by [dos].  This means that the file contains CR/LF line endings used by DOS and Windows operating systems.  It may not seem like a big deal, but those hidden extras can bite you.

For fixing things up, I like awk.  With a simple command, you can replace all that DOS/Windows nastiness with *nix goodness:

# awk '{sub (/\r$/,"");print}' vcma01.cer >vcma01.crt

Check the new file with vi and notice that the [dos] is gone. Notice that I fixed the file into a new file rather than clobbering the original.  We’ll deal with the .crt file from this point forward.

Configure the VCMA

Accept the EULA

Before you can do anything useful with the VCMA, you must accept the EULA.  You can do that via the web interface of the VCMA https://YOUR_VCMA_IP_ADDRESS:5480, or from the command line.  We’re already there, so I use that.  If you prefer to click rather than type, go nuts.  I’ll wait while your web browser loads the UI and you login.

# vpxd_servicecfg eula accept

Accepting the EULA does all kinds of things for us, but the most important part is the VC_CFG_RESULT=0 line at the end.  That indicates that our vpxd_servicecfg command was successful.  Don’t worry about any of the other red or yellow stuff at this point.

Initialize the Database

Why do you have to configure the database now?  Well, if you don’t, the certificate swapping function in vpxd_servicecfg fails.  Beyond that, I have no idea. In this example, I’m using the embedded database.  You can use an external database and configure it via the command line as well, but I don’t have an external Oracle DB in my test environment.

# vpxd_servicecfg db write embedded

See those Host name lookup failure lines? As I mentioned before, I’d forgotten to configure name resolution during this run. Don’t worry, the database initialized fine. Be warned, however, that this can take a long time if you’re doing it against slow storage.  As before, the important line is the VC_CFG_RESULT=0.

As I mentioned before, I’m big into checking certificate thumbprints. The first two keys’ fingerprints should match, but will be different from the one you want to install (the third one):

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware/etc/lighttpd/server.pem

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware-vpx/ssl/rui.crt

# /usr/bin/openssl x509 –fingerprint –noout –in ~/KEYS/vcma01.crt

Making the certificate swap is accomplished with a single command

# vpxd_servicecfg certificate change vcma01.crt vcma01.key

Note that you will get an error, VC_CFG_RESULT=653, if you were jumping ahead (as in the following picture) and got the service started before we got the certificates swapped.

The following command will take care of that so you can get back on track:

# service vmware-vpxd stop

When all goes as planned, you get the VC_CFG_RESULT=0

Re-checking the fingerprints of the same files as before, we see that they all match, indicating that the certificate swap was a success.

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware/etc/lighttpd/server.pem

# /usr/bin/openssl x509 –fingerprint –noout –in /opt/vmware-vpx/ssl/rui.crt

# /usr/bin/openssl x509 –fingerprint –noout –in ~/KEYS/vcma01.crt

 

Go ahead and login to the VCMA with your vSphere client and you should no longer receive a certificate warning.

 

NOTE: I have noticed that the vSphere Web Client on the VCMA (https://YOUR_VCMA_IP_ADDRESS:9443/vsphere-client/) appears to use a different certificate.

That will have to wait for another blog post…

{ 1 comment }

I’ve been meaning to write this post for a long time.  That’s nothing special since there are several posts that I’ve been meaning to write for quite a while.  As a quick aside, I have to highlight that I have a great deal of respect for @DuncanYB, @FrankDenneman, @scott_lowe, and the countless other bloggers who manage to post quality technical content just about every day – consistently and continuously.  Honestly, I don’t know how they do it.

Ever since the Windows Powershell cmdlets for VI3 were released, in the VMware community, there has been a friendly rivalry between the Perl and Powershell camps — I seem to recall an epic VMworld 2010 air hockey battle between @lamw (Perl) and @alanrenouf (Powershell):

I realize that there are other ways to access the vSphere API these days, including Java, C# (via the vSphere Web Services SDK), and even Ruby (and a VMware Labs fling here).  Unfortunately, I lack experience with those versions, so I will restrict this post to Perl and Powershell.  If you’re interested about the SDKs in general, more information is available on VMware’s support site.

My Experience

Before I begin, I’d like to tell you a little about me so you know where I’m coming from.

Perl

I have been using Perl for various tasks since 1993 when I picked up a copy of Learning Perl (“the llama book”) by Randal Schwartz  so I could more easily manage a website I was working on at the time.  I had a slightly more scenic road than your average Perl novice since I was learning Perl on a Macintosh. Before they were UNIX-based or even had a command line. Fun? If you’ve ever embedded Hypercard XCMDs in a Perl script, you know what I’m talking about. I published something in 1998 when the AutoStart 9805 worm was running rampant on Macs around the world.  My code was not pretty, but the first version was written in an afternoon and did what I needed it to do.

Powershell

I did not encounter Powershell until 2008 when the VMworld scripting lab team offered me the opportunity to assist in developing the material.  Originally, I agreed because I thought my Perl experience would be helpful and I wanted to spend some time digging around the VI API.  As it turned out, there were already some Perl ninjas on the team and more help was needed with the Powershell material.  I am no @alanrenouf, but I spent many late night hours playing with both plain Powershell and the VMware add-ons so I could understand how the language was put together.  I learned a lot during those days, and even more helping attendees in the labs.

Bias?

I don’t feel that I have a language bias either way, and I tend to develop in whichever language is supported on the target platform(s).  Sure, I can run Perl on a Windows machine, but it still feels unnatural when Powershell is available natively.  Likewise, if it makes more sense to run a certain script from the vMA or from my Mac, I’ll use Perl.  I’ve spent some time on the VMware Community Forums translating from Perl into Powershell and vice versa.  You can do it!

Maybe it is just my perception because I learned Perl first, but I discovered that any significantly complex Powershell script looks a heck of a lot like Perl.  I’ll demonstrate this later.

Perl vs.(?) Powershell

As a team for the scripting lab, we came up with the (crazy?) idea to fork the single 2008 scripting lab into Perl and Powershell options that worked through the same exercises.  The cool thing was that I was able to experience both methods of accessing the virtual infrastructure to accomplish our defined tasks — attendees could go through the lab twice, or switch between languages as well.

While coding and testing the solutions to our lab exercises, it became clear that either language could be used to perform whatever action we required, and the complexity varied based on availability of Powershell cmdlets or pre-existing scripts and understanding of the API.  Some tasks were dead easy in Powershell but quite painful in Perl. Believe it or not, some tasks were easier in Perl because navigating the API’s object structure is common practice in Perl and just felt wrong in Powershell.  At least, until you got the hang of it.

Cmdlets and Scripts

From a Powershell perspective, the cmdlets included with the VMware Infrastructure Toolkit for Windows (now known as vSphere PowerCLI) provide convenient shortcuts for common actions.  The old VMware Infrastructure Perl Toolkit (now vSphere SDK for Perl), includes some pre-written scripts for performing common actions.  The added benefit of the Perl SDK is that the code for the scripts is available so you can see how they work, copy them, tweak, and repurpose as needed.

Interactivity

Hands down, PowerCLI provides superior interactivity.  Before you think I’m bashing Perl, let me explain.  Powershell was designed differently than Perl: it is an interactive shell that allows batch execution of commands while Perl is an interpreted language.  Think of Powershell more like your traditional command interpreters like csh, bash, ksh, or even (dare I say it?) DOS.

When using the Perl SDK, a per-script binding is created with the virtual infrastructure (login), the code is executed, and the binding is destroyed (logout).  Because Powershell is an interactive shell and PowerCLI extends that, a binding can persist beyond the lifetime of a single script’s execution.  In fact, it is possible to interact with the environment via that binding by executing cmdlets against the objects within the environment, accessed via the persistent binding or bindings.  Don’t misunderstand, it is possible to do some of this using the Perl SDK, but that’s not how it is normally used. Perl is like a swiss army knife or duct tape: if you can dream it, you can probably make Perl do it — thank you Larry Wall!

Along these lines, I noticed that VMware has grouped their various APIs and SDK-related tools according to the intended audience:

Audience	vSphere
Developer	vSphere Web Services SDK 4.1
Developer	vSphere SDK for Perl 4.1
Developer	vSphere SDK for Java (Tech Preview)
Administrator	vSphere Management Assistant 4.1
Administrator	vSphere Command Line Interface (vCLI) 4.1
Administrator	vSphere PowerCLI 4.1

A Simple Example

All of that talk about theory and technical differences… but no code yet.  Who’s bored?  Okay, here comes my first set of examples.  In this one, based on exercises from the VMworld US 2008 Scripting lab, we compare the task of getting a list of all development VMs registered in a vCenter and report their current power state. To make things manageable, each development VM has the string “DEV” in its name:

Powershell

$vc = Connect-VIServer -Server MYVC
Get-VM *DEV*
Disconnect-VIServer -Server $vc -Confirm:$false

Perl

#!/usr/bin/perl -w
# Import runtime libraries
use strict;
use warnings;
use VMware::VIRuntime;
use VMware::VILib;

# Read and validate command-line parameters
Opts::parse();
Opts::validate();

# Connect to the server and login
Util::connect();

# Perform the requested action
my $List = GetVmByName('DEV');
PrintVmPowerState($List);

# Close server connection
Util::disconnect();

##############################################################################

sub GetVmByName {
  my $vmname  = shift;
  return Vim::find_entity_views(
  view_type => 'VirtualMachine',
  filter => {
    name => qr/$vmname/i
  });
}

sub PrintVmPowerState {
  my $vms = shift;
  foreach my $vm (@$vms) {
    print "Virtual machine " .
    $vm->name . " power state is: " .
    $vm->runtime->powerState->val . "\n";
  }
}

To execute this script, it needs to be saved into a file and called:
perl GetVmByName.pl --server MYVC --username MYUSER

That may look very complicated, but, to be fair, the only code that needs to be written is the two subroutines (under the line of hash marks, pound signs, capital 3′s … whatever you want to call them) and their calls from the main program.  The rest is a template created to handle the setup and teardown of the vSphere API connection.  I’ve also taken care to make this script readable – Perl ninjas can shrink this thing down to a few lines if that sort of thing is important.

So, what does that mean?

In the labs, we had several people come in without any scripting experience.  I was totally impressed – it was fantastic to see so much interest in scripting among the VI administrator community.  When we asked whether they were interested in Perl or Powershell scripting, we were asked most of the time for a suggestion.  We came up with one simple question:

1. What operating system(s) are you most comfortable with and which do you intend to use for running your scripts?

If the answer was Linux or MacOS we’d suggest Perl, otherwise we suggested Powershell.  We found that Powershell was much more accessible to people who had never written scripts before.  Sure, writing “hello world” in either Perl or Powershell is just as trivial, but accessing the vSphere API can be more daunting in the Perl world than Powershell’s simple Connect-VIServer cmdlet followed by Get-VM.  More success faster = happier new users.

What do I do?

I don’t keep the vSphere API in my head – I swear that @lamw does! I would probably be able to knock out a Perl script to do whatever I wanted pretty quickly if I spent a lot of my time writing that kind of code, but I forget things if I don’t use them regularly.  At this point in my life, I have few development responsibilities, so I may have short periods of scripting with LONG periods of other work in between.  Like many of you, I am typically looking for a quick, simple script to report on or change some aspect of my environment.  I need to be able to dust off my scripting brain and crank out the report. For that, PowerCLI works very well.

Better Together?

Looking back on my recent work, I noticed something interesting:  when I need to understand how an object “looks” in the API, I regularly jump to the PowerCLI console, even if I am going to write the final script in Perl.  Why would I do that?  The answer goes back to the interactivity of Powershell.  Sure, I could fire up a web browser and go to the vSphere API Reference or even go to the Managed Object Browser (MOB), but there is something about the command line that just feels better to me: I can see how my script will see the objects, minus any translation or filtering.

Examples

Reporting

In this case, I want to take a look at an Advanced Setting on my VMs to ensure that it is set properly. Setting it properly would be nice, too. Once I understand how the object is constructed and how I can access that data, I can construct either Perl or Powershell code to access the object.  Here, the vSphere API Reference or MOB are good for getting an idea of where I should look, but I like to poke around on my own sometimes just to see what’s out there.

My example VM is called “MYVM” and we’re looking at the “isolation.tools.paste.disable” setting. Here’s how I find it:

Powershell

PowerCLI C:\> $vmView = Get-VM MYVM | Get-View
PowerCLI C:\> $vmView

Capability           : VMware.Vim.VirtualMachineCapability
Config               : VMware.Vim.VirtualMachineConfigInfo
Layout               : VMware.Vim.VirtualMachineFileLayout
…

PowerCLI C:\> $vmView.Config

ChangeVersion                : 2011-12-20T16:10:43.434877Z
Modified                     : 1/1/1970 12:00:00 AM
Name                         : wintest2
GuestFullName                : Microsoft Windows Server 2008 R2 (64-bit)
Version                      : vmx-07
…
ExtraConfig                  : {nvram, virtualHW.productCompatibility, pci…
CpuFeatureMask               :
…

PowerCLI C:\> $vmView.Config.ExtraConfig

Key                           Value
---                           -----
nvram                         wintest2.nvram
virtualHW.productCompatibi... hosted
pciBridge0.present            true
sched.scsi0:0.throughputCap   off
disk.EnableUUID               true
pciBridge4.present            true
snapshot.action               keep
deploymentPlatform            windows
… (that's a LONG list... let's filter it!)

PowerCLI C:\> $vmView.Config.ExtraConfig | ? {$_.Key -like "isolation*"} | ft -a

Key                                  Value
---                                  -----
isolation.device.connectable.disable true
isolation.tools.connectable.disable  true
isolation.tools.copy.disable         true
isolation.tools.diskShrink.disable   true
isolation.tools.diskWiper.disable    true
isolation.tools.dnd.disable          true
isolation.tools.paste.disable        true
isolation.tools.setGUIOptions.enable false
isolation.tools.setinfo.disable      true

I now know that the value of the “isolation.tools.paste.disable” Advanced Setting is accessible on my VM at the following location: VM->Config.ExtraConfig[isolation.tools.paste.disable]  For the sake of this example, we’ll assume that the key is present and set to something (true).

Perl

I know I may take some heat from the Perl purists who know that they can achieve the same result using Perl SDK and some simple print/printf or Data::Dumper commands, but the above process is how I tend to work.  (For more information on using Data::Dumper with vSphere, you can go here.) Armed with an understanding of where the data that interests me lives, I can create some Perl code to perform the appropriate query in the same manner.  Note the process is pretty much the same as in Powershell:

1. Obtain a View of the VM
2. Grab the Config->extraConfig property
3. Find the key(s) in question
4. Print the value(s) associated with the key(s)

# Connect to the server and login
Util::connect();

my $vm_name = 'MYVM';
my $check_key = 'isolation.tools.paste.disable';
my $vm_view = Vim::find_entity_view(
view_type => 'VirtualMachine',
filter => {'name' => qr/$vm_name/i}
);

my $extraConf = $vm_view->config->extraConfig;

foreach(@$extraConf) {
  my $vm_key = lc($_->key);
  if($vm_key =~ m/$check_key/) {
    print "$vm_key\t" . lc($_->value) ."\n";
  }
}

# Close server connection
Util::disconnect();

Making Changes

Making changes is a little more difficult, but looks even more similar between languages:

Perl

my $vm_name = 'MYVM';
my $vm_view = Vim::find_entity_view(
  view_type => 'VirtualMachine',
  filter => {'name' => qr/$vm_name/i}
);

my $virtualMachineConfigSpec =
  VirtualMachineConfigSpec->new (
    extraConfig => [OptionValue->new(
      key => 'isolation.tools.paste.disable',
      value => 'true' ),] );

$vm_view->ReconfigVM( spec => $virtualMachineConfigSpec);

Powershell

$vmView = Get-VM MYVM | Get-View

$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmConfigSpec.extraConfig += New-Object VMware.Vim.optionvalue
$vmConfigSpec.extraConfig[-1].Key = "isolation.tools.paste.disable"
$vmConfigSpec.extraConfig[-1].Value = "true"

$vmView.ReconfigVM($vmConfigSpec)

I hope this is useful, or at least interesting for people to see. If you made it this far, thanks for reading!

{ 0 comments }