My XenServer 6.2 systems were pretty far out of date with regard to updates and hotfixes. So, yesterday I decided to apply the hot fixes available from XenServer.org.

We just use the software available from the XenServer.org site. We anticipated that we’d be able to apply these hot fixes using the XenCenter GUI but it seems that this isn’t an option if one doesn’t have a support license from Citrix.

I use and license some other products from Citrix but I don’t currently pay for support for XenServer. I might going forward as this was kind of a bear. But, we may also look at some other options, too. And, even with the ability to upgrade via the XenCenter GUI it still seems kind of a throw back in the current days where one can just apt-get update and apt-get dist-upgrade to keep a Linux (Debian based/derived anyway) fully patched.

It turns out that one can still apply these hot fixes but it requires getting the packages to the server and using some command line tools. In my case I have a pool — several XenServer hosts grouped together — so I wanted to update all the servers in my pool.

I downloaded the following service packs and hotfixes from XenServer.org

XS62ESP1
XS62ESP1001 (New Version of XenCenter)
XS62ESP1002
XS62ESP1003
XS62ESP1004
XS62ESP1005
XS62ESP1008
XS62ESP1009
XS62ESP1011

XS62ESP1013 and XS62ESP1013 were also listed on the site at the time but we’re advisories not actual packages.

I unzipped all of the above hotfixes and copied the contents of each into a single folder called HotFixes.

Before I get into the details I ran into a few of obstacles before I even started…

1. I couldn’t upgrade XenCenter on my Windows 8.1 workstations as the installer complained about not having dotNet3.5 installed. I used the Windows control panel to add this feature but
   still no joy. I’m not sure if this is a problem on the XenCenter side or the Windows side. Turns out I already had the current version: 6.2 (build 1377) so bascially a lot of clicking around for no reason.

2. I used WinSCP to upload my HotFixes to the /root directory on the server but my first attempt to apply the first service pack failed because there was not enough room on / partition to complete the install. So, I moved my HotFixes folder to a share on one of my regular Windows servers and then just used the mount command to reach the package files. The commands issued on the XenServer we along the lines of:

mkdir /mnt/shareOnWindowsServer
mount -t cifs //NameOfWindowsServer/NameOfShareOnWindowsServer -o username=UserName,password=myPassword /share/shareOnWindowsServer
cd /share/shareOnWindowsServer/HotFixes

1. I learned that High Availability needed to be turned off for these steps to work. From within XenCenter click on the pool you are trying to upgrade. Click on the HA tab and disable High Availability.

I now had High Availability OFF and access to all the HotFixes without using any space on my local / partition on the XenServer pool master.

So, now the fun begins…

1. Get a list of all the UUIDs for each of the servers in your pool

xe host-list

As an example two of the UUIDs in my pool look like:

6c5fc1d8-d0be-46b9-abb8-912701ca41ef and e44fc0df-d147-4a2e-be62-6a1d51620f00

Copy and paste yours into a text doc as you will need the UUID for each host in the next few steps.

1. Upload each patch to the pool. Very important!: After uploading each patch make a note (copy and paste to a txt document) the UUID issued directly after you issue the upload command. You will need these UUIDs later as well as the UUIDs you collected about the servers in your pool above.

xe patch-upload file-name=XXS62ESP1.xsupdate
xe patch-upload file-name=XXS62ESP1002.xsupdate
xe patch-upload file-name=XXS62ESP1003.xsupdate
xe patch-upload file-name=XXS62ESP1004.xsupdate
xe patch-upload file-name=XXS62ESP1005.xsupdate
xe patch-upload file-name=XXS62ESP1008.xsupdate
xe patch-upload file-name=XXS62ESP1009.xsupdate
xe patch-upload file-name=XXS62ESP1011.xsupdate

For the above I got the following UUIDs. I’m not sure if they are randomly generated or not. But, use the ones issued to you if they are indeed different from mine.

0850b186-4d47-11e3-a720-001b2151a503
297f2f77-5603-4aaf-9e56-db49512d4592
c208dc56-36c2-4e91-b8d7-0246575b1828
996dd2e7-ad95-49cc-a0be-2c9adc4dfb0b
1c952800-c030-481c-a0c1-d1b45aa19fcc
59a75271-12f9-4e6a-8ba2-325c2f5b0b47
a24d94e1-326b-4eaa-8611-548a1b5f8bd5
7d670435-547c-419a-ab7e-296705a752b8

1. Apply the patches to EACH server in your pool. So for the following commands I’ll use the UUIDs for two of my servers above and the UUIDs that were issued to me by the patch-upload command. These are just examples. Use the actual UUIDs from the output of your own xe host-list and xe patch-upload commands.

On my pool master I issued the following commands to upgrade that specific server:

xe patch-apply uuid= host-uuid=


xe patch-apply uuid=0850b186-4d47-11e3-a720-001b2151a503 host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef
xe patch-apply uuid=297f2f77-5603-4aaf-9e56-db49512d4592 host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef
xe patch-apply uuid=c208dc56-36c2-4e91-b8d7-0246575b1828 host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef
xe patch-apply uuid=996dd2e7-ad95-49cc-a0be-2c9adc4dfb0b host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef
xe patch-apply uuid=1c952800-c030-481c-a0c1-d1b45aa19fcc host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef
xe patch-apply uuid=59a75271-12f9-4e6a-8ba2-325c2f5b0b47 host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef
xe patch-apply uuid=a24d94e1-326b-4eaa-8611-548a1b5f8bd5 host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef
xe patch-apply uuid=7d670435-547c-419a-ab7e-296705a752b8 host-uuid=6c5fc1d8-d0be-46b9-abb8-912701ca41ef

Again, while still on the command line of my pool master I issued the following command to upgrade another member of my pool.

xe patch-apply uuid=0850b186-4d47-11e3-a720-001b2151a503 host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00
xe patch-apply uuid=297f2f77-5603-4aaf-9e56-db49512d4592 host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00
xe patch-apply uuid=c208dc56-36c2-4e91-b8d7-0246575b1828 host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00
xe patch-apply uuid=996dd2e7-ad95-49cc-a0be-2c9adc4dfb0b host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00
xe patch-apply uuid=1c952800-c030-481c-a0c1-d1b45aa19fcc host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00
xe patch-apply uuid=59a75271-12f9-4e6a-8ba2-325c2f5b0b47 host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00
xe patch-apply uuid=a24d94e1-326b-4eaa-8611-548a1b5f8bd5 host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00
xe patch-apply uuid=7d670435-547c-419a-ab7e-296705a752b8 host-uuid=e44fc0df-d147-4a2e-be62-6a1d51620f00

The above needs to be repeated changing the host-uuid for each member of your pool.

1. I shutdown each of the servers.
2. I powered on the pool master first. Once it was fully booted I powered on the other members of the pool.

Everything came back up. XenCenter could see that the patches had been applied. But all was not well!!!

I was able to start each of my Windows virtual machines. But, I was not able to start ANY of my Ubuntu/Linux based virtual machines. Uggh!

I got the following error:

Error: The bootloader for this VM returned an error -- did the VM installation succeed? Unable to find the partition containing the kernel

Each of my Linux VMs are Ubuntu 14.04 server instances. These use grub2 which seems to be an issue for XenServer 6.2. In the past to get around this I made an edit to the /usr/lib/python2.4/site-packages/grub/GrubConf.py file as outlined here. But, this edit threw syntax errors when I tried the start the Linux VM.

When I made the edit outlined above I just got a new and different error complaining about the script syntax.

Oct 16, 2014 7:14:08 PM Error: Starting VM 'Gray' - Internal error: xenopsd internal error: VM = ffe05941-b5cf-38a0-f826-05658bdfe848; domid = 9; Bootloader.Bad_error Traceback (most recent call last):
File "/usr/bin/pygrub", line 26, in ?
import grub.GrubConf
File "/usr/lib/python2.4/site-packages/grub/GrubConf.py", line 428
if arg.strip() == "${saved_entry}" or if arg.strip() == "${next_entry}":
^
SyntaxError: invalid syntax

I’m not a Python programmer and my little fix wasn’t getting the job done. So, I took a shot in the dark and changed line 428 of /usr/lib/python2.4/site-packages/grub/GrubConf.py
from the original:
if arg.strip() == "${saved_entry}":

to:
if arg.strip() == "${next_entry}":

That did the trick and all my Linux VMs started with no problem. I of course had to edit /usr/lib/python2.4/site-packages/grub/GrubConf.py on each of my XenServers.

Generally, I have very few problems with XenServer 6.2 (the grub2 thing is the biggest annoyance). That being said when I look into alternatives I may just look at running Xen on Ubuntu. Like I wrote earlier being able to stay up to date just issuing apt-get update; apt-get dist-upgrade commands really spoiles you against procedures like those above. If anyone out there has thoughts/ideas/suggestions or opinions they’d like to share on virtualized environments/options my ears are open.

All of the above was quite a chore. I do hope these notes help someone else out.