Saturday was very rainy, so I thought I’d finally get around to upgrading my home lab from ESXi 5.5 to 6.5. I started with my #2 Gen 8 Microserver as I don’t have any live VMs on it, and thus began many wasted hours of reboot after reboot failing to get anything to work.

Slow iLO

The Integrated Lights Out (iLO) management on the Gen 8s is their best feature, and I was able to start out by mounting the .iso for ESXi 6.5 through the iLO remote console and rebooting the server into that to kick off the upgrade.

Sadly the first pass failed on one of the VIBs that’s standard in the HPE ESXi 5.5 bundle (and not used by the Microserver).

After that things spiralled into doom. It wouldn’t boot from the Virtual CD, then it wouldn’t boot from a USB stick, then it wouldn’t complete the upgrade of the internal USB stick I boot from, then it wouldn’t boot from a fresh USB stick.

All the while the iLO was really slow compared to my #1 Microserver.

I did all the usual stuff

Power cycles, iLO resets, BIOS updates, iLO firmware updates (which took ages).

I thought maybe the CPU was overheating and checked the thermal paste.

Maybe it’s a USB problem

I disconnected the internal boot USB stick (and checked it in my laptop)

I disconnected the external KVM (it’s not like I really need it with iLO) in case that was causing issues.

And still the iLO was slow, and USB boot wasn’t working

At this stage I took a close look at the Active Health System Log, which was about 1GB.

A thread I’d found on Reddit ‘HP iLO4 very slow‘ after Googling ‘slow iLO’ suggested that flash issues could cause problems, and maybe a giant AHS log file could be the cause of flash issues.

Reformatting iLO Flash

Perhaps I should have just cleared the log, but I instead went for reformatting the iLO NAND.

Curiously the iLO Self-Test wasn’t reporting a problem with the embedded Flash/SD_CARD, so I wasn’t able to do things the easy way from the (now v2.7.0) iLO web interface. I had to download the HP Lights-Out Configuration utility and feed it a lump of XML to send over to the iLO.

HPQLOCFG.exe -f Force_Format.xml -s iLO_IP -u administrator -p mypassword

<RIBCL VERSION="2.0">
<LOGIN USER_LOGIN="administrator" PASSWORD="mypassword">
<RIB_INFO MODE="write">
<FORCE_FORMAT VALUE="all"/>
</RIB_INFO>
</LOGIN>
</RIBCL>

How did this happen?

I still don’t know. I’d love to read the AHS logs, but the tools to do that live behind a redirect loop on the HPE web site, and may require an active support contract.

Perhaps it’s because the server was powered down, but with the iLO still running, for a few years?

The iLO is supposed to be out of band, and so it shouldn’t affect things like the host USB bus, but I’m guessing that a few corners might have been cut to keep down the cost of adding iLO to the Gen 8 Microserver. I’m also guessing that decision hasn’t impacted many users because a lot of those machines went into home labs yet it seems this isn’t a common problem.

Reformatting worked

After the reformat and a reboot of the iLO it was back to snappy performance. Better still USB storage started working again, so I could finally do that ESXi upgrade.