Automating HPE server configuration with iLOrest

January 7, 2025

One of the most beloved configuration management and automation tools in a Linux sysadmin’s arsenal these days is undoubtably, Ansible. But what if you cannot use Ansible or another configuration management tool? This article dives into configuring a cluster of HPE servers using the iLOrest tool.

Table of contents:

• 99 Bottles of Beer
• If the mountain won’t come to Mohammed
• Security through obscurity
• Debugging
• Sharing our solution

99 Bottles of Beer

A client of ours was setting up new clusters of servers consisting of 30+ HPE servers, which needed to be configured before the OS could be installed. This configuration is a tedious and repetitive task, with each server taking 1 day to configure. If only Ansible could be used like on the OS to automa… wait, there are playbooks for that, great! HPE provides Ansible playbooks to interface with their iLO out-of-band management interface, that sounds like the problem is fixed then, right?

If only things were so simple! There were some constraints:

• The existing Ansible infrastructure did not meet the version requirements for the HPE playbooks
• This was a completely new network with almost no connections to the existing infrastructure
• Internet access was not permitted

But… there was one machine which had access to the new network and it had HPE’s iLOrest installed. As the name suggests, iLOrest is an open source tool to interface with iLO using the REST API it serves. iLO implements Redfish: this is a standard which describes how to talk to and manage hardware via a REST API, which is the same mechanism the aforementioned HPE iLO-Ansible playbooks use.

With iLOrest, we can read values from the BIOS/UEFI, change settings, make changes to RAID configurations (if supported by your iLO version and RAID controller), …

In our client’s case we had to:

1. Check if the iLO license was activated
2. Update the firmware
3. Reset the BIOS/UEFI
4. Disable the internal SD card slot
5. Diagnose a problem with the fans

If the mountain won’t come to Mohammed

These steps take time and servers need to be rebooted in between these steps. iLOrest allows us to perform these actions using one simple command instead of using the GUI for each server, but iLOrest is only made to interface with one server at a time.

A breakdown of how the ilorest command works. By default ilorest is an interactive command, but if you add the --logout parameter, the command will run non-interactively.

ilorest <verb> [--parameter] --url=<IP-of-iLO> --user=<ilo-username> --password=<ilo-password> --logout

We needed to somehow parallellize these tasks for the more than 30 servers, so the solution we came up with was to wrap the iLOrest tool in a for loop and make it run à la Ansible. We ask for a file with list of the iLO IPs, a file with the iLO credentials and voilà! As it turned out, this would be a huge time saver for the project, cutting down the required coniguration time from 7 weeks to 5 days, which included the time to write and test the different scripts.

The problem that kickstarted this solution, was to find a way to update the firmware for every server, a task that easily takes 2 to 3 hours per server. This has been converted into a firmware update script, please note that only the core logic of the script is shown:

1. Get the server model (we were working with two different models)

get_server_model() {
    local ip=$1
    local server_model=$(ilorest serverinfo --system --url=$ip --user=$username --password=$password --logout | awk -F " " '/Model/ {print $4}')
    echo $server_model
}

2. Get the system ROM version:

get_system_rom_version() {
    local ip=$1
    local rom_version=$(ilorest serverinfo --firmware --url=$ip --user=$username --password=$password --logout | grep "^System ROM" | cut -d ':' -f 2 | cut -d ' ' -f 3)
    echo $rom_version
}

3. Launch the firmware update if the system ROM version is smaller than the latest/desired version, depending on the server model:

# Create an associative array of server models and the desired version of their system ROM 
declare -A desired_system_rom_versions_map=(
    [XL450]="v3.01" # Not real version numbers
    [XL420]="v2.02" # Please check your relevant SPP for the desired system ROM for your systems
)

start_firmware_update() {
    local ip=$1
    local server_model=$(get_server_model "$ip")
    local current_rom_version=$(get_system_rom_version "$ip")
    local desired_system_rom_version=${desired_system_rom_versions_map[$server_model]}     

    if [[ -z "$desired_system_rom_version" ]]; then
        echo "Unknown server model $server_model for IP $ip. Skipping."     
    fi     

    # Check if the current ROM version is smaller than the desired version     
    compare_system_rom_versions "$current_rom_version" "$desired_system_rom_version"     
    local version_comparison=$?     

    if [[ $version_comparison -eq 0 ]]; then
        echo "System ROM is already $desired_system_rom_version for $ip, skipping."
    elif [[ $version_comparison -eq 1 ]]; then
        echo "Current ROM version ($current_rom_version) is lower than desired version ($desired_system_rom_version) for $ip, updating."

        if [[ "$(get_server_state $ip)" != "PowerOff" ]]; then
            echo "Server $ip is not powered off, skipping."
        else
            echo "Mounting $iso on $ip".
            ilorest virtualmedia 2 $iso --bootnextreset --reboot=ForceRestart --url=$ip --user=$username --password=$password --logout             
            power_on_server $ip
        fi
    else
        echo "Current ROM version ($current_rom_version) is greater than the desired version ($desired_system_rom_version) for $ip, skipping."
fi
}

Gradually, we saw that our needs were bigger than simply updating the firmware of all the servers. Referring back to the required configuration steps, we divided these into their most basic tasks, such that we could chain the scripts together to perform the actions in the required order.

Security through obscurity

Some of you may have noticed that the ilorest command takes in (among others) the credentials for the iLO interface, which is normal. The problem with this however, is that the password for the associated iLO account is entered into plain text on the CLI, which poses a certain risk. These scripts instead ask for a parameter -c|--credential-file with a path to the iLO credentials. This was deemed to be less risky rather than having the password in the shell history.

Debugging

During the configuration of the servers, we noticed that the fans of the Apollo 4510 Gen10 Plus servers were pinned to 85% fanspeed after (re)booting. This issue was not always present though, but it was present in ~80% of the reboots. After trying out different firmware versions, iLO resets and thermal configurations, we were unable to find the cause. We concluded that this was likely a bug with HPE. As of the time of publishing this article, this issue remains open. The only remedy we found for this particular server model was to reset the iLO, after which the fans would spin down according to the server load.

In order to debug more efficiently we relied on the following two scripts:

• One script for retrieving the fan speed of the servers
• And another script for resetting the iLO

During the server configuration we also attempted to configure a RAID array, but this failed during our testing with iLOrest. We

Sharing our solution

Seeing that more people may find this useful, even if only for inspiration of the possible uses with iLOrest, the original author has published the scripts on GitLab under a MIT license.

Every script can be run independently and asks for two following parameters:

• -f|--file for the file with the list of iLO IPs:

10.0.1.1
10.0.1.2
#10.0.1.3 <-- This IP will be ignored because it is commented out
10.0.1.4

• -c|--credential-file for the file with the iLO credentials:

username=<ilo-username>
password=<ilo-password>

For the complete list of scripts, please see the aforementioned GitLab repository.