The Restart That Hid a Libvirt Bug

On a Sunday morning a test VM on our Montreal region took 6 minutes and 44 seconds to create. It was an ordinary machine: 8 vCPUs, 8 GB of RAM, a 20 GB root disk and AlmaLinux 9. Eight seconds after CloudStack reported it created, the platform marked it Stopped, even though it was running.

Creating a VM should take seconds. This post covers how I found the cause, the two wrong turns I took on the way, and why the fix was a package update we already had access to.

Splitting the seven minutes

The CloudStack management log records every command it sends to a host and every answer it gets back. Splitting the create into steps gave me this:

The first VM in a new network also pays for the network’s virtual router. This accounts for about two minutes. The two VM steps were the problem. The disk itself is a Ceph RBD clone, which the host log showed finishing in under a second. The agent spent the other 141 seconds waiting.

The waiting was inside the CloudStack agent on the host. Each time the agent looked up a storage pool, 10 to 50 seconds passed between “Trying to fetch storage pool” and the next line. Running the same lookup by hand with virsh took about 50 milliseconds.

Wrong turn one: a story built on one sample

The agent on this host had been up for 40 days and sat at 100% CPU. A thread dump showed one worker thread busy in a VM stats call. The first explanation was tidy: one stuck thread, a 35-second baseline on healthy hosts, restart the agent and move on.

I did not accept it. The 35-second baseline came from a handful of hand-picked deploys. Nobody had measured the median across all of them.

So I pulled five months of management logs and measured every VM start, host by host. Four hosts show the pattern:

The AMD hosts had slowed by five to ten times since early September. The Intel hosts stayed at about 4 seconds. A second thread dump also corrected the first: all five agent worker threads had used almost the same CPU, about 374,000 CPU-seconds each. An Intel host with the same 40-day uptime had used about 8,400 each. No single thread was stuck. Every call was expensive.

I ruled out four other causes with measurements:

• NFS: the primary storage mount answered statfs in 7 ms with no retransmits.
• Stats traffic: about one storage stats call per minute per management server, nothing unusual.
• TLS: an Intel host on TLS libvirt was fast, so the transport was not the cause.
• Package drift: versions matched across hosts, apart from one Ceph client.

Then I ran perf record against the agent process. About 90% of its CPU was inside libglib, in GLib main loop code around g_source_ref. The agent loads the libvirt client library, which uses a GLib event loop. The profile fits a leak. The loop collects sources and never frees them, so every call walks a longer list than the last.

A restart proved the theory. I restarted the agent on the worst host. Running VMs were not affected, and the host reported Up again 32 seconds later. Storage lookups dropped from 8.4 seconds to 0.01 seconds, and a test VM pinned to that host was created in 12 seconds.

Wrong turn two: a self-healing timer

I now knew the cost grew with agent uptime and call volume. The busy AMD hosts went from healthy to slow in two to three weeks, and the quiet hosts took much longer.

I did not want another alert waking someone up. Our rule: once we understand a fault, a job checks for it, fixes it, and tells a human only when the fix fails. We already run one of these for outbound port 25 filtering on every hypervisor, so I copied the design:

• A daily timer on each host measures the storage lookup gap in the agent log.
• If the average is over one second, it restarts the agent, unless the agent is in the middle of a start, copy or migration.
• It sends a low-priority notification when it acts and a warning only if the restart does not help.

I built it, sent it through code, context and writing review, and tested it in dry-run mode on real hosts. It flagged a slow host in Ottawa and held off on a host restarted the same morning. It was ready to deploy.

One question stopped the deploy

Before deploying, I asked one question: does CloudStack already handle this, so I am not reinventing the wheel?

CloudStack 4.22 does have something close. Point the agent property agent.health.check.script.path at a script and turn on the global setting enable.kvm.host.auto.enable.disable. When the script fails, CloudStack stops placing new VMs on the host and re-enables it when the script passes again. This keeps new VMs off a sick host. It does not make the host healthy. It also removes capacity from hosts whose VMs cannot move elsewhere.

Asking whether this was a known bug got me further. It was. In June 2024 libvirt merged rpc: avoid leak of GSource in use for interrupting main loop. Its commit message describes our symptom: “an ever growing list of GSources attached to the main context, which will gradually slow down execution of the loop, as several operations are O(N) for the number of attached GSource objects.”

An earlier GSource leak in libvirt slowed OpenStack’s compute agent the same way. The patch that fixed it reports queries growing from about one second to over a minute within an hour. The agent ended up using 100% of a CPU thread.

The fix shipped in libvirt 10.5.0. Ubuntu 24.04 ships libvirt 10.0.0, with the fix backported into package 10.0.0-2ubuntu8.19. Our hosts still ran the earlier build.

My timer would have restarted agents every few weeks, forever, to work around a bug a routine package update already fixed.

The fix

I rolled the update across our KVM hosts in both regions, one host at a time:

1. Install only the libvirt packages, keeping our existing configuration files.
2. Restart the CloudStack agent, so it loads the fixed library.
3. Confirm the configuration is unchanged, every VM still runs with the same domain ID, and the host reports Up and secured.

The script stops at the first host to fail a check. None failed. Each host was back in about 30 seconds. No host rebooted and no VM restarted.

I deleted the self-healing timer from the repository before it ever reached production.

The result

Agent logs from July 1 to October 11 show lookup time building up between restarts. Through July and August it climbed, then fell to near zero each time a patch window restarted an agent. Nothing restarted the agents after September 1, and the worst host climbed for six weeks to 7 seconds.

The test creates ran at 16:31 UTC on October 11. I pinned each VM to one host, timed it, then deleted it. The original VM took 6 minutes 44 seconds. About two minutes of that went to the new network’s router. On the VM alone, the time dropped from about 290 seconds to under 20.

These numbers come from the first hours after the update. A freshly restarted agent is fast with or without the patch, so the real proof is the line staying flat over the next week. I am watching agent CPU on the two worst-affected hosts and will update this post then.

What I took from it

• Measure the whole history. One deploy produced a convincing but wrong story. Five months of data showed which hosts were slow and when the slowdown started.
• A restart fix is a clue. It tells you the problem builds up over time. Find out what is building up before you schedule more restarts.
• Ask whether it is a known bug before writing a workaround. Building the self-healing timer took longer than finding the upstream fix did.
• Patch visibility matters. The fix was already in the Ubuntu archive. Our patch reporting did not surface it, so nobody saw the update waiting.
• Push back on tidy answers. Both wrong turns began as neat answers, and both ended when I asked a plain question before acting.