Virtbench 2.0 Goes Beyond VM Benchmarks to Operational Validation

Version 2.0 extends the capabilities of our vendor-neutral toolkit for testing virtual machine performance, scale, and resilience on KubeVirt running on any Kubernetes distribution.

A virtual machine that boots successfully is a starting point, not proof that a virtualization platform is ready for everyday operations. Platform teams also need to understand what happens when many VMs start together, several nodes need maintenance, storage operations overlap, or a worker node fails.

Virtbench 2.0 brings those questions into a more consistent benchmarking workflow. It’s the latest version of the open-source toolkit built by Portworx, and it supports Red Hat OpenShift Virtualization as well as other Kubernetes distributions.

What’s New in Virtbench 2.0?

With a unified command-line interface, broader test coverage, and better-organized results compared to the previous version, it helps teams move beyond isolated performance measurements toward repeatable validation of their VM environment. The 2.0 release adds concurrent chaos testing, FIO and Elbencho storage benchmarking, a dedicated Day 2 operations command group, rewritten failure-recovery testing, and an optional AI-assisted workflow for adaptive validation runs:

• A unified virtbench CLI that provides the entry point for benchmark and operational workflows, with shared options for logging, kubeconfig selection, timeouts, and run identification.
• A dedicated concurrent chaos-benchmark workflow for exercising VM creation, volume resize and clone, VM restart, and snapshot activity under mixed load.
• Storage I/O benchmarking with the new FIO and Elbencho workflows, which run across multiple VMs to measure IOPS, throughput, and latency.
• A new disk-ops benchmark that measures disk hotplug and coldplug attach times for running and stopped VMs, with in-VM validation.
• A new vm-ops Day 2 operations group for node draining, VM rebalancing, batched VM snapshots, blkdiscard on VM data disks, and VM power operations.
• Expanded live-migration workflows, including busiest-node evacuation, round-robin and interleaved scheduling, and multi-source-node migration.
• A rewritten, node-targeted failure-recovery workflow with monitor, manual, and FAR-operator modes.
• First-class boot-storm documentation for single-node, multi-node, and existing-VM variants, plus more consistent result grouping and run artifacts.
• A full documentation site, consolidated I/O tooling, and co-located logs, detailed results, and summaries to make test runs easier to repeat and investigate.
• An optional AI-assisted workflow, usable from Claude or Codex, that turns natural-language prompts into capacity-aware test rounds. It keeps an auditable record of each decision and analyzes the run automatically.

Version 2.0 introduces virtbench as the documented entry point for benchmarks and VM operations, replacing workflows centered on running individual Python scripts. The kubectl-like interface gives operators a common command structure across VM creation, migration, recovery, storage I/O, and Day 2 operations.

The release also provides shared options for logging, kubeconfig selection, timeouts, and run identification. Instead of learning a different execution pattern for every scenario, teams can organize their testing around one CLI. 

Figure 1:  Virtbench command groups

A common entry point makes it easier to build a validation runbook, repeat a scenario after a configuration change, and retain the evidence needed to investigate a result.

Test startup behavior at scale, not just one VM at a time

Starting a single VM tells you little about how the environment behaves when many VMs need to come online together. A boot storm is a distinct operational event.

Virtbench 2.0 documents boot-storm testing as a first-class scenario through the –boot-storm option on datasource-clone. The documented variants cover single-node, multi-node, and existing-VM testing. The toolkit also measures VM provisioning and boot behavior, with network-readiness testing included in its broader scope. 

For platform teams, these scenarios help separate two questions: how quickly can the environment provision VMs, and how does it behave when many VMs start simultaneously? Testing both provides a more useful baseline for capacity planning than measuring an isolated launch.

Exercise migration workflows that resemble maintenance operations

The updated virtbench migration workflow supports automatic selection of the busiest node for evacuation, round-robin migration, interleaved scheduling, and migration from multiple source nodes. For multi-source testing, operators can specify a comma-separated node list:

virtbench migration --source-nodes worker-1, worker-2, worker-3

The workflow discovers matching VirtualMachineInstances directly from the cluster, removes VM and VMI nodeSelector settings before migration, and submits migrations in an interleaved order across source nodes.

That is a more representative validation test than moving a single VM between two otherwise quiet nodes. It lets teams examine migration behavior across a distributed VM population.

Validate behavior under overlapping operations

Production operations rarely happen one at a time. A VM may restart while another workload expands a volume and a third takes a snapshot. Testing each action separately can miss the contention that appears when operations overlap.

The new virtbench chaos-benchmark scenario runs concurrent VM creation, volume resize, volume clone, VM restart, and snapshot operations to stress-test cluster resilience under mixed load. This helps expose behavior that isolated benchmarks may hide. Teams can use this scenario alongside focused lifecycle and storage tests to investigate whether a configuration remains predictable when several kinds of work compete for resources.

Connect VM lifecycle testing with storage performance

VM provisioning and migration measurements are only part of the picture. Teams also need to understand the storage performance available to the workloads running inside those VMs.

Virtbench 2.0 adds FIO and Elbencho testing across multiple VMs, measuring IOPS, throughput, and latency, and consolidates the I/O benchmark tooling under a common repository structure. The new virtbench disk-ops command adds disk hotplug and coldplug testing, measuring attach times for running and stopped VMs, and optionally detach times.

An environment can provision VMs quickly without necessarily delivering the storage behavior an application needs. So a useful evaluation should combine lifecycle tests with storage I/O tests using configurations and workloads representative of the intended deployment.

Make recovery and Day 2 operations part of the test plan

A virtualization platform must be evaluated not only when everything works, but also when something fails or needs maintenance.

Virtbench 2.0 rewrites failure-recovery testing around a single test driver and exposes node-targeted testing through virtbench failure-recovery. The documented modes are monitor, manual, and far-operator. The far-operator mode uses Fence Agents Remediation (FAR) to fence the node. This brings previously separate scripts into a common recovery-testing workflow.

The release also introduces a dedicated vm-ops group:

• drain-nodes drains Kubernetes nodes and measures drain time.
• rebalance-vms redistributes VMs across worker nodes.
• vm-snapshot creates VirtualMachineSnapshots in batches.
• run-blkdiscard runs blkdiscard on data disks inside VMs.
• power-toggle-vms powers VMs on or off.

Rebalancing targets worker nodes by default; control-plane nodes are included only when explicitly requested with –include-master-nodes. Together, these capabilities make maintenance and recovery part of the evaluation rather than an afterthought. 

Keep results useful after the benchmark ends

A benchmark is most useful when someone else can understand what was tested and trace the outcome back to the run.

Version 2.0 keeps logs, detailed results, and summary results together in the same run directory. VM creation and boot-storm outputs are grouped by storage-driver label and disk count, with JSON and CSV results alongside run logs. CLI commands are logged with sensitive values redacted where applicable. The project also includes a dashboard generator that turns saved results into an HTML dashboard.

These improvements help teams preserve context, share findings, and investigate differences between runs. Meaningful comparisons still require controlled test conditions: hardware, networking, storage configuration, VM sizing, and concurrent activity can all affect the outcome.

Plan adaptive validation runs with an optional AI assistant

Deciding how many VMs to test, and how far to push a cluster, usually takes several manual iterations. Virtbench 2.0 adds an optional AI-assisted workflow: operators describe a run in plain language from Claude or Codex, and a deterministic planner carries it out.

A prompt such as:

@virtbench vm-create, bootstorm and migrate 4 rounds max VMs 120

becomes a planner run. The planner does the cluster math, not the language model. It reads node allocatable resources and current pod requests, keeps 15% headroom by default, estimates a VM ceiling, and spreads VM totals across the requested rounds. An optional max-VM value sets a hard cap. If the best stable result reaches that cap, the summary reports it as user-capped rather than as the cluster’s maximum.

In each round, the workflow creates a fresh VM set, runs a boot storm against it, rebalances it across worker nodes, and runs multi-source live migration against the same VMs. It then cleans up before the next round. Each step can also run on its own, for example a boot storm or migration against existing VMs. Adaptive planning also covers FIO searches for maximum IOPS or bandwidth, and disk-ops testing. For Elbencho, chaos, failure-recovery, and cluster validation, the assistant helps run the existing commands without changing them.

Each AI run writes virtbench_ai_rounds.json, an auditable record of the planner’s decisions. Alongside it is a JSON and Markdown analysis with per-round status, duration trends, failure causes, and suggested next actions. The results dashboard shows these round decisions next to the matching results and can compare runs, such as 1-disk and 4-disk workflows, in separate tabs.

Build a repeatable validation baseline with Virtbench

A practical starting sequence is to validate the cluster, create a representative VM population, test boot behavior, rebalance placement, and exercise multi-source migration against the same VM set. The optional AI-assisted workflow can run the create, boot storm, rebalance, and migrate sequence across multiple rounds for you. Add storage I/O, mixed-operation, Day 2, and node-failure tests according to the questions your team needs to answer. 

Before testing, define:

• Which VM sizes, disk layouts, and storage configurations represent the intended workloads?
• What startup, migration, maintenance, and recovery criteria will determine acceptance?
• Which disruptive operations are authorized, and what test data can be discarded?
• Which configuration details and run artifacts must be retained for a repeatable comparison?

Ready to start? Explore the official Virtbench documentation, 2.0 release notes, and GitHub repository to build a validation plan for your environment.

What’s Next?

As virtbench helps you establish a clear performance baseline for KubeVirt virtual machines, the next question is how to deliver that performance consistently for production workloads. Join our upcoming hands-on lab, One Copy, Zero Compromise: Portworx KDS on OpenShift, to work directly with Portworx Kube Datastore (KDS) backed by Everpure FlashArray. You’ll provision and manage stateful workloads, see capacity scale dynamically as demand grows, and explore how to maintain resilience without host-side replicas – all practical foundations for running modern virtualized applications with enterprise performance and efficient data protection.