Same Cluster, 33 Points More Utilization: What Changed Was the Order

In our previous analysis, we posited that the primary bottleneck for enterprise AI is no longer raw intelligence, but rather the efficient utilization of compute resources. We concluded that the industry lacks a standardized playbook for mature GPU management. Today, we are sharing our own.

By developing a constraint-aware GPU allocator and benchmarking it against the industry-standard FIFO (First-In, First-Out) scheduler across seven distinct scenarios, we discovered something profound: on identical hardware running identical workloads, GPU utilization increased by as much as 33 percentage points. Furthermore, priority-weighted output surged in every single test, with gains reaching as high as 105%.

The hardware remained static. The only variable that changed was the logic governing the sequence of allocation decisions.

The Illusion of "Keeping GPUs Busy"

The directive to "keep the GPUs busy" is a goal, not an executable instruction. The actual operational challenge is far more granular: determining which specific GPU runs which specific job, at what time, and at what priority level. Formally, this is a binary decision-making process across a multi-dimensional grid of GPUs, jobs, and timesteps.

The complexity arises from the competition between four distinct workload types:

• Training: Batch-oriented, requiring contiguous GPU blocks for extended durations.
• Batch Inference: Similar to training, requiring uninterrupted, dedicated resources.
• Quantization: A batch-like process that, while often overlooked, consumes significant compute cycles.
• Real-time Inference: Elastic and volatile, driven by fluctuating demand curves that shift every timestep.

The core conflict lies in the incompatibility of these "shapes." Training and batch jobs demand static, contiguous blocks, while real-time inference requires the flexibility to scale up and down. When these competing demands hit the same hardware, the scheduling strategy becomes the deciding factor between efficiency and waste.

The Hidden Cost of FIFO Scheduling

Most current systems rely on a FIFO-based scheduler, which serves real-time inference via fixed reservations and processes all other jobs in the order they arrive.

When a cluster has significant slack, FIFO is perfectly adequate. However, under contention, the cost of this "first-come, first-served" approach becomes catastrophic. This cost manifests in two ways:

1. The Reservation Trap: Because FIFO lacks a mechanism to dynamically reclaim GPUs during troughs in real-time traffic, operators are forced to reserve GPUs based on peak daily demand. If an application needs six GPUs at noon but only two at 4:00 AM, it holds all six for the entire 24-hour cycle. Those four idle GPUs are effectively "grounded," unavailable to batch jobs, yet not actually performing work. 2. The Ordering Penalty: FIFO ignores job priority and future capacity requirements. It commits resources to the first request it sees, often blocking high-priority, high-value tasks that arrive milliseconds later. It is the computational equivalent of an airline assigning aircraft to the first charter that calls, only to find they have no planes left for the high-paying, scheduled routes.

"Order is not a tiebreaker applied after the capacity question is settled. Order is a capacity decision."

Quantifying the Gains

Our allocator addresses these inefficiencies by treating real-time demand as a dynamic curve rather than a static ceiling. It allows batch-like work to occupy the "troughs" of real-time demand, bounded by strict swap-cost constraints.

Across five benchmark scenarios designed to simulate genuine contention, our allocator delivered superior results on both utilization and priority-weighted value:

• Utilization: Improved from a 52–85% baseline to a 72–88% range.
• Priority-Weighted Value: Increased by an average of 52%, with a peak improvement of 105.1% in a training-heavy workload scenario.

In the most extreme case—a training-heavy workload on 8 GPUs—utilization jumped from 53.6% to 87.0%. We effectively recovered 33 points of a depreciating asset simply by reclaiming reserved standby capacity and re-sequencing the remaining workload.

Why Occupancy Isn't Everything

Utilization measures occupancy—the fraction of time a GPU is "busy." However, occupancy is a poor proxy for value. In our scale test (64 GPUs, 30 jobs), both the FIFO scheduler and our allocator achieved identical 44.9% utilization. Yet, our allocator delivered 15.9% more priority-weighted value.

The cluster produced the same amount of "work," but the quality and relevance of that work were significantly higher. This confirms that an objective function failing to price priority will inevitably result in a cluster that is "full" but underperforming.

Writing the Problem Down: The Formal Model

To solve this, we moved away from a list of heuristic rules toward a formal, constraint-based model. A legal allocation must satisfy five structural requirements: 1. A GPU serves only one job per timestep. 2. Jobs must respect demand ranges and maintain existing allocations. 3. Batch-like jobs must occupy contiguous, power-of-two sized blocks. 4. Real-time jobs are subject to strict "swap" caps between timesteps. 5. Running jobs cannot be interrupted.

Our objective function balances these constraints by assigning rewards to batch work based on priority and time-decay, while applying heavy penalties for failing to meet real-time demand. By pricing the real-time penalty at 5 to 10 times the cost of batch work, we enforce latency obligations within the optimization itself, eliminating the need for a separate, competing autoscaler.

The Heuristic Hot Path

Because this is an NP-hard combinatorial problem, we cannot run a full formal optimization for every incoming API request. Instead, we use a high-performance heuristic on the "hot path" that is designed to produce valid allocations by construction.

• Fast Mode: Runs the heuristic to return a grid in 1–2 milliseconds.
• Full Mode: Uses the heuristic grid as a starting point for the formal model to refine the plan during periodic reviews.

This architecture allows the system to "see" the entire scheduling horizon before placing a single job, ensuring that free capacity is held in shapes that future, high-priority jobs can actually use.

The Foundation: Accurate Forecasting

None of this optimization matters if the underlying demand forecasts are inaccurate. We have moved away from generic estimators, instead utilizing specialized forecasters for each workload type:

• Training: Conditioned on 22 features, including specific training variants (e.g., LoRA vs. full fine-tuning), which can vary GPU memory and duration by orders of magnitude.
• Quantization: Treated as a first-class schedulable job with distinct handling for different algorithms (bitsandbytes, AWQ, etc.).
• Real-time Inference: Forecast as a continuously recalibrated weekly demand profile, mapped to GPU counts that align with the scheduler’s swap-cost logic.

Conclusion: Optimize the Day, Commit the Hour

The "end-of-world" effect—where an optimizer makes short-sighted decisions that wreck future performance—is mitigated by our rolling horizon approach. We optimize for a 24-hour window but commit only the current timestep, re-running the model every 30 to 60 minutes. This allows the system to absorb forecast errors and update plans without thrashing.

Airlines did not achieve efficiency through raw computing power; they achieved it through operational discipline in turnaround sequences and crew rostering. By encoding the physical constraints of the cluster into the order of allocation decisions, we have achieved a similar leap in GPU efficiency.

The GPUs were already there. The gain was found in how we chose to spend them.