Turning a CPU + GPU node into one elastic pool with an operator-set objective. Lightweight, high-volume agent steps are absorbed by the CPU to protect scarce, power-hungry accelerator capacity across Data Centers; heavy and latency-critical steps earn the accelerator on measured merit; and because the accelerator’s cost is read from its own telemetry, a saturated GPU de-prioritizes itself with no manual flag and no code change.
Chun Tao, Souvik Kundu · Intel · 2026-08-24 · Tech · 18 min read
inference orchestration agentic-workloads heterogeneous-compute cpu gpu intel-arc xeon vllm bandits telemetry benchmark
TL;DR. Two signals place each step: how hard it is, from a complexity classifier, and how expensive the accelerator is right now, from live telemetry. One policy weight decides how the two trade off. Measured on a single node across four levels of co-tenant contention, with the same 80-step batch and the same code at every level:
- The pool self-balances as the accelerator fills. CPU share rose 52.5% → 78.8% as measured accelerator load rose 0.31 → 0.83 and its cost reward fell 0.63 → 0.03. Nothing was reconfigured between levels, and no operator set an “overloaded” flag.
- Offload is the faster option under contention, not just the cheaper one. At the highest level the routed policy finished in 15.2 s against 18.2 s for accelerator-only (−17%), with all 80 steps completing under both policies.
- Capacity comes back to the pool. About 20 W less per tile, and the GPU engine held 17 concurrent requests instead of 32 — headroom a co-tenant or a latency-critical step can use.
- Hard work never left the accelerator.
COMPLEXandREASONINGsteps stayed on the 24B model at every level, so the judged quality cost was 0.02–0.06 on a 0–1 scale.- One knob spans the operating range.
QUALITY_WEIGHTmoved placement 70% → 25% CPU — a 45-point swing between a cost-first and a quality-first node — for 0.12 of judged quality and, again, no code change.
Setup and notation. One node: 4 × Intel® Arc™ Pro B70 and an 88-thread Intel® Xeon® w9-3575X, one endpoint in front of two open-weight models — Devstral-Small-2507 (24B dense) on two GPU tiles, Qwen3-0.6B on the host CPU. A co-tenant process progressively claimed the GPU’s unreserved memory.
A tile is one Arc Pro B70 device; load is the composite telemetry reading in [0, 1] defined below; congestion reward is the reward the router assigns to the GPU arm, (1 − load)²; makespan is the wall-clock time to finish all 80 steps; p95 is the 95th-percentile per-step latency. One number in this post is not measured — the 9 s p95 SLO reference line, chosen to sit near the measured band.
Agentic workloads are a mixed-difficulty stream by construction. A single agent run is composed of planning, reasoning, tool-call, and often retrieval steps. Each of these steps may have different level of complexity, and thus does not need the same model. Also, they certainly do not need the same hardware to host the models.
Majority of the serving stacks route them identically anyway: one model, one endpoint, one device pool. The accelerator absorbs everything, including the steps a small model would answer well, while host CPUs that are already present alongside and already powered on, remain near idle during most of the inference steps.
Complexity-aware routing fixes the fixed-model task navigation issue. Classify the step, send the easy ones to a smaller models, keep the hard ones for more capable models. It is a static improvement: the same step gets the same placement whether the accelerator is idle or saturated.
On a shared node the GPU accelerator’s business changes minute to minute. Due to a co-tenant — another team’s job, another model, another agent fleet — that is competing for the same tiles and resources. A router that is unable to account for this changing GPU telemetry, and is not resource aware may not make the right decision in model selection.
This post measures a router that uses both signals: step complexity from a classifier, and GPU congestion reward from live telemetry.
The router has a feedback reward computed as a weighted linear combination of quality, efficiency, and congestion reward. This reward influences the selection probability of a model under a specific complextiy tier. Each tier has > 1 model option with its own bandit probability of selection, that dynamically gets updated based on the reward. We collect live telemetry to get the composite GPU load from VRAM pressure. For a tier we host relatively smaller model to the CPU while keeping the larger one to GPU accelerator. Thus we have option to route each step to the CPU arm (smaller model on an 88-thread Xeon) or the GPU arm (larger model hosted on Arc Pro B70 tiles, congestion reward computed from live telemetry).

Router architecture. An operator sets one weight, QUALITY_WEIGHT, which splits the reward between quality, efficiency and cost. Live telemetry forms a composite GPU load from VRAM pressure weighted 0.65, compute-engine utilization 0.22 and memory bandwidth 0.13; cost equals (1 minus load) squared and drops to zero past 0.90. A complexity classifier feeds per-tier bandits whose GPU draw is discounted by that cost, routing each step to the CPU arm (0.6B model on an 88-thread Xeon, cost fixed at 1.00) or the GPU arm (24B model across two Arc Pro B70 tiles, cost from live telemetry). Each result is measured back into one reward in 0 to 1 that updates the bandit.
Figure 1 — The placement path. The two blue arrows are the mechanism: the same telemetry reading prices the decision before a step is dispatched, and prices the reward after it returns.
The routing steps include:
A complexity classifier sorts each step into one of four tiers — SIMPLE, MEDIUM, COMPLEX, REASONING — for this work we used a rule-based approach to classify. No model call, no learned parameters. It does not choose hardware; it only states how hard the step looks.