Product & Environment
- Product: qbcompiler
- Compiler Version: 1.2.0
- OS: Ubuntu 24.04.1 LTS, kernel 6.8.0-106-generic
Description
Summary
First of all, thank you for the detailed answer on the OLMOCR2/Qwen2.5-VL thread — very helpful.
I wanted to share a successful run and ask about its compile time. We compiled Qwen/Qwen3-Embedding-0.6B and Qwen/Qwen3-Reranker-0.6B (small text-only models, hidden_size 1024, 0.6B params) to aries-rb with bits=w8v8, preset=R1R2, optq=true. Both compiles finished successfully and passed our own provenance checks (correct calibration mix, max_seq_length=2048, correct preset in the output filename). We’re happy about that — but the wall-clock time surprised us: each 0.6B model took ~22 hours to compile, running the two compiles in parallel on two separate MIG slices of the same GPU.
What we ran
Two new recipes cloned from our existing 4B recipes, changing only the model id, calibration corpus, and quantization preset:
recipe.yaml (reranker; embedder is the same shape, mix: "autorag:216" instead of "autoragq:44"):
name: qwen3-reranker-0.6b-autorag
kind: reranker
engine: qwen3
model_id: Qwen/Qwen3-Reranker-0.6B
revision: null
device:
aries-rb: {inference_scheme: all}
regulus-rb: {inference_scheme: single}
compile:
max_seq_length: 2048
weight_dtype: bfloat16
tag: autorag
calib:
dir: qwen3-reranker-0.6b-autorag
source: 44 real (query, gold-document) pairs from a held-out retrieval-benchmark split
mix: "autoragq:44"
max_calib: 256
max_len: 2048
runtime:
family: qwen3
measured: {}
quant.yaml (both recipes):
bits: w8v8
preset: R1R2
optq: true
Actual invocation (via our own compile_mxq.sh wrapper, which just calls qbcompiler’s mxq_compile() under the hood through compile/lib/qwen3_compile.py):
CUDA_VISIBLE_DEVICES=MIG-fe24ab8c-4f8c-5248-90c9-e7e98a0afa04 \
python3 compile/lib/qwen3_compile.py \
--model-id /path/to/Qwen3-Reranker-0.6B \
--bits w8v8 --preset R1R2 \
--calib-data-path /path/to/calib_data/qwen3-reranker-0.6b-autorag \
--out-dir /path/to/mxq/aries-rb/reranker/qwen3-reranker-0.6b-autorag \
--target-device aries-rb --inference-scheme all \
--max-seq-length 2048 --weight-dtype bfloat16 --optq --tag autorag
CUDA_VISIBLE_DEVICES=MIG-1bfd7bf5-50bc-53b9-8bc8-63af55af478e \
python3 compile/lib/qwen3_compile.py \
--model-id /path/to/Qwen3-Embedding-0.6B \
--bits w8v8 --preset R1R2 \
--calib-data-path /path/to/calib_data/qwen3-embedding-0.6b-autorag \
--out-dir /path/to/mxq/aries-rb/embedder/qwen3-embedding-0.6b-autorag \
--target-device aries-rb --inference-scheme all \
--max-seq-length 2048 --weight-dtype bfloat16 --optq --tag autorag
Both invocations run concurrently, one per MIG slice, from the same qbcompiler install.
Environment
| Component | Version |
|---|---|
| qbcompiler | 1.2.0 |
| Python | 3.10.21 |
| torch | 2.7.1+cu126 |
| torchvision | 0.22.1+cu126 |
| transformers | 4.57.6 |
| accelerate | 1.15.0 |
| einops | 0.8.2 |
| psutil | 7.2.2 |
| numpy | 1.26.4 |
| OS | Ubuntu 24.04.1 LTS, kernel 6.8.0-106-generic |
| CPU | 2x Intel Xeon 6767P (64 cores / 128 threads each socket, 256 logical CPUs total) |
| RAM | 2.0 TiB total |
| GPU | 1x NVIDIA H200 NVL |
| Driver | 570.211.01 (CUDA 12.8) |
| GPU partitioning | MIG enabled, 2x 1g.18gb instances — one recipe per slice, run concurrently |
| Target device | aries-rb |
What we observed (timing + resource usage)
Both jobs went through the same stages, and both showed the same bottleneck stage: the FB quantizer’s Performing Quantization (FB)... phase (calibration tensor loading → internal Hessian/OPTQ rounding) dominated the entire run. From our log timestamps:
Reranker (started 2026-09-17 00:43, finished 2026-09-17 23:26 — ~22.7h total):
00:48:12— calibration tensors loaded,Performing Quantization (FB)...begins22:19:14—quantization done.(~21.5 hours in this one stage)22:19→23:26(~67 min) — backend Build passes (Single → Global → GlobalCluster) + MXQ export, i.e. everything after quantization was fast
Embedder (started 2026-09-17 01:30, finished 2026-09-17 23:36 — ~22.1h total):01:37:32— calibration tensors loaded,Performing Quantization (FB)...begins23:07:01—quantization done.(~21.5 hours, essentially identical to the reranker despite running on a separate MIG slice/process)23:07→23:36(~29 min) — backend Build passes + MXQ export
So >95% of the wall-clock time in both jobs was the FB quantization/OPTQ stage, and it took almost exactly the same duration (~21.5h) for both models despite the embedder and reranker having slightly different calibration set sizes (216 vs. 44 examples) and different heads.
While this stage ran,psconsistently showed ~590–660% CPU for each process (of 256 logical CPUs available) — clearly CPU-bound, using only a few cores’ worth of parallelism — whilenvidia-smi --query-gpu=power.drawstayed flat around 94W of a 600W power limit the entire time, for both MIG slices. That reads to us as: the OPTQ/Hessian-gathering compute for this stage is running on CPU, not GPU, even though we launched withdevice="gpu"semantics (each process pinned to its own MIG slice viaCUDA_VISIBLE_DEVICES).
Question
What can we change to reduce this compile time — is there a configuration knob (batch size, worker/thread count, a GPU-accelerated OPTQ path, etc.) for the Performing Quantization (FB) / OPTQ stage that we’re missing? And if possible, could you help us understand why this stage takes long compile time for a 0.6B-parameter model with 44–216 calibration examples — CPU-bound at ~590–660% while GPU power stays flat at ~94W/600W — so we can avoid hitting the same bottleneck on future, larger runs?
Happy to share the full compile logs (attached below) or run additional diagnostics — this isn’t blocking us (both .mxq artifacts compiled correctly), but a ~22-hour compile for a 0.6B model seems worth flagging, especially since we’re hoping to iterate on calibration recipes for these models going forward.
Logs & Attachments
rerank_0.6b.log— full reranker compile log (attached)embed_0.6b_run2.log— full embedder compile log (attached)
Logs & Attachments
embed_0.6b_run2.log (994.8 KB)
rerank_0.6b.log (944.1 KB)