When you hand a frozen model to a hardware-specific compiler, what actually happens? The answer matters because it determines how much of the theoretical efficiency gain from custom silicon is achievable in practice, and what the engineer's job is during the specialization process. Most teams interact with model compilation as a black box — they feed in an ONNX file and receive a compiled binary. The intermediate steps are where the meaningful work happens.
This is an account of that process — the analysis passes, partitioning decisions, and transformation stages that convert a general-purpose computation graph into a hardware-bound dataflow program.
Starting Point: What the Frozen Graph Contains
An ONNX export of a frozen transformer model is a directed acyclic graph where nodes are operators (MatMul, LayerNorm, Softmax, GELU, etc.) and edges are tensors with known shapes and data types. The graph is abstract — it describes computation, not how that computation should be scheduled onto any specific hardware.
Critically, the graph produced by PyTorch or TensorFlow's tracing mechanisms often contains artifacts from the training framework that are irrelevant or actively harmful for inference. Common issues:
- Dropout nodes with training-mode masks. In inference, dropout is a no-op, but naive export tools preserve the node with a p=0 probability. It still requires memory and scheduling.
- Redundant reshape and transpose operations. Training frameworks insert layout transformations to satisfy gradient computation requirements. Many of these are absent from the forward inference path but still present in the exported graph.
- Symbolic batch dimensions. Training graphs often parameterize batch size as a symbolic variable. A frozen-graph compiler can specialize for a fixed batch range, unlocking tiling decisions that symbolic compilation cannot make.
- Unmerged normalization layers. BatchNorm or LayerNorm followed immediately by a scale/shift operation can be algebraically folded into the preceding linear layer's weight matrix, eliminating both the normalization kernel and the activation read/write cycle.
Before any hardware-specific work begins, a good compiler runs a graph cleaning pass that resolves all of these. The graph that reaches hardware-specific compilation should contain only the operations that are necessary for forward inference with the specific input shape range it will serve.
Operator Analysis: Arithmetic Intensity Classification
After cleaning, each operator in the graph is characterized by its arithmetic intensity — the ratio of floating-point operations to memory bytes accessed. This classification determines scheduling strategy.
A dense matrix multiply (GEMM) with large M and N dimensions is compute-bound: hundreds of operations per byte fetched. An element-wise activation function (GELU, ReLU, Swish) is heavily memory-bound: roughly 1–4 operations per byte. Softmax sits between them: it requires a reduction pass for the denominator that forces multiple reads of the input tensor.
The hardware target's roofline model — peak TOPS versus peak memory bandwidth — defines which operators can run at compute-peak efficiency and which will be bandwidth-limited. A 7nm ASIC with 32 TOPS and 2.4 TB/s of LPDDR5X access has a balance point (ridge point) of 13.3 ops/byte. Any operator above 13.3 ops/byte is compute-bound; any below is memory-bound on this chip. The compiler uses this to prioritize fusion candidates: if a memory-bound operator follows a compute-bound operator, they're fusion candidates — the output of the first stays in SRAM and feeds directly into the second.
Graph Partitioning and Fusion
Operator fusion is one of the highest-leverage transformations a compiler can make. The goal is to eliminate HBM round-trips by keeping intermediate tensor data in on-chip SRAM between operations that would otherwise each require a load/store cycle to off-chip memory.
The canonical example in transformer models is the attention block. An unoptimized attention forward pass reads Q, K, V from memory, computes QK^T, writes the result to memory, applies softmax (another memory round-trip), then multiplies by V. FlashAttention-style tiled attention fusion fuses all of these into a single kernel that tiles across the sequence dimension, keeping the working set in SRAM throughout. On a model with 32 layers and 512-token sequences, eliminating those intermediate HBM writes yields a substantial reduction in memory bandwidth consumption — typically 30–50% for the attention block alone.
On a purpose-built accelerator, the fusion opportunity is broader than on a GPU. Because the compiler knows the exact hardware's SRAM dimensions, scratchpad layout, and memory bandwidth characteristics at design time — not at runtime through profiling — it can make fusion decisions that are globally optimal for the hardware rather than locally optimal within CUDA kernel constraints. The GPU compiler's fusion pass is constrained by the CUDA programming model; an ASIC compiler writes directly to the hardware's dataflow fabric.
Quantization as a Graph Transformation
INT8 post-training quantization (PTQ) is not just a weight-compression step — it's a graph transformation that inserts quantize and dequantize nodes at specific positions and changes the datatype annotations on edges between them. A compiler that handles quantization natively can fuse the quantization scale factors into adjacent operators so that no explicit quantize/dequantize kernel is needed at runtime.
Consider a linear layer followed by GELU followed by another linear layer. In FP16, this is three operators. With INT8 quantization naively applied, it becomes: quantize, INT8 GEMM, dequantize, GELU (FP16), quantize, INT8 GEMM, dequantize — seven operations. A compiler that handles quantization-aware fusion can collapse this: INT8 GEMM with fused scale application, INT8-approximate GELU, INT8 GEMM — three operations, with no format conversion at runtime.
This matters more on fixed-function hardware than on GPUs. GPU INT8 pipelines are flexible enough to absorb the conversion overhead. An ASIC's dataflow pipeline has fixed-width datapaths; format conversions that aren't fused away consume real datapath cycles.
Hardware-Specific Dataflow Generation
The final compilation stage translates the optimized, fused, quantized graph into a hardware-executable dataflow program. This is where the architecture of the target hardware shapes the output most directly.
For a systolic array architecture (the common structure in Google TPUs and many custom accelerators), the compiler's primary job is tiling: partitioning each matrix multiply into tiles that fit the systolic array's dimensions, scheduling tiles to maximize pipeline utilization, and pre-fetching the next tile's weight data while the current tile executes. The systolic array achieves peak utilization only when the tile dimensions match its internal processing element grid — typically a power-of-two square for square arrays, or a specific rectangular ratio for asymmetric designs. A model whose matrix dimensions don't factor cleanly into the array's tile size either pads (wasting array utilization) or requires the compiler to stitch partial tiles (adding scheduling overhead).
This is why the specific model shapes — weight matrix dimensions, attention head counts, hidden dimensions — matter for hardware compatibility before any compilation work begins. A model designed with 4096-dimensional hidden states and 32 attention heads fits cleanly into a systolic array with 128x128 processing elements. A model with 5120 hidden dimensions and 40 heads requires more careful tiling to avoid significant padding waste.
What the Compiler Cannot Do Without the Hardware
There's an important asymmetry that shapes how this work gets done in practice. A great compiler is necessary but not sufficient for hardware-level efficiency. The compiler can only extract what the hardware physically supports. If the hardware doesn't have a fixed-function attention accelerator, the compiler cannot fuse attention into a single-cycle operation — it must decompose it into the best available primitive sequence. If the SRAM budget is smaller than the KV-cache working set for the target model, the compiler must tier to HBM regardless of how aggressively it fuses everything else.
This is why the model intake process — before RTL is written — matters. The graph compilation target and the hardware architecture specification must be developed together, with the specific model's operator mix, tensor shapes, and memory access patterns shaping both the silicon design and the compiler's optimization strategy simultaneously. Compilers written post-silicon for general hardware are always making the best of constraints that were set without the workload in mind.
The value of graph specialization is real and measurable. The ceiling it can reach depends entirely on what the hardware was designed to support in the first place.