The Hardware Challenge of Speculative Decoding
Speculative decoding—using a small draft model to generate candidate tokens verified in parallel by a large target model—yields 3x token generation speedups. However, running two distinct models concurrently on standard hardware causes resource contention and scheduling friction. InferAccel solves this through dedicated silicon co-design.
Dedicated Draft Execution Units and Target Tensor Cores
InferAccel accelerators feature partitioned hardware execution domains: lightweight draft execution units optimized for high-speed sequential token sampling, operating alongside massive target tensor cores dedicated to parallel batch verification passes.
Zero-Copy Token Verification Pipelines
Passing candidate token sequences from draft units to target verification cores on standard systems introduces PCIe or system memory overhead. InferAccel utilizes on-die ring buses and shared SRAM buffers, passing token candidates between draft and target engines in single-digit nanoseconds.
Adaptive Speculation Length and Acceptance Optimization
Draft accuracy fluctuates depending on prompt complexity. InferAccel's hardware runtime monitors draft acceptance rates in real time, dynamically scaling speculation length (K-tokens) to maintain optimal throughput without wasting compute on rejected tokens.
