Dataset Description SOL (Speed Of Light) ExecBench is a real world CUDA kernel benchmarking dataset of 235 kernel level computational workload specifications derived from open source HuggingFace model architectures. The problems span a wide range of AI model workloads — covering text, vision, and speech models' forward and backward passes — and include core algorithms such as matrix multiplications, convolutions, attention variants, mixture of experts, and norms across FP32, BF16, FP16, FP8, and NVFP4 data types. Runtimes will be compared against reference implementation speed but also theoretical hardware "speed of light": the fastest that kernels can run given the hardware's peak compute and memory bandwidth limits. The dataset was constructed by scraping model architectures from the HuggingFace Transformers and Diffusers repositories and using an LLM agent to extract kernel relevant computational subgraphs, which are then transformed into structured JSON definitions. Each entry defines a computational kernel workload including: Symbolic axis definitions (constant, variable, expression based) Tensor input/output specifications (shape, dtype, constraints) A PyTorch reference imple…
We use cookies for essential functionality and analytics. You can accept or reject analytics cookies.Cookie policy