Faster AI Responses.
Less GPU Cost.
Your AI model may have more performance headroom than you think. In our Llama 3.2 tests on an NVIDIA Tesla T4, the fastest inference setup generated output 10.3× faster than the tested Hugging Face/PyTorch baseline, 112.3 versus 10.9 tokens per second at a 4,096-token context. The right serving stack can make a substantial difference to response speed and GPU capacity.
Specific benchmark results, not guaranteed outcomes for another model, GPU, or deployment. *Memory measurement notes below.
Longer prompts can make AI slower and harder to serve.
We investigated where the time and memory went, then tested custom attention paths and a different inference runtime. The goal was to find the best validated deployment option, not simply build a faster-looking kernel.
01 / Identify
Measure the backend the system actually runs. In this setup, PyTorch used a slower attention fallback.
02 / Optimize
Build and test a custom Triton path and a native CUDA path, then compare with an established runtime.
03 / Validate
Check numerical and model behavior, output speed, and GPU memory before choosing a route.
Four approaches. One clear performance range.
All four results use the Llama 3.2-1B model family on a Tesla T4 at 4,096-token context. The setups differ in model format and inference software; this is a deployment-stack comparison, not a race between isolated kernels.
Output speed higher is better
What we learned
PyTorch baseline: 10.9 tokens/s in this configuration.
Custom Triton: 26.8 tokens/s inside the framework path.
Custom CUDA in llama.cpp: 78.8 tokens/s.
Built-in llama.cpp path: 112.3 tokens/s, the fastest tested option. Selecting the mature path was the best deployment decision here.
| Tested setup | Output at 4K | Relative to PyTorch baseline |
|---|---|---|
| Hugging Face/PyTorch baseline | 10.9 tokens/s | 1.0× |
| PyTorch with custom Triton attention | 26.8 tokens/s | 2.46× |
| llama.cpp with custom CUDA Split-K | 78.8 tokens/s | 7.2× across stacks |
| llama.cpp with built-in attention | 112.3 tokens/s | 10.3× across stacks |
See what happens as prompts get longer.
These line graphs keep the FP16 Hugging Face/PyTorch setup fixed and compare its available attention path with the custom Triton path. Teal is the optimized route.
Output speed higher is better
At 6,144 tokens: Triton 21.8 vs PyTorch 6.1 tokens/s (3.57×).
Peak allocated GPU memory lower is better
At 6,144 tokens: Triton 3,018 MB vs PyTorch 13,388 MB for the full run.
Peak memory covers the measured prefill-and-decode run, not memory used per token. The Triton path used slightly more memory at the two shortest tested contexts.
How did the built-in llama.cpp CUDA path compare?
On the same Llama 3.2-1B model family and Tesla T4, the built-in llama.cpp path generated more tokens per second at every shared context length. These are complete deployment stacks: PyTorch used FP16 weights, while llama.cpp used a quantized Q4_K_M model.
Output speed higher is better
At 4,096 tokens: 10.9 tok/s for the PyTorch baseline, versus 112.3 tok/s for llama.cpp's built-in CUDA attention.
Reported peak GPU memory lower is better
At 4,096 tokens: 7,380 MB peak allocated by PyTorch, versus 1,309 MiB llama.cpp/Split-K process peak. The built-in llama.cpp memory figure was not reported.
The memory graph converts both series to decimal GB for display, but the tools report different kinds of peaks. The model format and runtime also differ. It shows the observed footprint gap, not a controlled memory saving or measured GPU cost reduction.
The native setup had a much smaller reported footprint.
Tested PyTorch baseline
7,380 MB
Peak GPU allocation in the framework run.
Custom CUDA in llama.cpp
1,309 MiB
Reported process peak GPU memory in the native run (about 1,373 MB).
Turn benchmark evidence into a deployment decision.
BynaryCode can assess your model, traffic, latency goals, GPU setup, and current spending. We profile the bottleneck, test promising routes, and compare speed, memory, output quality, and cost per useful result before recommending a change.
Faster generation
Identify which part of inference limits your real workload.
Memory headroom
Evaluate runtime, cache, and quantization choices that could improve capacity.
Cost clarity
Model potential GPU savings using actual utilization and pricing instead of assuming a benchmark speedup becomes a bill reduction.
Technical details
Model/hardware: meta-llama/Llama-3.2-1B, Tesla T4; framework path FP16, llama.cpp path Q4_K_M GGUF. The tested PyTorch SDPA path fell back to math attention. The native custom Split-K kernel was validated against numerical and model-level references; it improved an earlier custom CUDA prototype but remained slower than llama.cpp's built-in path.
How much could your inference stack improve?
Share your model, current setup, and goals. We’ll identify a focused test and measure the result before recommending a change.