Write the kernel.
Match the libraries.
The same Zyth program runs on the CPU, on one GPU or on many. Tensors stay on the device, kernels are written in Zyth, and the heavy paths are measured against PyTorch on the same hardware.
fp8 on one H100. PyTorch: 1,125.
attention forward and backward. FlashAttention-2: 0.91.
SmolLM2 1.7B on an H100. Modular MAX: 409.
Llama 3.2 1B (q4) on an Apple M4. MLX: 123.
Matrix products on an H100
8192 × 8192, TFLOP/s, Zyth against PyTorch on the same GPU.
| Precision | Zyth | PyTorch |
|---|---|---|
| fp8 | 1,276 | 1,125 |
| bf16 | 782 | 781 |
| TF32 | 406 | 404 |
| f32 | 52 | 52 |
Training in fp8
A transformer feed-forward block (4096 → 16384 → 4096, batch 8,192, AdamW), milliseconds per step on an H100. Turn it on with one line: @mixed_precision(compute: fp8).
| What | ms per step |
|---|---|
| Zyth fp8 | 11.9 |
PyTorch torchao float8, torch.compile | 12.7 |
| Zyth bf16 | 13.0 |
| PyTorch bf16, compiled | 13.9 |
| PyTorch torchao float8, eager | 20.8 |
Delayed scaling (scaling: delayed, as NVIDIA's Transformer Engine) saves one pass over every tensor, and fp8 also works with sharded weights and across processes.
Attention
FlashAttention-2 style kernels, forward and backward, bf16, milliseconds on an H100.
| Shape | Zyth | FlashAttention-2 |
|---|---|---|
| B 8, T 1,024, 12 heads of 64, causal | 0.66 | 0.91 |
| B 4, T 4,096, 8 × 64, causal | 1.64 | 1.91 |
| B 4, T 4,096, 8 × 64 | 2.41 | 2.57 |
| B 4, T 4,096, 8 × 128 | 4.71 | 4.58 |
Language models
The built-in engine has a paged cache, continuous batching and tensor parallelism. Greedy decoding, tokens per second:
| Model and GPU | Zyth | Compared with |
|---|---|---|
| SmolLM2 1.7B, H100 | 423 (f16), 657 (q4_0) | Modular MAX 409 (bf16) |
| Qwen2.5 7B, 32 prompts, 2 H100 | 3,152 | one H100: 2,214 |
| Llama 3.2 1B, 32 prompts, RTX 3090 | 987 | one at a time: 81 |
Many GPUs and machines
- Data parallel:
fit(model, data, loss, gpus: 8), gradients combined with NCCL while the backward pass runs. - Sharded weights for models bigger than one GPU, tensor parallel layers as Megatron, pipelines (1F1B), experts for mixture-of-experts, ring attention for long sequences.
zyth launchruns one process per GPU across machines, under Slurm or Kubernetes, and restarts every process from the newest checkpoint when one fails.
zyth launch --nproc 8 --nnodes 4 --node-rank 0 --master node0 train.zyth
Apple silicon, natively through Metal
On Macs, tensors live in memory shared by the CPU and the GPU. A kernel fn compiles to Metal Shading Language, tensor-core tiles (mma) become Apple's simdgroup_matrix products, and bf16 runs on the GPU. Large products go to Metal Performance Shaders, and attention and convolutions to MPSGraph. Measured on an Apple M4:
| Language model, greedy decoding | Zyth | MLX 0.29 | PyTorch 2.8 MPS |
|---|---|---|---|
| SmolLM2 135M, f16 / q8 / q4, tokens/s | 309 / 412 / 623 | 302 / 425 / 601 | 68 (f16) |
| Llama 3.2 1B, f16 / q8 / q4, tokens/s | 41.6 / 71.0 / 130.9 | 40.2 / 71.8 / 123.3 | 34.4 (f16) |
| What | Zyth | PyTorch 2.8 MPS |
|---|---|---|
| f32 product, 4096 × 4096, GFLOP/s | 3,455 | 3,456 |
| MLP training step (1024-4096-1024, AdamW), ms | 11.2 | 15.5 |
| Attention, B 8, T 1,024, not causal, ms | 11.7 | 21.8 |
| Attention, B 8, T 1,024, causal, ms | 23.5 | 21.7 |
Zyth is ahead in f16 and q4, MLX slightly ahead in q8. The same text comes out as on the CPU in every precision. Llama 3.2 1B in f16 runs at the M4's memory bandwidth.
Several GPU makers in one program
One Linux build drives NVIDIA and AMD GPUs through CUDA and HIP and every other GPU through WebGPU, chosen per device while the program runs. zyth doctor lists each GPU with the backend that runs it, and gpu("nvidia", 0) or gpu("best") picks one. Sharded tensors and collectives (all_reduce, all_gather, reduce_scatter) work on every backend: through NCCL when every piece is on an NVIDIA GPU, through peer or host copies otherwise. Training with fit(..., gpus: 3) across an RTX 3090 and two Vulkan devices gives the same losses as one GPU and the CPU.
let a = gpu("nvidia", 0)
let b = gpu("any", 1)
let x = tensor.randn(4096, 4096).to(a)
let y = x.to(b) // moves through host memory between backends
Kernels in Zyth
kernel fn saxpy(a: f32, x: tensor<f32>, y: mut tensor<f32>) {
let i = thread.global
if i < y.len() {
y[i] = a * x[i] + y[i]
}
}
A kernel fn compiles to CUDA for NVIDIA, HIP for AMD, Metal Shading Language for Apple GPUs and WGSL for WebGPU, and has shared memory, atomics and tensor-core tiles (mma). On an H100 this saxpy moves 2.6 TB/s.
All numbers from Zyth's measurement log, on rented H100 SXM GPUs (RunPod), a local RTX 3090 and an Apple M4, October and November 2026. The AMD path is written but has not run on an AMD GPU; several makers in one program was tested with software Vulkan devices beside an NVIDIA GPU.