Zyth
NVIDIA · AMD · Apple · WebGPU

Write the kernel.
Match the libraries.

The same Zyth program runs on the CPU, on one GPU or on many. Tensors stay on the device, kernels are written in Zyth, and the heavy paths are measured against PyTorch on the same hardware.

1276TFLOP/s

fp8 on one H100. PyTorch: 1,125.

0.66ms

attention forward and backward. FlashAttention-2: 0.91.

423tokens/s

SmolLM2 1.7B on an H100. Modular MAX: 409.

131tokens/s

Llama 3.2 1B (q4) on an Apple M4. MLX: 123.

Matrix products on an H100

8192 × 8192, TFLOP/s, Zyth against PyTorch on the same GPU.

PrecisionZythPyTorch
fp81,2761,125
bf16782781
TF32406404
f325252

Training in fp8

A transformer feed-forward block (4096 → 16384 → 4096, batch 8,192, AdamW), milliseconds per step on an H100. Turn it on with one line: @mixed_precision(compute: fp8).

Whatms per step
Zyth fp811.9
PyTorch torchao float8, torch.compile12.7
Zyth bf1613.0
PyTorch bf16, compiled13.9
PyTorch torchao float8, eager20.8

Delayed scaling (scaling: delayed, as NVIDIA's Transformer Engine) saves one pass over every tensor, and fp8 also works with sharded weights and across processes.

Attention

FlashAttention-2 style kernels, forward and backward, bf16, milliseconds on an H100.

ShapeZythFlashAttention-2
B 8, T 1,024, 12 heads of 64, causal0.660.91
B 4, T 4,096, 8 × 64, causal1.641.91
B 4, T 4,096, 8 × 642.412.57
B 4, T 4,096, 8 × 1284.714.58

Language models

The built-in engine has a paged cache, continuous batching and tensor parallelism. Greedy decoding, tokens per second:

Model and GPUZythCompared with
SmolLM2 1.7B, H100423 (f16), 657 (q4_0)Modular MAX 409 (bf16)
Qwen2.5 7B, 32 prompts, 2 H1003,152one H100: 2,214
Llama 3.2 1B, 32 prompts, RTX 3090987one at a time: 81

Many GPUs and machines

  • Data parallel: fit(model, data, loss, gpus: 8), gradients combined with NCCL while the backward pass runs.
  • Sharded weights for models bigger than one GPU, tensor parallel layers as Megatron, pipelines (1F1B), experts for mixture-of-experts, ring attention for long sequences.
  • zyth launch runs one process per GPU across machines, under Slurm or Kubernetes, and restarts every process from the newest checkpoint when one fails.
zyth launch --nproc 8 --nnodes 4 --node-rank 0 --master node0 train.zyth

Apple silicon, natively through Metal

On Macs, tensors live in memory shared by the CPU and the GPU. A kernel fn compiles to Metal Shading Language, tensor-core tiles (mma) become Apple's simdgroup_matrix products, and bf16 runs on the GPU. Large products go to Metal Performance Shaders, and attention and convolutions to MPSGraph. Measured on an Apple M4:

Language model, greedy decodingZythMLX 0.29PyTorch 2.8 MPS
SmolLM2 135M, f16 / q8 / q4, tokens/s309 / 412 / 623302 / 425 / 60168 (f16)
Llama 3.2 1B, f16 / q8 / q4, tokens/s41.6 / 71.0 / 130.940.2 / 71.8 / 123.334.4 (f16)
WhatZythPyTorch 2.8 MPS
f32 product, 4096 × 4096, GFLOP/s3,4553,456
MLP training step (1024-4096-1024, AdamW), ms11.215.5
Attention, B 8, T 1,024, not causal, ms11.721.8
Attention, B 8, T 1,024, causal, ms23.521.7

Zyth is ahead in f16 and q4, MLX slightly ahead in q8. The same text comes out as on the CPU in every precision. Llama 3.2 1B in f16 runs at the M4's memory bandwidth.

Several GPU makers in one program

One Linux build drives NVIDIA and AMD GPUs through CUDA and HIP and every other GPU through WebGPU, chosen per device while the program runs. zyth doctor lists each GPU with the backend that runs it, and gpu("nvidia", 0) or gpu("best") picks one. Sharded tensors and collectives (all_reduce, all_gather, reduce_scatter) work on every backend: through NCCL when every piece is on an NVIDIA GPU, through peer or host copies otherwise. Training with fit(..., gpus: 3) across an RTX 3090 and two Vulkan devices gives the same losses as one GPU and the CPU.

let a = gpu("nvidia", 0)
let b = gpu("any", 1)
let x = tensor.randn(4096, 4096).to(a)
let y = x.to(b)        // moves through host memory between backends

Kernels in Zyth

kernel fn saxpy(a: f32, x: tensor<f32>, y: mut tensor<f32>) {
    let i = thread.global
    if i < y.len() {
        y[i] = a * x[i] + y[i]
    }
}

A kernel fn compiles to CUDA for NVIDIA, HIP for AMD, Metal Shading Language for Apple GPUs and WGSL for WebGPU, and has shared memory, atomics and tensor-core tiles (mma). On an H100 this saxpy moves 2.6 TB/s.

All numbers from Zyth's measurement log, on rented H100 SXM GPUs (RunPod), a local RTX 3090 and an Apple M4, October and November 2026. The AMD path is written but has not run on an AMD GPU; several makers in one program was tested with software Vulkan devices beside an NVIDIA GPU.