engine-mlx — how a chat request runs

One OpenAI-compatible request, from HTTP to tokens, entirely on your Mac

engine-mlx — how a chat request runs One OpenAI-compatible request, from HTTP to tokens, entirely on your Mac 01 / HTTP API (axum) 02 / MLX worker thread 03 / MLX / Metal (mlx-c) Intake Tokenize + prefill Decode loop Single GPU thread (mpsc queue) Graph on Metal OpenAI client · POST /v1/chat/completions · HTTP API (axum) › Intake OpenAI client POST /v1/chat/completions chat_completions · parse messages, spawn_blocking · HTTP API (axum) › Intake chat_completions parse messages, spawn_blocking SSE response · word-by-word chunks · HTTP API (axum) › Decode loop · not per-token SSE response word-by-word chunks not per-token Infer queue · mpsc → GPU thread · MLX worker thread › Single GPU thread (mpsc queue) › Intake · single-threaded GPU Infer queue mpsc → GPU thread single-threaded GPU Static KV cache · pre-allocated BF16 buffers · MLX worker thread › Single GPU thread (mpsc queue) › Tokenize + prefill Static KV cache pre-allocated BF16 buffers Tokenize · pure-Rust BPE encode · MLX worker thread › Single GPU thread (mpsc queue) › Tokenize + prefill Tokenize pure-Rust BPE encode Prefill · embed + full-seq forward · MLX worker thread › Single GPU thread (mpsc queue) › Tokenize + prefill · alloc static KV Prefill embed + full-seq forward alloc static KV Decode step · 1 token / step, argmax in-graph · MLX worker thread › Single GPU thread (mpsc queue) › Decode loop Decode step 1 token / step, argmax in-graph Detokenize · token ids → text · MLX worker thread › Single GPU thread (mpsc queue) › Decode loop Detokenize token ids → text Compiled step · scatter KV + masked SDPA + argmax · MLX / Metal (mlx-c) › Graph on Metal › Decode loop Compiled step scatter KV + masked SDPA + argmax async_eval on Metal · mlx-c closure apply · MLX / Metal (mlx-c) › Graph on Metal › Decode loop · GPU async_eval on Metal mlx-c closure apply GPU first logits generated ids enqueue launch init cache scatter + swap full text token ids JSON request forward step next token · until EOS / max_tokens Legend User UI Agent logic Tool action Context / trace Cloud service External system