After the Compiler Finishes - The Relax Virtual Machine Runtime of apache/tvm
Our first walk of apache/tvm followed the compiler’s happy path: Relax programs built by the BlockBuilder, rewritten by fifty-six C++ passes, lowered into TensorIR, and scheduled by dlight rules. That tour stopped, deliberately, at the boundary marked “executable” — the...
From Python Call to cuLaunchKernelEx - The Triton Launch Path
Our first walk of openai/triton followed the compiler’s front door: how a decorated Python function becomes an AST, then TTIR, then the GPU-shaped dialects, then PTX or GCN. That leaves the other half of the story untouched — the part...
The Quiet Workhorse of RAG - Inside huggingface/text-embeddings-inference
Every retrieval system has a generation model that gets the applause and an embedding model that does the work, and the embedding side is exactly where huggingface/text-embeddings-inference lives. TEI — as everyone calls it — is a Rust server whose...
From Run() to Buffers - The ONNX Runtime Execution Machinery
Our first walk of microsoft/onnxruntime stayed at the architectural level: the versioned C API, the fusion transformers, MLAS, the CUDA provider, and the training stack. That post ends where every session ends up on a hot path — a call...
Engine Actions and the Token Treadmill - The C++ Serve Core of mlc-ai/mlc-llm
Our first walk of mlc-ai/mlc-llm told the compile-and-serve story at the repository scale: passes that fuse weight dequantization into matmuls, quantization schemes, the async Python engine, and a note that a compact C++ core does the real work. That C++...
Inside TurboMind: The C++ Engine Core of InternLM/lmdeploy
Our first visit to InternLM/lmdeploy stayed above the waterline: the CLI, the lite quantization toolkit, the OpenAI-compatible server, and the fork where a request picks either the readable PyTorch engine or the compiled TurboMind core. This visit goes under the...
Inside the Seek Loop - Timestamps, Fallback, and Hallucination Guards in SYST...
Our first walk of SYSTRAN/faster-whisper introduced the cast: WhisperModel, the batched wrapper, PyAV audio, the Silero VAD, and the claim that a reimplementation on CTranslate2 makes the same model four times faster. What that tour did not open up is...
From Export to Bare Metal - The .pte Path Through pytorch/executorch
Every framework we have toured in this series — Triton for kernels, vLLM for throughput, MLC for cross-platform compilation — assumes you have an operating system, a Python runtime, and memory to spare. ExecuTorch, PyTorch’s edge sibling, drops all three...
Inference Rebuilt for Batching - The Ragged Engine of DeepSpeed Inference v2
Our first walk of microsoft/DeepSpeed stayed on the training side of the tree: ZeRO’s three sharding stages, the pipe engine’s micro-batch schedules, the launcher that SSHes into clusters, and the accelerator layer that speaks for six hardware vendors. That tour...
Inside Microsoft's BitNet: Ternary Lookup-Table Kernels on CPU and CUDA
Two years of quantization work have convinced most teams that 4-bit is the practical floor for running large language models on everyday hardware. Microsoft’s BitNet refuses that floor entirely. Its weights live in {-1, 0, +1}, roughly 1.58 bits each,...
vLLM: Speculative Drafts and Grammar Gates - A Second Tour of vllm-project/vllm
Our first tour of vLLM walked the main road: requests enter the OpenAI-compatible server, the scheduler batches them, and the executor drives GPU workers that run the model. That post left two guard towers unvisited, and they turn out to...
Monkey-Patch Your Way to 2x Training - Inside unslothai/unsloth
Most fine-tuning libraries you meet are wrappers: they take Hugging Face Trainer, add a config file, and hope the stock kernels are fast enough. Unsloth is the opposite bet. Instead of orchestrating around transformers, it reaches inside them, replacing attention,...
The Scheduler Moved to Rust: A Second Tour of huggingface/text-generation-inf...
Our first tour of TGI at /tgi/ told the maintenance-mode story: a server that officially stepped back and let the ecosystem bloom around it. That framing, while accurate on the marketing page, undersells the code, because the main branch is...
TensorRT-LLM: Many Processes, One Engine - Inside the Executor Lattice
Our first tour of TensorRT-LLM followed the classic question: how does a request become tokens? We walked the configuration contract in LlmArgs, the two engine paths, the C++ batch manager with its KV cache arithmetic, and the serving tier on...
The 52K Recipe That Started a Thousand Fine-Tunes - Inside tatsu-lab/stanford...
Some repositories matter because of their engineering, and some because of what they proved. stanford_alpaca, from five Stanford grad students advised by Tatsunori Hashimoto, is the second kind: fourteen files that showed a 7B LLaMA model fine-tuned on 52K machine-generated...
The Admission Decision: A Third Tour of sgl-project/sglang
Two earlier visits covered SGLang’s broad architecture and its newest serving tricks; the first tour at /sglang/ walked the front door to the GPU, and the second at /sglang-2/ chased the recent performance work. This third visit narrows onto the...
OpenVINO: From Core to InferRequest - Inside openvinotoolkit/openvino
Our recent tours have lived close to the GPU: TensorRT-LLM’s process lattice, vLLM’s core engines, SGLang’s router fan-out. This visit goes somewhere quieter and, arguably, harder — the CPU. OpenVINO has been shipping since before “inference framework” was a job...
Ray, vLLM, and One Controller - Inside OpenRLHF/OpenRLHF
Reinforcement learning from human feedback is the most distributed-shaped problem in LLM training: you need a policy being trained, a reference model frozen in time, a reward model scoring outputs, a critic estimating values, and a generation engine producing rollouts...
The Landlord of Your VRAM - The Scheduler Inside ollama/ollama
Run two models back to back in Ollama and something peculiar happens: the second model loads in about the time a browser tab takes to appear, and when you return to the first, it is warm again. That feeling of...
One Framework from Tuning to Deployment - Inside modelscope/ms-swift
ModelScope’s ms-swift has a scope that makes most fine-tuning frameworks look modest: supervised tuning, preference optimization, GRPO-style reinforcement learning, knowledge distillation, multimodal training, Megatron-backed large runs, quantized export, and deployment, all in one package. The current development line, version 4.7.0.dev0...
LocalAI: One Binary, Many Engines - A Source Tour of mudler/LocalAI
Most serving engines we have toured on this blog are monoliths by conviction: vLLM, SGLang, and TGI all pull the scheduler, the sampler, and the CUDA kernels into one process because batch-level coordination demands it. LocalAI bets the other way....
Five Calls from Data to Fine-Tuned Model - Inside OptimalScale/LMFlow
Every fine-tuning toolbox has a philosophy. axolotl bets on YAML, LLaMA-Factory on a hundred CLI flags, unsloth on monkey-patching, and ms-swift on mixin-composed trainers. LMFlow, from the Statistics and Machine Learning Research Group at HKUST, bets on something subtler: a...
The Ruler Every LLM Paper Trusts - Inside EleutherAI/lm-evaluation-harness
Every benchmark number you have seen attached to an open model - MMLU, HellaSwag, GSM8K, ARC, TruthfulQA - has a decent chance of having been produced by this repository. EleutherAI’s Language Model Evaluation Harness is the framework behind the Open...
The Memory Core - Inside ggml-org/llama.cpp's KV Cache Redesign
The most consequential project in local inference has quietly moved house - the repository now lives as ggml-org/llama.cpp - and while the logo stayed, the internals have been rebuilt around an abstraction that barely existed when this series first toured...