vLLM: Speculative Drafts and Grammar Gates - A Second Tour of vllm-project/vllm
Our first tour of vLLM walked the main road: requests enter the OpenAI-compatible server, the scheduler batches them, and the executor drives GPU workers that run the model. That post left two guard towers unvisited, and they turn out to...
Monkey-Patch Your Way to 2x Training - Inside unslothai/unsloth
Most fine-tuning libraries you meet are wrappers: they take Hugging Face Trainer, add a config file, and hope the stock kernels are fast enough. Unsloth is the opposite bet. Instead of orchestrating around transformers, it reaches inside them, replacing attention,...
The Scheduler Moved to Rust: A Second Tour of huggingface/text-generation-inf...
Our first tour of TGI at /tgi/ told the maintenance-mode story: a server that officially stepped back and let the ecosystem bloom around it. That framing, while accurate on the marketing page, undersells the code, because the main branch is...
TensorRT-LLM: Many Processes, One Engine - Inside the Executor Lattice
Our first tour of TensorRT-LLM followed the classic question: how does a request become tokens? We walked the configuration contract in LlmArgs, the two engine paths, the C++ batch manager with its KV cache arithmetic, and the serving tier on...
The 52K Recipe That Started a Thousand Fine-Tunes - Inside tatsu-lab/stanford...
Some repositories matter because of their engineering, and some because of what they proved. stanford_alpaca, from five Stanford grad students advised by Tatsunori Hashimoto, is the second kind: fourteen files that showed a 7B LLaMA model fine-tuned on 52K machine-generated...
The Admission Decision: A Third Tour of sgl-project/sglang
Two earlier visits covered SGLang’s broad architecture and its newest serving tricks; the first tour at /sglang/ walked the front door to the GPU, and the second at /sglang-2/ chased the recent performance work. This third visit narrows onto the...
OpenVINO: From Core to InferRequest - Inside openvinotoolkit/openvino
Our recent tours have lived close to the GPU: TensorRT-LLM’s process lattice, vLLM’s core engines, SGLang’s router fan-out. This visit goes somewhere quieter and, arguably, harder — the CPU. OpenVINO has been shipping since before “inference framework” was a job...
Ray, vLLM, and One Controller - Inside OpenRLHF/OpenRLHF
Reinforcement learning from human feedback is the most distributed-shaped problem in LLM training: you need a policy being trained, a reference model frozen in time, a reward model scoring outputs, a critic estimating values, and a generation engine producing rollouts...
The Landlord of Your VRAM - The Scheduler Inside ollama/ollama
Run two models back to back in Ollama and something peculiar happens: the second model loads in about the time a browser tab takes to appear, and when you return to the first, it is warm again. That feeling of...
One Framework from Tuning to Deployment - Inside modelscope/ms-swift
ModelScope’s ms-swift has a scope that makes most fine-tuning frameworks look modest: supervised tuning, preference optimization, GRPO-style reinforcement learning, knowledge distillation, multimodal training, Megatron-backed large runs, quantized export, and deployment, all in one package. The current development line, version 4.7.0.dev0...
LocalAI: One Binary, Many Engines - A Source Tour of mudler/LocalAI
Most serving engines we have toured on this blog are monoliths by conviction: vLLM, SGLang, and TGI all pull the scheduler, the sampler, and the CUDA kernels into one process because batch-level coordination demands it. LocalAI bets the other way....
Five Calls from Data to Fine-Tuned Model - Inside OptimalScale/LMFlow
Every fine-tuning toolbox has a philosophy. axolotl bets on YAML, LLaMA-Factory on a hundred CLI flags, unsloth on monkey-patching, and ms-swift on mixin-composed trainers. LMFlow, from the Statistics and Machine Learning Research Group at HKUST, bets on something subtler: a...
The Ruler Every LLM Paper Trusts - Inside EleutherAI/lm-evaluation-harness
Every benchmark number you have seen attached to an open model - MMLU, HellaSwag, GSM8K, ARC, TruthfulQA - has a decent chance of having been produced by this repository. EleutherAI’s Language Model Evaluation Harness is the framework behind the Open...
The Memory Core - Inside ggml-org/llama.cpp's KV Cache Redesign
The most consequential project in local inference has quietly moved house - the repository now lives as ggml-org/llama.cpp - and while the logo stayed, the internals have been rebuilt around an abstraction that barely existed when this series first toured...
100+ Models, One Command - Inside hiyouga/LLaMA-Factory
Where axolotl bets on one exhaustive YAML schema, LLaMA-Factory bets on breadth: one CLI that fine-tunes over a hundred model families, one data loader that speaks every open dataset format, and one stage dispatcher that walks from continuous pretraining through...
The Traffic Cop of the AI Gateway - The Router Inside BerriAI/litellm
Every company that ships LLM features eventually writes the same program: an HTTP front door, a list of model deployments, and a way to spread traffic across them without falling over. LiteLLM, whose README bills it in one line as...
Xinference: Where Models Live - A Second Tour of xorbitsai/inference
Our first visit to Xinference at /xinference/ walked the wide road: actors, the REST surface, the model catalog, and how one command becomes a cluster. This second tour goes down the narrow stairwell that first visit skipped: placement, the machinery...
Your Language Model Is Secretly a Reward Model - Inside eric-mitchell/direct-...
In May 2023 a Stanford team - Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn - posted “Direct Preference Optimization: Your Language Model Is Secretly a Reward Model” (arXiv 2305.18290), and the title carried...
One YAML, Every Fine-Tune - Inside axolotl-ai-cloud/axolotl
Fine-tuning an open model used to mean writing a training script: dataset glue, tokenizer surgery, trainer arguments, sharding flags, and a prayer that the checkpoint resumes. Axolotl replaces the script with a single YAML file and a codebase whose whole...
The Recipe Book Behind Zephyr and SmolLM3 - Inside huggingface/alignment-hand...
Some repositories impress with volume, and some with restraint. The Alignment Handbook, from Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Carlos M. Patino, Alexander M. Rush, and Thomas Wolf at Hugging Face, is...
The Machine Learning Compiler That Speaks Python - Inside apache/tvm
Most deep learning systems ship as fixed inference engines: you bring a model, the engine runs it, and the optimizations are whatever the vendor froze into the release. Apache TVM is the opposite bet. Its README calls it an Open...
A Python Compiler for the GPU - Inside openai/triton
Writing a fast GPU kernel by hand means warp shuffles, shared memory swizzling, and a week in the profiler, which is why most deep learning teams would rather not. Triton, started by Philippe Tillet and described in the 2019 MAPL...
FP8 Without Rewriting Your Stack - Inside NVIDIA/TransformerEngine
Training large models in FP8 promises roughly double the throughput and half the memory traffic of BF16, and the 2022 paper FP8 Formats for Deep Learning is the reason the industry believes it. The catch has always been numerics: eight-bit...
Post-Training, the PyTorch Way - Inside pytorch/torchtune
Fine-tuning an LLM used to mean choosing between a framework that hid PyTorch from you and a notebook that hid correctness from you. Torchtune, released by Meta’s PyTorch team in April 2024 under the BSD 3-Clause license, took a third...