ideas, notes, long reads, and tools.
clean, concise, and clear. Also mirrored at aimlds.org.
-
inspiration
Merging Is Not Training
Model merging combines two or more fine-tuned LLMs into a single model without any gradient updates. No data. No compute budget. No training run. The result inherits capabilities from every source model — if you pick the right algorithm.
-
inspiration
The Tokenizer Is the Bug
Every LLM failure starts with the same invisible step: tokenization. It runs before inference, produces no logs, and degrades outputs silently. Most debugging sessions end at the model. They should start at the tokenizer.
-
inspiration
GGUF Is a Container, Not Just Weights
Every self-hosted AI practitioner downloads .gguf files. Few understand what they are. GGUF is not a weight dump — it is a self-contained container that carries the model, the tokenizer, the quantization scheme, and the chat template in a single file. That design decision changed how open-source models are distributed.
-
inspiration
Flash Attention Is an IO Problem
Standard attention is slow not because of arithmetic — it is slow because of memory traffic. Flash Attention solves the IO problem, not the compute problem. That distinction matters for how you think about every inference optimization that follows.
-
whitepaper
Evals Are Not Optional
Benchmark scores are not evaluations. Contamination is widespread, Goodhart's Law is in effect, and the gap between a leaderboard number and production behaviour is unbridged without a real eval pipeline. This paper defines what evals are, why the major benchmarks are unreliable in isolation, and how to build an evaluation practice that actually catches failures.
-
inspiration
Prefix Caching Is Free Throughput
Automatic Prefix Caching in vLLM reuses already-computed KV cache blocks across requests that share identical prefixes — delivering 30–50% throughput gains and up to 10x latency reduction at zero engineering cost beyond a single configuration flag.
-
inspiration
Continuous Batching: The Throughput Multiplier
Static batching wastes GPU cycles waiting for the slowest request. Continuous batching fixes this at the scheduler level — and the gains are not marginal.
-
inspiration
LoRA Is Not Fine-Tuning
LoRA does not update your model. It adds a thin, low-rank correction on top — and that distinction changes how you think about deployment, switching, and scale.
-
inspiration
Steering Is Not Prompting
Prompts influence what a model says. Activation steering changes what the model is, mid-inference. They are not the same tool.
-
inspiration
The KV Cache Is Your Real Memory Budget
The KV cache — not the model weights — is what limits how many tokens you can generate and how many requests you can serve. Understanding it changes how you provision hardware and tune inference.
-
inspiration
Attention Sinks: The Tokens That Hold Everything Together
Transformers quietly route a disproportionate share of attention to their first tokens — not because those tokens are important, but because softmax needs somewhere to put mass. Understanding this changes how you think about KV cache design.
-
whitepaper
Fine-Tuning, RAG, or Prompting: An Engineering Decision
Three techniques can improve LLM output quality: prompt engineering, retrieval-augmented generation, and fine-tuning. Each solves a different problem. Choosing the wrong one wastes months and produces worse results than the right one done simply.
-
inspiration
The Tool Is Not the Model
A language model does not execute functions. It describes them. The execution lives elsewhere — in your code, your runtime, your responsibility.
-
inspiration
Embeddings Are Not Optional
Every RAG pipeline, semantic search index, and similarity feature runs on embeddings. The generation model gets the credit. The embedding model does the work.
-
inspiration
Temperature Is Not Creativity
Temperature is a probability reshaper, not a creativity dial. Calling it a creativity parameter is a category error — one that leads to misconfigured systems and wasted inference budget.
-
inspiration
Retrieval Is the Weakest Link
RAG systems fail at retrieval, not generation. Engineers blame the LLM. The problem is upstream.
-
inspiration
Prompt Caching Is Free Money
Every time your app resends the same system prompt, you pay to compute it again. Prompt caching eliminates that cost by reusing precomputed KV tensors across requests. It requires no code changes and delivers up to 90% input token savings.
-
booklet
From Free Tier to Sovereignty: Running Inference on Cloud ARM Instances
Free tier cloud compute promises self-hosted AI. The reality is capacity lotteries, region lock-in, and silent deprecation. This booklet documents what actually works, what does not, and how to build an inference setup that survives policy changes.
-
booklet
Ollama Beyond Defaults: Custom Model Paths on Windows and WSL
Ollama assumes default paths. When your models live elsewhere, the documentation stops helping. This booklet covers every configuration path for Windows native, WSL2, and cross-boundary access.
-
booklet
OpenCode with Local Models: Pointing Your Coding Agent at Your Own Inference
OpenCode is a terminal-first AI coding agent. It expects cloud APIs by default. This booklet shows how to wire it to Ollama, vLLM, or any OpenAI-compatible local endpoint — and what breaks when you do.
-
inspiration
Structured Outputs Are a Contract
Constrained generation is not a convenience feature. It is a systems boundary — a contract between your model and every downstream component that consumes its output.
-
booklet
The LocalLLM Engine Stack: One API, Multiple Backends, Zero Lock-in
A single OpenAI-compatible endpoint that routes across Ollama, llama.cpp, and FreeLLMAPI with automatic failover. This booklet documents the architecture, routing logic, and deployment of the localllm-engine.
-
inspiration
The Model Is Not the Agent
An LLM does not call tools. It requests them. The loop is the agent — and most broken agents are broken loops, not broken models.
-
whitepaper
Choosing Your Inference Engine: llama.cpp, Ollama, and vLLM
llama.cpp, Ollama, and vLLM are not interchangeable. They solve different problems at different scales. This paper maps the architectural differences, performance characteristics, and deployment tradeoffs to help you pick the right engine for your workload — and understand why the wrong choice costs you in ways that are hard to undo.
-
inspiration
Speculative Decoding: The Free Tokens
Speculative decoding cuts inference latency 2–3x without changing a single output token. The gain is real. So is the catch.
-
inspiration
Quantization Is a Design Decision
Quantization is not just compression. It is a tradeoff you are making about accuracy, speed, and memory — and it belongs in your architecture docs, not your deployment scripts.
-
inspiration
Your Infrastructure Should Not Need Permission
If a vendor's policy change can delete your workload overnight, you do not have infrastructure. You have a lease.
-
inspiration
Context Is Not Memory
A large context window does not make an LLM remember. It makes it attend. The distinction changes how you build.
-
inspiration
The Cost of Abstraction
Every layer you add is a layer someone else must debug.
-
inspiration
Clarity is Kindness
Clear writing is not a style preference. It is a form of respect.