diff --git a/paper/blis.tex b/paper/blis.tex index 23ba010..24061f7 100644 --- a/paper/blis.tex +++ b/paper/blis.tex @@ -377,33 +377,631 @@ \subsection{The Capacity Planning Error Cascade} \section{BLIS: Blackbox Inference Simulator} \label{sec:design} -(To be written: factorized architecture---learned execution physics, pluggable policies, fixed constraints.) - +Serving a large language model requires reasoning simultaneously about GPU execution +physics, cluster-wide control-plane decisions, and time-varying workload behavior. +In a running system these concerns are entangled: a routing policy change reshapes +KV-cache hit rates, which alters batch composition, which changes step time. +Yet they are \emph{analytically separable}, and that separation is the design center +of BLIS. + +\paragraph{Design principle.} +BLIS factorizes inference simulation into three independent, swappable axes +(Figure~\ref{fig:blis-arch}): +(1)~\emph{learned execution physics}---one of four interchangeable latency-model +backends predicts how long a batch step takes; +(2)~\emph{pluggable control-plane policies}---admission, routing, scheduling, and +memory management replicate the deterministic serving logic faithfully in software, +without GPUs; +and~(3)~\emph{composable workload models}---a declarative specification generates +realistic request streams without a live deployment. +Any axis can be replaced without touching the other two. +This factorization makes BLIS \emph{portable} (no engine source code required), +\emph{accurate} (7\% MAPE in step-time prediction across four architectures, +\S\ref{subsubsec:training}), and \emph{fast} (a 1{,}000-request simulation +completes in under two seconds on a commodity CPU). + +The simulation core is a deterministic discrete-event simulator (DES) with a +priority event queue ordered by \emph{(timestamp, event type, sequence ID)}. +This ordering delivers two properties operators need from a fitness function: +\textbf{reproducibility}---identical seeds produce byte-identical outputs, enabling +repeatable algorithm search---and \textbf{conservation}---every injected request +is accounted for as completed, queued, running, or dropped, eliminating simulator +artifact from throughput measurements. + +% -------------------------------------------------------------------------- \subsection{Architecture Overview} \label{subsec:design-arch} + +Figure~\ref{fig:blis-arch} shows the full BLIS architecture. +A \code{WorkloadSpec} drives a generator that emits a time-ordered request stream; +requests enter the cluster simulator, which applies admission control, routes each +request to one of $N$ instance simulators, and collects per-request metrics at +completion. +Each instance simulator runs its own DES loop: at every \emph{step} it orders its +wait queue via a pluggable \code{InstanceScheduler}, forms a batch via +\code{BatchFormation}, allocates KV blocks via \code{KVStore}, and advances the +clock by the step duration from \code{LatencyModel}. +The cluster layer introduces \textsc{Admission} and \textsc{Routing} event types +that fire before any instance sees a request, so cluster-level policy decisions +can be recorded, replayed, and swapped independently of per-instance execution. + +% ---- Figure A: BLIS Architecture ---- +\begin{figure*}[t] +\centering +\begin{tikzpicture}[ + comp/.style={draw, rounded corners=3pt, fill=white, + minimum width=2.05cm, minimum height=0.50cm, + align=center, font=\footnotesize, inner sep=2pt}, + plug/.style={draw=gray!55, rounded corners=3pt, fill=white, dashed, + minimum width=2.05cm, minimum height=0.50cm, + align=center, font=\footnotesize, inner sep=2pt}, + hdr/.style={font=\small\bfseries, align=center}, + arr/.style={->, >=Stealth, semithick}, + darr/.style={->, >=Stealth, semithick, dashed, green!50!black}, +] +%% Background colour bands +\fill[blue!5, rounded corners=4pt] (-1.35,-2.55) rectangle (1.55, 0.95); +\fill[orange!5, rounded corners=4pt] (1.70,-2.55) rectangle (10.05,0.95); +\fill[green!5, rounded corners=4pt] (10.20,-2.55) rectangle (13.45,0.95); + +%% Column headers +\node[hdr, blue!65!black] at (0.10, 1.25) {Workload\\Model}; +\node[hdr, orange!70!black] at (5.87, 1.25) {Control-Plane Policies \emph{(pluggable)}}; +\node[hdr, green!55!black] at (11.82,1.25) {Execution\\Physics}; + +%% ---- Workload column (x≈0.10) ---- +\node[comp, fill=blue!12] (ws) at (0.10, 0.52) {\textbf{WorkloadSpec}}; +\node[comp, fill=blue!5] at (0.10,-0.08) {arrival process}; +\node[comp, fill=blue!5] at (0.10,-0.60) {token distribution}; +\node[comp, fill=blue!5] at (0.10,-1.12) {SLO class}; +\node[comp, fill=blue!5] at (0.10,-1.64) {cohort dynamics}; + +%% ---- Admission column (x≈3.10) ---- +\node[comp, fill=orange!14] (adm) at (3.10, 0.52) {Admission}; +\node[plug] at (3.10,-0.08) {AlwaysAdmit}; +\node[plug] at (3.10,-0.60) {TokenBucket}; + +%% ---- Router column (x≈5.87) ---- +\node[comp, fill=orange!14] (rtr) at (5.87, 0.52) {Router}; +\node[plug, text width=3.0cm, align=center, font=\tiny, inner sep=3pt] + at (5.87,-0.42) {prefix-affinity \\ queue-depth \\ kv-utilization \\ load-balance}; +\node[font=\tiny, gray] at (5.87,-1.22) {weighted composite scorer}; + +%% ---- Instance column (x≈8.64) ---- +\node[comp, fill=orange!14, minimum width=2.40cm] (inst) at (8.64, 0.52) + {Instance ${\times}\,N$}; +\node[comp, fill=orange!5] at (8.64,-0.08) {Scheduler}; +\node[comp, fill=orange!5] at (8.64,-0.60) {Batch formation}; +\node[comp, fill=orange!5] at (8.64,-1.12) {KV Store}; +\node[plug, fill=green!8] (lm) at (8.64,-1.64) {Latency Model $\leftarrow$}; + +%% ---- Physics column (x≈11.82) ---- +\node[plug] at (11.82, 0.52) {Blackbox}; +\node[plug] at (11.82,-0.08) {Roofline}; +\node[plug] at (11.82,-0.60) {Cross-model}; +\node[plug, thick, fill=green!18] (tr) at (11.82,-1.12) {\textbf{Trained-roofline}}; +\node[font=\tiny, gray, align=center] at (11.82,-1.72) + {$\leftarrow$ no data\hspace{1.6em}data-efficient $\rightarrow$}; + +%% ---- Metrics bar ---- +\node[comp, fill=gray!12, minimum width=5.0cm] (met) + at (5.87,-2.18) {Metrics:\enspace TTFT\enspace TPOT\enspace E2E\enspace SLO}; + +%% ---- Arrows ---- +\draw[arr] (ws.east) -- node[above,font=\tiny]{requests} (adm.west); +\draw[arr] (adm.east) -- (rtr.west); +\draw[arr] (rtr.east) -- (inst.west); +\draw[darr] (tr.west) -- ++(-0.45,0) |- (lm.east); +\draw[arr] (inst) -- ++(0,-0.38) -| (met); +\end{tikzpicture} +\caption{\textbf{BLIS architecture.} + Three swappable axes: workload model, control-plane policies (dashed borders = + pluggable implementations), and execution physics. + Requests flow left to right through admission, routing, and instance simulation to + metrics. + The latency model (green) is selected from four backends spanning the + data-availability spectrum; the trained-roofline backend (\S\ref{subsubsec:trained-roofline}) + achieves 7\% MAPE with 13 production traces.} +\label{fig:blis-arch} +\end{figure*} + +% -------------------------------------------------------------------------- \subsection{Learned Execution Physics} \label{subsec:design-latency} -\subsubsection{The Trained-Roofline Latency Model} + +Step-time error dominates end-to-end prediction error. +A simulator that misestimates batch step durations propagates that error into +every downstream quantity it computes: queue wait times accumulate over step +boundaries, so a 10\% step-time overestimate inflates TTFT, TPOT, and tail-latency +predictions by the same factor. +BLIS offers four latency-model backends, from a zero-data analytical bound to a +data-efficient learned model, all behind a uniform four-method interface +(step time, queueing time, post-decode overhead, per-token processing time). + +% ---- Figure B: Roofline to Trained-Roofline ---- +\begin{figure}[t] +\centering +\begin{tikzpicture}[scale=0.92] +%% ---- (a) Roofline chart ---- +\begin{scope} + \draw[->] (0,0) -- (4.6,0) + node[right, font=\scriptsize] {Arith.\ intensity (FLOP/B)}; + \draw[->] (0,0) -- (0,3.4) + node[above, font=\scriptsize] {Perf.\ (FLOP/s)}; + + % Memory-bandwidth slope + \draw[thick, blue!60!black] (0,0) -- (2.2, 2.75); + \node[font=\tiny, blue!60!black, rotate=51] at (0.75,1.05) {peak BW}; + + % Peak-compute ceiling + \draw[thick, blue!60!black] (2.2, 2.75) -- (4.4, 2.75); + \node[font=\tiny, blue!60!black] at (3.5, 2.95) {peak FLOP/s}; + + % FP16 ridge + \draw[densely dashed, gray!70] (2.2,0) -- (2.2,2.75); + \node[font=\tiny, gray!80, below] at (2.2,-0.05) {\tiny FP16}; + + % INT4 ridge (shifted left — quantized weight BW rises) + \draw[densely dashed, orange!70!black] (1.2,0) -- (1.2,1.50); + \node[font=\tiny, orange!70!black, below] at (1.2,-0.05) {\tiny INT4}; + + % Decode operating point (memory-bound) + \fill[red!65!black] (0.65, 0.81) circle (2.5pt); + \node[font=\tiny, above right] at (0.65, 0.81) {decode}; + + % Prefill operating point (compute-bound) + \fill[green!55!black] (3.40, 2.75) circle (2.5pt); + \node[font=\tiny, below right] at (3.40, 2.75) {prefill}; + + \node[font=\small\bfseries] at (2.2, 3.25) {(a) Analytical roofline}; +\end{scope} + +%% ---- (b) Trained-roofline term decomposition ---- +\begin{scope}[xshift=5.4cm] + \node[font=\small\bfseries] at (1.9, 3.25) {(b) Trained-roofline: term breakdown}; + + % Axis + \draw[->] (0,0) -- (0,3.0) node[above, font=\scriptsize] {}; + \draw[->] (0,0) -- (3.9,0) node[right, font=\scriptsize] {}; + + % Three batch columns: prefill-heavy, mixed, decode-heavy + \foreach \x/\lbl in {0.5/prefill-heavy, 1.9/mixed, 3.3/decode-heavy}{ + \node[font=\tiny, below] at (\x, -0.05) {\lbl}; + } + + % Stacked bars: colours for each term + % Prefill-heavy: T_pf_comp dominates + \fill[blue!55] (0.2, 0.00) rectangle (0.8, 1.70); % pf compute + \fill[green!45] (0.2, 1.70) rectangle (0.8, 2.00); % weight + \fill[gray!40] (0.2, 2.00) rectangle (0.8, 2.25); % fixed + + % Mixed + \fill[blue!55] (1.6, 0.00) rectangle (2.2, 0.90); % pf compute + \fill[red!45] (1.6, 0.90) rectangle (2.2, 1.65); % dc kv + \fill[green!45] (1.6, 1.65) rectangle (2.2, 2.00); % weight + \fill[gray!40] (1.6, 2.00) rectangle (2.2, 2.25); % fixed + + % Decode-heavy: T_dc_kv dominates + \fill[red!45] (3.0, 0.00) rectangle (3.6, 1.80); % dc kv + \fill[green!45] (3.0, 1.80) rectangle (3.6, 2.10); % weight + \fill[gray!40] (3.0, 2.10) rectangle (3.6, 2.35); % fixed + + % Legend + \fill[blue!55] (0.0, 2.65) rectangle (0.25, 2.85); + \node[font=\tiny, right] at (0.28, 2.75) {$\beta_1 T^\text{pf}$ (prefill)}; + \fill[red!45] (0.0, 2.40) rectangle (0.25, 2.60); + \node[font=\tiny, right] at (0.28, 2.50) {$\beta_2 T^\text{dc}$ (decode)}; + \fill[green!45] (2.0, 2.65) rectangle (2.25,2.85); + \node[font=\tiny, right] at (2.28, 2.75) {$\beta_3 T_\text{weight}$}; + \fill[gray!40] (2.0, 2.40) rectangle (2.25,2.60); + \node[font=\tiny, right] at (2.28, 2.50) {$\beta_5 L + \beta_6 B + \beta_7$}; +\end{scope} +\end{tikzpicture} +\caption{\textbf{Latency-model backends.} + (a)~The analytical roofline places prefill in the compute-bound regime + and decode in the memory-bandwidth-bound regime. + Quantization shifts the roofline ridge left (INT4 vs.\ FP16), reducing KV-cache + pressure and moving the decode point closer to the ridge. + (b)~The trained-roofline decomposes step time into four physically-motivated terms + whose relative contributions shift with batch composition.} +\label{fig:latency-model} +\end{figure} + +\subsubsection{The Analytical Roofline Model} +\label{subsubsec:roofline} + +The roofline model bounds step time by the slower of compute throughput and memory +bandwidth: +\begin{equation} + T_\text{step} = \max\!\left(\frac{F}{\hat{P}_\text{flops}},\;\frac{M}{\hat{P}_\text{bw}}\right), + \label{eq:roofline} +\end{equation} +where $\hat{P}_\text{flops} = P_\text{flops} \times \mathrm{MFU}$ and +$\hat{P}_\text{bw} = P_\text{bw}$ are the hardware's effective FLOP/s and byte/s +throughput. +BLIS computes $F$ from first principles for each batch: QKV projection, attention +score computation, output projection, and MLP (two-matrix SwiGLU convention), +with each term differentiated by whether tokens are in the prefill or decode phase. +The memory term $M$ covers three sources: model weight bytes loaded once per step, +KV-cache blocks written during prefill, and KV-cache blocks read and written during +decode. + +The model uses \emph{phase-specific} MFU correction factors (\code{MfuPrefill}, +\code{MfuDecode}) because prefill and decode operate at fundamentally different +arithmetic intensities (Figure~\ref{fig:latency-model}a). +For quantized models, BLIS uses \code{EffectiveWeightBytesPerParam()}---auto-detected +from HuggingFace quantization config, model-name conventions (e.g., \code{w4a16} +$\rightarrow$ 0.5 bytes/param), or the \code{torch\_dtype} fallback---for the weight +term while keeping the full-precision byte count for KV-cache and activation memory. +This correctly shifts the roofline ridge for quantized models without requiring +separate profiling. + +\subsubsection{The Trained-Roofline Model} \label{subsubsec:trained-roofline} + +The analytical roofline captures the right \emph{structure} but the wrong +\emph{scale}: hardware non-idealities---NCCL synchronization, scheduler overhead, +memory system effects---shift the contribution of each roofline term by a +model-specific factor that no analytical formula predicts. +The trained-roofline model promotes each term to a learnable basis function with a +fitted scalar correction: +\begin{equation} +\boxed{ + T_\text{step} = + \beta_1\max(T^\text{pf}_\text{comp},\,T^\text{pf}_\text{kv}) + + \beta_2\max(T^\text{dc}_\text{comp},\,T^\text{dc}_\text{kv}) + + \beta_3\,T_\text{weight} + + \beta_4\,T_\text{tp} + + \beta_5\,L + + \beta_6\,B + + \beta_7. +} + \label{eq:trained-roofline} +\end{equation} +Each term has a precise physical interpretation. +$T^\text{pf}_\text{comp}$ and $T^\text{pf}_\text{kv}$ are the prefill +compute time and KV-write bandwidth time; their max captures the phase-level +bottleneck. +$T^\text{dc}_\text{comp}$ and $T^\text{dc}_\text{kv}$ are the analogous +decode terms. +$T_\text{weight}$ is the weight-loading time, amortized over the batch. +$T_\text{tp} \approx 0$ on H100 NVLink (TP communication is absorbed into the +per-layer overhead $\beta_5 L$). +$B$ is the batch size. + +BLIS fits these seven coefficients from 13 vLLM experiments spanning four +architectures (Llama-3, Qwen3, DeepSeek-MoE, Granite): +\[ + \bm{\beta} = [0.773,\;1.127,\;1.056,\;0.0,\;43.5\,\mu\text{s},\;48.8\,\mu\text{s},\;0.0]. +\] +That $\beta_1$--$\beta_3$ are near 1.0 is not coincidental: the roofline basis +captures the right physics, and the corrections are small rather than compensatory. +Figure~\ref{fig:latency-model}b shows how term contributions shift across +prefill-heavy, mixed, and decode-heavy batches. +For MoE models, the weight term uses $n_\text{eff} = \min(N,\max(k,\,B{\cdot}k))$ +expert loads per step, modelling the routing fan-out without additional coefficients. + \subsubsection{End-to-End Request Latency Decomposition} \label{subsubsec:e2e-decomp} + +End-to-end latency decomposes as +\begin{equation} + \mathrm{E2E}(r) \;=\; + \underbrace{\alpha_0}_{\text{API overhead}} + + \sum_{s \,\in\, \mathrm{steps}(r)} T_s + + \underbrace{\alpha_1}_{\text{post-decode}} + + \underbrace{\alpha_2 \cdot y_r}_{\text{detokenization}}, + \label{eq:e2e} +\end{equation} +where $y_r$ is the number of output tokens. +TTFT is the time from arrival to the first decode step; TPOT is the mean +inter-token interval during decode. +For the trained-roofline backend, $\alpha_0$ is a constant API overhead (the +prefill cost is already captured in step-time terms), $\alpha_1$ is a fixed +post-decode cost, and $\alpha_2$ is the per-token detokenization latency. +For the blackbox and cross-model backends, $\alpha_0$ grows linearly with +input length, modelling the end-to-end queueing and KV-prefill overhead observed +in trace data. + \subsubsection{Training Pipeline} \label{subsubsec:training} + +\begin{sloppypar} +BLIS fits the trained-roofline coefficients in three sequential phases using +non-negative least squares (NNLS). +Phase~1 isolates $\beta_3$ from batches where $T_\text{weight}$ dominates +(zero prefill and decode tokens). +Phase~2 fixes $\beta_3$ and fits $\beta_1$, $\beta_2$ from mixed batches that +vary prefill-to-decode ratio. +Phase~3 fits $\beta_5$, $\beta_6$, $\beta_7$ from the step-time residual. +This three-phase structure exploits physical separability: earlier +coefficients are not disturbed by later phases. +Across 137{,}000 requests in 13 experiments, the trained-roofline achieves +\textbf{7\% MAPE} (mean absolute percentage error) in step-time prediction. +The physical grounding is also what makes the model data-efficient: 13 experiments +across four architectures suffice because the basis functions are right. +A purely data-driven polynomial fit requires far more data to generalize across +held-out architectures, since its basis functions must be discovered rather than +derived from physics. +\end{sloppypar} + +% -------------------------------------------------------------------------- \subsection{Pluggable Control-Plane Policies} \label{subsec:design-policies} + +The control plane of an LLM serving system is deterministic given its inputs: +routing decisions, scheduling orders, and admission outcomes depend only on +observable queue state, not on GPU execution. +BLIS replicates each control-plane mechanism behind a single-method interface, +making every policy independently swappable and testable. +Faithful replication matters: a routing policy that silently ignores prefix +affinity causes cache misses that inflate step times, conflating two distinct +sources of error that must be diagnosed and corrected independently. + \subsubsection{Admission Control} \label{subsubsec:admission} + +Admission fires at the cluster level before any instance sees a request. +The \code{AdmissionPolicy} interface takes a request and the current cluster +state (\code{RouterState}) and returns an admit/reject decision with a reason. +BLIS ships two implementations: \textsc{AlwaysAdmit} (the default, +suitable for overload-free regimes) and \textsc{TokenBucket} (rate-limiting +with configurable capacity and refill rate, cost proportional to input-token +count). +Because rejection happens before routing, dropped requests never consume +instance queue state, enabling clean overload experiments where the admission +threshold is the variable under study. + \subsubsection{Request Routing} \label{subsubsec:routing} + +\begin{sloppypar} +BLIS supports four routing policies: \textsc{RoundRobin}, +\textsc{LeastLoaded}, \textsc{WeightedScoring}, and a pathological +\textsc{AlwaysBusiest} baseline. +The real contribution is the \emph{composable scorer framework} inside +\textsc{WeightedScoring} (Figure~\ref{fig:scorer}). +Pure load-balancing ignores prefix locality and incurs cold-cache misses on +every step; pure affinity routing ignores load imbalance and overloads +hot-prefix instances. +The composable scorer resolves this tension: each scorer encodes one signal, +and operators control the tradeoff with a weight vector. +Each scorer maps every candidate instance to a score in $[0,1]$. +The composite score is $\sum_i w_i \cdot s_i$ with normalised weights; +the instance with the highest composite score is selected, with random +tie-breaking for reproducibility. +\end{sloppypar} + +BLIS provides four built-in scorers. +\emph{prefix-affinity} scores by prefix-match ratio, using a router-side LRU +index that tracks block hashes per instance---the router maintains a shadow of +each instance's KV-cache topology without reading engine internals. +\emph{queue-depth} uses min-max normalised effective load. +\emph{kv-utilization} scores as $1 - \mathrm{KVUtil}$. +\emph{load-balance} uses an inverse-load transform that preserves absolute +load differences. +The default profile \code{prefix-affinity:3,\,queue-depth:2,\,kv-utilization:2} +matches the llm-d endpoint picker~\cite{llmd2024}, enabling direct sim-to-real +comparison without re-tuning weights. + +% ---- Figure C: Composable Scorer Framework ---- +\begin{figure}[t] +\centering +\begin{tikzpicture}[ + inst/.style={draw, rounded corners=3pt, fill=gray!8, + minimum width=1.55cm, minimum height=0.45cm, + align=center, font=\footnotesize}, + arr/.style={->, >=Stealth, semithick}, + lbl/.style={font=\tiny}, +] +% Three instances across the top +\foreach \i/\x in {1/0.0, 2/2.2, 3/4.4}{ + \node[inst] (I\i) at (\x, 2.8) {Instance$_\i$}; +} + +% Stacked score bars for each instance +% Colours: blue=prefix-affinity, orange=queue-depth, teal=kv-util +\foreach \x/\pa/\qd/\kv in {0.0/1.40/0.70/0.45, 2.2/0.30/1.05/0.85, 4.4/0.85/0.60/0.60}{ + \pgfmathsetmacro{\base}{0.0} + \fill[blue!50] (\x-0.55, \base) rectangle (\x+0.55, \base+\pa); + \fill[orange!60] (\x-0.55, \base+\pa) rectangle (\x+0.55, \base+\pa+\qd); + \fill[teal!50] (\x-0.55, \base+\pa+\qd) rectangle (\x+0.55, \base+\pa+\qd+\kv); + \draw (\x-0.55, 0) rectangle (\x+0.55, \base+\pa+\qd+\kv); +} + +% Score axis label +\draw[->] (-0.85,0) -- (-0.85, 2.75) node[above, font=\tiny] {score}; +\foreach \y in {0,0.5,1.0,1.5,2.0,2.5}{ + \draw[gray!40] (-0.90,\y) -- (5.0,\y); +} + +% Argmax marker +\draw[arr, red!70!black, very thick] (0.0, 2.60) -- (0.0, 2.82); +\node[font=\tiny, red!70!black, above] at (0.0, 2.82) {\textbf{selected}}; + +% Legend — weights shown next to each scorer name +\fill[blue!50] (5.4, 2.4) rectangle (5.7, 2.6); +\node[font=\tiny, right] at (5.72, 2.50) {prefix-affinity\ ($w{=}3$)}; +\fill[orange!60] (5.4, 2.1) rectangle (5.7, 2.3); +\node[font=\tiny, right] at (5.72, 2.20) {queue-depth\ ($w{=}2$)}; +\fill[teal!50] (5.4, 1.8) rectangle (5.7, 2.0); +\node[font=\tiny, right] at (5.72, 1.90) {kv-utilization\ ($w{=}2$)}; + +% Composite formula note +\node[font=\tiny, gray, align=center] at (2.2, -0.45) + {composite $= \textstyle\sum_i w_i \cdot s_i$,\enspace $s_i \in [0,1]$}; +\end{tikzpicture} +\caption{\textbf{Composable scorer framework.} + Three scorers evaluate all candidate instances independently on $[0,1]$. + Weighted combination selects Instance$_1$ here because its high prefix-affinity + score (large blue bar) outweighs lower load-balance scores. + The default weights (pa:3, qd:2, kv:2) match the llm-d endpoint picker, + enabling direct sim-to-real comparison. + New scorers plug in by implementing a single-method interface.} +\label{fig:scorer} +\end{figure} + \subsubsection{Instance Scheduling and Batch Formation} \label{subsubsec:scheduling} + +Inside each instance, \code{InstanceScheduler} re-orders the wait queue before +every step. +BLIS provides three meaningful scheduling policies: \textsc{FCFS} (default), +\textsc{PriorityFCFS} (sort by SLO class then arrival time), and +\textsc{SJF} (sort by input-token count). +Priority values come from a separate \code{PriorityPolicy}; the +\textsc{SLOBased} variant uses $\text{base} + w_\text{age} \times \text{wait}$, +where the aging term prevents starvation across SLO classes. + +Batch formation replicates vLLM's continuous batching with chunked prefill and +preemption. +Phase~1 processes continuing requests: long prompts are chunked to +\code{LongPrefillTokenThreshold} tokens per step, and decode tokens fill the +remainder of the \code{MaxScheduledTokens} budget. +Phase~2 admits new requests from the wait queue, allocating KV blocks and +looking up prefix-cached blocks. +If KV allocation fails during Phase~1, the formation preempts from the batch +tail---preserving earlier blocks that more requests may share---until memory +is available or the batch reaches size one. +This two-phase algorithm is the same structure used in production vLLM, ensuring +that BLIS batch dynamics match real deployments. + \subsubsection{KV Cache and Memory Tiering} \label{subsubsec:kvcache} + +BLIS models KV-cache pressure at block granularity. +The single-tier GPU cache maintains an O(1) LRU free list and a hash-to-block +index for prefix reuse. +Block hashes use a hierarchical SHA-256 chain: each block's hash includes the +previous block's hash, so the prefix-match lookup costs $O(K \cdot B_\text{sz})$ +rather than $O(K^2 \cdot B_\text{sz})$, where $K$ is the cached-prefix length +in blocks and $B_\text{sz}$ is the block size. +A failed allocation rolls back atomically---restoring the free list, hash index, +and block states---so the batch formation can retry or preempt without leaving +the cache in an inconsistent state. + +The tiered GPU\,+\,CPU backend (\code{TieredKVCache}) composes the GPU tier with +a CPU LRU mirror (Figure~\ref{fig:tiered-kv}). +After each batch step, \code{MirrorToCPU} copies every completed full block's +hash and token data to the CPU---lazy mirroring rather than eviction-triggered +transfer. +When GPU allocation fails on a cache miss, \code{reloadPrefixFromCPU} walks the +prefix hash chain, pops a GPU free block for each CPU hit, and accumulates a PCIe +transfer stall of $t_\text{base} + \lceil B_\text{sz} / \text{BW}_\text{PCIe} \rceil$ +per reloaded block. +The stall is injected into the next step via \code{ConsumePendingTransferLatency}, +modelling the real system's PCIe serialisation without cycle-accurate simulation. +KV capacity is computed analytically: total GPU memory minus model weights +(three-matrix SwiGLU, consistent with the trained-roofline convention), +activation memory, and non-torch overhead, divided by bytes-per-block at the +configured precision. + +% ---- Figure D: Tiered KV Cache ---- +\begin{figure}[t] +\centering +\begin{tikzpicture}[ + mem/.style={draw, rounded corners=3pt, minimum height=0.80cm, + minimum width=5.6cm, align=center, font=\small}, + blk/.style={draw, minimum width=0.62cm, minimum height=0.62cm, + font=\tiny, align=center}, + arr/.style={->, >=Stealth, semithick}, +] +%% GPU tier — raised to y=2.7 to open up the inter-tier gap +\node[mem, fill=blue!8] (gpu) at (0, 2.7) {}; +\node[font=\small\bfseries, blue!60!black, left=0.1cm] at (gpu.west) {GPU VRAM}; + +\foreach \i/\col/\lab in { + 1/blue!25/{A}, 2/blue!25/{B}, 3/orange!30/{C}, + 4/green!25/{D}, 5/gray!20/{free}, 6/gray!20/{free}}{ + \node[blk, fill=\col] at (-2.1 + \i*0.75, 2.7) {\lab}; +} + +%% CPU tier at y=0.5 +\node[mem, fill=orange!6] (cpu) at (0, 0.5) {}; +\node[font=\small\bfseries, orange!70!black, left=0.1cm] at (cpu.west) {CPU DRAM}; + +\foreach \i/\col/\lab in { + 1/blue!15/{A}, 2/blue!15/{B}, 3/green!15/{D}, + 4/gray!15/{E}, 5/gray!15/{F}, 6/gray!15/{G}}{ + \node[blk, fill=\col] at (-2.1 + \i*0.75, 0.5) {\lab}; +} + +%% Inter-tier gap: GPU bottom = 2.30, CPU top = 0.90 +%% Arrows to the right of all named blocks (x > 2.71) +\draw[arr, blue!55!black] (3.4, 2.30) -- (3.4, 0.90) + node[midway, right, font=\tiny, blue!55!black] {lazy mirror}; + +\draw[arr, dashed, orange!65!black] (3.0, 0.90) -- (3.0, 2.30) + node[midway, right, font=\tiny, orange!65!black] {reload on miss}; + +%% PCIe stall — below CPU box (CPU box bottom = 0.10) +\node[draw=orange!50, rounded corners=2pt, fill=orange!5, + font=\tiny, align=center, inner sep=2pt] + at (3.2, -0.28) {PCIe: $t_\text{base}+\lceil B_\text{sz}/\mathrm{BW}\rceil$\\per block}; +\draw[->, very thin, dashed, orange!45!black] (3.2, -0.08) -- (3.2, 0.87); +\end{tikzpicture} +\caption{\textbf{Tiered KV cache.} + After each step, BLIS lazily mirrors newly completed blocks from GPU to CPU DRAM. + On a GPU miss, matching CPU blocks reload to GPU and accumulate a PCIe transfer + stall injected into the next step's execution time. + Orange block~C is on GPU but not yet mirrored; green block~D has been mirrored to both tiers.} +\label{fig:tiered-kv} +\end{figure} + +% -------------------------------------------------------------------------- \subsection{Composable Workload Models} \label{subsec:design-workloads} + +Workload fidelity matters as much as execution-physics accuracy: a simulator fed +a naive Poisson stream with fixed token lengths will predict a deployment that +no real workload ever produces. +ServeGen~\cite{xiang2025servegen} shows that per-client compositional generation +avoids the 50\% under-provisioning errors of aggregate workload generation; +BLIS's workload model follows the same per-client structure. + +A \code{WorkloadSpec} organises parameters at three levels. +At the \emph{cluster level}: aggregate request rate, simulation horizon, and RNG seed. +At the \emph{client level}: each client specifies an arrival process (four +families, from Poisson to heavy-tailed Weibull), token-length distributions +(five families, including an empirical type that replays a ServeGen trace with +zero re-parameterisation~\cite{xiang2025servegen}), SLO class, prefix group, +and session mode (open-loop or closed-loop with configurable think time). +At the \emph{cohort level}: population dynamics---diurnal scaling, traffic spikes, +client drain---produce the bursty, non-stationary arrival patterns observed in +production. +For closed-loop sessions the generator emits only round-0 requests at +initialisation; subsequent rounds are generated at runtime as completions arrive, +preserving the causal dependency between turns. + +% -------------------------------------------------------------------------- \subsection{Metrics and Fitness Evaluation} \label{subsec:design-metrics} +BLIS measures three per-request latency quantities operators care about: +\emph{time to first token} (TTFT), \emph{time per output token} (TPOT), and +\emph{end-to-end latency} (E2E, Equation~\ref{eq:e2e}). +For each, it reports the full empirical distribution and its percentiles (p50, +p95, p99), along with per-SLO-class breakdowns, aggregate throughput (tokens/s +and requests/s), and KV-cache hit rate. + +For algorithm evolution (\S\ref{sec:eval-evolution}), BLIS exposes a scalar +fitness function: +\begin{equation} + \mathcal{F}(\pi,\,\mathcal{W}) \;=\; -\,\mathrm{p95}(\mathrm{E2E}) + \quad\text{s.t.}\quad + \mathrm{p99}(\mathrm{TTFT}) \;\leq\; T_\mathrm{slo}, + \label{eq:fitness} +\end{equation} +The objective minimises p95(E2E)---the experience of the slowest one-in-twenty +users---while the constraint prevents the optimizer from sacrificing first-token +responsiveness for throughput. +$\pi$ is a policy configuration (routing weights, scheduling policy, +admission threshold) and $\mathcal{W}$ is a workload scenario. +A single simulation evaluates $\mathcal{F}$ in under two seconds on a laptop CPU, +making 1{,}000-candidate evolutionary search feasible in under 30 minutes---orders +of magnitude faster than real-hardware evaluation, which requires minutes of GPU +time per configuration. + % ============================================================ % SECTION 5: SIM-TO-REAL METHODOLOGY % ============================================================ diff --git a/paper/howtowrite.txt b/paper/howtowrite.txt new file mode 100644 index 0000000..03ac0d6 --- /dev/null +++ b/paper/howtowrite.txt @@ -0,0 +1,501 @@ +How (and How Not) to Write a Good Systems Paper +[This article first appeared in ACM SIGOPS Operating Systems Review, Vol. 17, No. 3 (July, 1983), pages 35-40. Every effort has been made to keep the text in this document identical to that of the original article. The text in this file was scanned using OCR technology and has been carefully proofread, but some scanning errors may remain. This document is being made available with the permission of the authors.] + +An Evaluation of the Ninth SOSP Submissions +-or- +How (and How Not) to Write a Good Systems Paper +Roy Levin and David D. Redell, +Ninth SOSP Program Committee Co-chairmen +Introduction +On March 21, 1983, the program committee for the 9th Symposium on Operating System Principles, having read the eighty-three papers submitted, selected sixteen for presentation at the symposium. This acceptance ratio of about one in five approximates those of past SOSPs, although the number of submissions was somewhat lower than in recent years. Several members of the program committee found it surprisingly easy to separate the good papers from the bad ones; indeed, the ten committee members quickly agreed on the disposition of over 80% of the papers. As the acceptance ratio indicates, most of these were rejections. + +After the committee had completed its selectio n process, several members expressed disappointment in the overall quality of the submissions. Many of the rejected papers exhibited similar weaknesses, weaknesses that the committee felt should have been evident to the authors. In the hope of raising the quality of future SOSP submissions, and systems papers generally, the committee decided to describe the criteria used in evaluating the papers it received. This article combines the criteria used by all of the members of the committee, not just the authors. + +To try to avoid sounding preachy or pedagogic, we have cast this presentation in the first and second person and adopted a light, occasionally humorous style. Nevertheless, the intent is serious: to point out the common problems that appear repeatedly in technical papers in a way that will make it easier for future authors to avoid them. As you read this article, then, suppose yourself to be a prospective author for the 10th SOSP or for TOCS. You've done some work you would like to publish, so you sit down to write a paper. What questions should you be asking yourself as you write? These are also the questions that we, the reviewers of your paper, will be asking to determine its suitability for publication. + +Classes of Papers +Your paper will probably fall naturally into one of three categories: + +It presents a real system, either by a global survey of an entire system or by a selective examination of specific themes embodied in the system. +It presents a system that is unimplemented but utilizes ideas or techniques that you feel the technical community should know. +It addresses a topic in the theoretical areas, for example, performance modelling or security verification. +Obviously, a single set of evaluation criteria cannot be applied uniformly across these categories; nevertheless, many criteria apply equally well to all three. As we describe each one below, we will try to emphasize the classes of papers to which it applies. Often it will be evident from context. + +Criteria for Evaluation of Submissions +Original Ideas +Are the ideas in the paper new? There is no point in submitting a paper to a conference or journal concerned with original work unless the paper contains at least one new idea. + +How do you know? You must be familiar with the state of the art and current research in the area covered by your paper in order to know that your work is original. Perhaps the most common failing among the submissions in the first category (real systems) was an absence of new ideas; the systems described were frequently isomorphic to one of a small number of pioneering systems well-documented in the literature. + +Can you state the new idea concisely? If your paper is to advance the state of knowledge, your reader must be able to find the new ideas and understand them. Try writing each idea down in a paragraph that someone generally versed in the relevant area can understand. If you can't, consider the possibility that you don't really understand the idea yourself. When you have the paragraphs, use them in the abstract for the paper. + +What exactly is the problem being solved? Your reader cannot be expected to guess the problem you faced given only a description of the solution. Be specific. Be sure to explain why your problem couldn't be solved just as well by previously published techniques. + +Are the ideas significant enough to justify a paper? Frequently, papers describing real systems contain one or two small enhancements of established techniques. The new idea(s) can be described in a few paragraphs; a twenty-page paper is unnecessary and often obscures the actual innovation. Since construction of a real system is a lot of work, the author of the paper sometimes unconsciously confuses the total effort with the work that is actually new. ("My team worked on this system for two years and we're finally done. Let's tell the world how wonderful it is.") If the innovation is small, a small paper or technical note in a suitable journal is more appropriate than an SOSP submission. + +Is the work described significantly different from existing related work? An obvious extension to a previously published algorithm, technique, or system, does not generally warrant publication. Of course, the label "obvious" must be applied carefully. (Remember the story of Columbus demonstrating how to make an egg stand on end (by gently crushing it): "it's obvious once I've shown you how".) You must show that your work represents a significant departure from the state of the art. If you can't, you should ask yourself why you are writing the paper and why anyone except your mother should want to read it. + +Is all related work referenced, and have you actually read the cited material? You will have difficulty convincing the skeptical reader of the originality of your efforts unless you specifically distinguish it from previously published work. This requires citation. Furthermore, you will find it harder to convince your reader of the superiority of your approach if he has read the cited works and you haven't. + +Are comparisons with previous work clear and explicit? You cannot simply say: "Our approach differs somewhat from that adopted in the BagOfBits system [3]." Be specific: "Our virtual memory management approach uses magnetic media rather than punched paper tape as in the BagOfBits system [3], with the expected improvements in transfer rate and janitorial costs." + +Does the work comprise a significant extension, validation, or repudiation of earlier but unproven ideas? Implementation experiences supporting or contradicting a previously published paper design are extremely valuable and worthy candidates for publication. Designs are cheap, but implementations (particularly those based on unsound designs) are expensive. + +What is the oldest paper you referenced? The newest? Have you referenced similar work at another institution? Have you referenced technical reports, unpublished memoranda, personal communications? The answers to these questions help alert you to blind spots in your knowledge or understanding. Frequently, papers with only venerable references repeat recently published work of which the author is unaware. Papers with only recent references often "rediscover" (through ignorance) old ideas. Papers that cite only unpublished or unrefereed material tend to suffer from narrowness and parochialism. Remember that citations not only acknowledge a debt to others, but also serve as an abbreviation mechanism to spare your reader a complete development from first principles. If the reader needs to acquire some of that development, however, he must be able to convert your citations into source material he can read. Personal communications and internal memoranda fail this test. Technical reports are frequently published in limited quantities, out-of-print, and difficult to obtain. Consequently, such citations as source material should be avoided wherever possible. + +Reality +Does the paper describe something that has actually been implemented? Quite a few of the SOSP submissions proceeded for fifteen pages in the present tense before revealing, in a concluding section (if at all), that the foregoing description was of a hypothetical system for which implementation was just beginning or being contemplated. This is unacceptable. Your reader has a right to know at the outset whether the system under discussion is real or not. +If the system has been implemented, how has it been used, and what has this usage shown about the practical importance of the ideas? Once again, a multiple man-year implementation effort does not of itself justify publication of a paper. If the implemented system contains new ideas, it is important to explain how they worked out in practice. A seemingly good idea that didn't pan out is at least as interesting as one that did. It is important to be specific and precise. "Our weather prediction system is up and running and no one has complained about its occasional inaccurate forecasts" is much less convincing than "everytime we fail to forecast rain, the users hang their wet shirts over the tape drives to dry". In the latter case, at least we know that people are using and depending on the system. + +If the system hasn't been implemented, do the ideas justify publication now? This can be a difficult question for an author to answer dispassionately, yet any reviewer of the paper will make this judgment. It is always tempting to write a design paper describing a new system, then follow it up in a year or two with an "experience" paper. The successful papers of this genre nearly always include initial experience in the closing sections of the design paper. The subsequent experience paper then deals with the lessons learned from longer-term use of the system, frequently in unanticipated ways. Reviewers are very skeptical of design-only papers unless there are new ideas of obviously high quality. + +Lessons +What have you learned from the work? If you didn't learn anything, it is a reasonable bet that your readers won't either, and you've simply wasted their time and a few trees by publishing your paper. +What should the reader learn from the paper? Spell out the lessons clearly. Many people repeat the mistakes of history because they didn't understand the history book. + +How generally applicable are these lessons? Be sure to state clearly the assumptions on which your conclusions rest. Be careful of generalizations based on lack of knowledge or experience. A particularly common problem in "real system" papers is generalization from a single example, e.g., assuming that all file system directories are implemented by storing the directory in a single file and searching it linearly. When stating your conclusions, it helps to state the assumptions again. The reader may not have seen them for fifteen pages and may have forgotten them. You may have also. + +Choices +What were the alternatives considered at various points, and why were the choices made the way they were? A good paper doesn't just describe, it explains. Telling your readers what you did doesn't give them any idea how carefully considered your choices were. You want to save future researchers from following the same blind alleys. You also want to record potentially interesting side-streets you didn't happen to explore. Make sure to state clearly which is which. +Did the choices turn out to be right, and, if so, was it for the reasons that motivated them in the first place? If not, what lessons have you learned from the experience? How often have you found yourself saying "this works, but for the wrong reason"? Such a pronouncement represents wisdom (at least a small amount) that may benefit your reader. Many papers present a rational argument from initial assumptions all the way to the finished result when, in fact, the result was obtained by an entirely different path and the deductive argument fashioned later. This kind of "revisionist history" borders on dishonesty and prevents your readers from understanding how research really works. + +Context +What are the assumptions on which the work is based? The skeptical reader is unlikely to accept your arguments unless their premises are stated. Make sure you get them all; it's easy to overlook implicit assumptions. +Are they realistic? For "unimplemented systems" papers, this amounts to asking whether the assumptions of the design can hope to support a successful implementation. Many paper designs are naive about the real characteristics of components they treat abstractly, e.g., communication networks or humans typing on terminals. For theoretical studies, it must be clear how the assumptions reflect reality, e.g., failure modes in reliability modelling, classes of security threats in security verification, arrival distributions in queuing systems. + +How sensitive is the work to perturbations of these assumptions? If your result is delicately poised on a tall tower of fragile assumptions, it will be less useful to a reader than one that rests on a broader and firmer foundation. + +If a formal model is presented, does it give new information and insights? Simply defining a model for its own sake is not very useful. One deep theorem is worth a thousand definitions. + +Focus +Does the introductory material contain excess baggage not needed for your main development? "Real system" papers are particularly guilty of irrelevant description. If your subject is distributed file systems, the physical characteristics of the connection between computer and communication network are probably not germane. Avoid the temptation to describe all major characteristics of your system at the same level of depth. Concentrate instead on the novel or unusual ones that (presumably) will be the focus of the original technical content of the paper. +Do you include just enough material from previously published works to enable your reader to follow your thread of argument? Do not assume that the reader has read every referenced paper within the last week and has them at his fingertips for instant reference. If you want your reader to get past page three, avoid introductory sentences of the form "We adopt the definition of transactions from Brown [4], layering it onto files as described by Green [7, 18], with the notions of record and database introduced by Black [10] and White [12] and later modified by Gray [6]". On the other hand, don't burden your reader unnecessarily with lengthy extracts or paraphrases from cited works. + +Presentation +Are the ideas organized and presented in a clear and logical way? +Are terms defined before they are used? + +Are forward references kept to a minimum? Readers get annoyed when they repeatedly encounter statements like "Each file consists of a sequence of items, which will be described in detail in a later section". The reader has to remember the technical term "item", but the term has no semantics yet. It's all right to ask him to do this once or twice, but only when absolutely necessary. Even if you can't afford the digression to explain "item" at this point, give the reader enough information to attach some meaning to the term: "Each file consists of a sequence of items, variable-sized, self-identifying bit sequences whose detailed interpretation will be discussed below under 'Multi-media Files'." Your reader may not yet understand your concept of files completely, but at least he has some glimpse of the direction in which you are leading him. + +Have alternate organizations been considered? Theoretical papers, particularly of a mathematical character, are generally easier to organize than papers describing systems. The expected sequence of definition, lemma, theorem, example, corollary works well for deductive argument, but poorly for description. In "real system" papers, much depends on the intent: global survey or selective treatment. Frequently, difficulties in organization result from the author's unwillingness to commit to either approach. Decide whether you are surveying your system or focusing on a specific aspect and structure the paper accordingly. + +Was an abstract written first? Does it communicate the important ideas of the paper? Abstracts in papers describing systems are sorely abused. The abstract is more often a prose table of contents than a precis of the technical content of the paper. It tends to come out something like this: "A system based on Keysworth's conceptualization of user interaction [4] has been designed and implemented. Some preliminary results are presented and directions for future work considered." No reader skimming a journal is likely to keep reading after that. Avoid the passive voice (despite tradition) and include a simple statement of assumptions and results. "We designed and implemented a user interface following the ideas of Keysworth and discovered that converting the space bar to a toe pedal increases typing speed by 15%. However, accuracy decreased dramatically when we piped rock music instead of Muzak (tm) into the office." Leave discussion and argument for the paper. It helps to write the abstract before the paper (despite tradition) and even the outline, since it focusses your attention on the main ideas you wants to convey. + +Is the paper finished? Reviewers can often help you to improve your paper, but they can't write it for you. Moreover, they can't be expected to interpolate in sections marked "to be included in the final draft". In a mathematical paper, a reviewer regards the statement of a theorem without proof with suspicion, and, if the theorem is intended to culminate prior development, with intolerance. Similarly, in a paper describing a system, a reviewer cannot tolerate the omission of important explanation or justification. Omitting sections with a promise to fill them in later is generally unacceptable. + +Writing Style +Is the writing clear and concise? +Are words spelled and used correctly? + +Are the sentences complete and grammatically correct? + +Are ambiguity, slang, and cuteness avoided? + +If you don't have sufficient concern for your material to correct errors in grammar, spelling, and usage before submitting it for publication, why should you expect a reviewer to read the paper carefully? Some reviewers feel that this kind of carelessness is unlikely to be confined to the presentation, and will reject the paper at the first inkling of technical incoherence. Remember that you are asking a favor of your reviewers: "Please let me convince you that I have done interesting, publishable work." A reviewer is more favorably disposed toward you if he receives a clean, clear, carefully corrected manuscript than if it arrives on odd-sized paper after ten trips through a photocopier and looking like it was composed by a grade-school dropout. Even if you aren't particularly concerned with precise exposition, there is certain to be someone in your organization who is. Give your manuscript to this conscientious soul and heed the resulting suggestions. + +Summary +These thirty-odd questions can help you write a better technical paper. Consult them often as you organize your presentation, write your first draft, and refine your manuscript into its final form. Some of these questions address specific problems in "systems" papers; others apply to technical papers in general. Writing a good paper is hard work, but you will be rewarded by a broader distribution and greater understanding of your ideas within the community of journal and proceedings readers. + +*** + +Tips about writing systems papers +by Lin Zhong, May 2015 + +If I can give one tip, that would be: respect your readers. + +Put yourself into your readers' shoes +It is important to visualize your readers and how they read your paper. You hope they would enjoy your paper like you enjoyed the Harry Potter books; you hope they would study your work like you studied your Physics 101 textbook; at least you hope they would read your paper like how you read a classic paper recently. Such hopes are completely false, because nobody reads technical papers for recreation, you are obviously not writing a textbook, and more importantly, your paper is not yet published, let alone being a classic. + +So how does a reader read your paper? Or more precisely, how does a reviewer read your paper? They try to finish the job as fast as possible. Peer-review is a service that does not pay. Reviewers try to understand the work as fast as possible so that they can do their job: write a review and decide if they want to champion for reject, accept or they don't really care. Often they have a dozen or so papers to review and likely are already behind the deadline. So you can imagine their most likely mood at this point: impatient, perhaps irritable. Well if you are really really lucky, one of your reviewers is reading your paper one month before the deadline on the beach with a glass of sparkling wine. But I can tell you that rarely happens. + +Importantly, top publication venues usually have a very low acceptance rate, accepting one out of five submissions or less. That means the majority of the papers read by a reviewer are likely to be rejected. Therefore, it is not unusual a reviewer starts reading each paper with a subconscious bias looking for reasons to reject it. So much so that many TPC chairs these days try very hard to guide the reviewers to read submissions with a positive mindset and to focus on identifying values, instead of flaws. Until they are successful, you can safely assume your reviewers are hostile in that they are looking for reasons to reject your paper. Your job is to win them back. + +In the rest of this article, readers include reviewers, the very first outside readers of your paper. + +General techniques +Once you decide to make readers' life easier, you would immediately realize there are numerous ways, each of which, of course, requires you to spend more time with your draft. Below I summarize a few useful general techniques, roughly categorized as presentation and logic. + +Presentation +Be concise. Every word slows the reader a little. See if you can say the same thing with fewer words. Remove a word, sentence, paragraph, figure, section unless they are absolutely necessary. The same principle applies to conceptual units: remove a definition, system component, design detail unless they are absolutely necessary. Resist the urge to tell readers everything. Instead, tell them enough for them to appreciate your points. +Write top-down. Always put your point upfront because readers are impatient and skimming. This rule applies to structures at all levels from paragraph to section. A point is what you want the readers to remember, not a summary of the content but the conclusion supported by it. For instance, the point of the evaluation section is ``Experimental evidence demonstrates our solution advances the state of the art'', not ``We report extensive experimental evaluation that compares our solution against the state of the art.'' +Use active voice. You don't want readers to guess who did something or contributed an opinion. You don't want them to guess which component of your system is behind an action. +Use structure. Your readers, Homo sapiens, have 5 to 9 entries in their short-term memory. Structure helps them abstract/chunk materials in order to use these entries efficiently. It also makes relationships between structural components obviously. A very long paragraph, a page of paragraph after paragraph, and a section of a large number (7 or more) of subsections are signs that structure may be necessary. The lack of structure often results from the lack of focus or abstraction. Focus requires one to ignore unimportant things; abstraction requires one to identify unifying themes, crosscutting principles, and high-level relations. +Use simple English. Many graduate students incorrectly believe a good paper should read like the essays they studied for GRE. Remember the GRE essays were selected to evaluate your reading capability. You absolutely don't want your own readers to feel that you are evaluating their reading capability. Avoid long sentences; avoid complicated sentence structures; avoid arcane terms/idioms; avoid big, showy words. You are welcome to try them after you have got some grey hairs. +Use graphs effectively. Most readers pay attention to graphs. Make sure each graph deliver an important point. Make sure it can do so by itself (so make good use of its caption). Do refer to the graph in the text so that it assists understanding. +Logic +Don't surprise. Set up the right expectation. Don't promise early on, e.g., in abstract and introduction, more than you deliver later, e.g., design, implementation and evaluation. A very common reviewer comment is ``I started reading this paper with excitement but the more I read the more I was disappointed." +Don't handwave. Quantify a claim. For example, when you say ``our solution significantly outperforms the state of the art'', don't stop there but quantify ``significantly" with ``improving the data rate by three times''. For another example, when you say ``our design requires very small changes to the Linux kernel", quantify it with ``with only 3 added code line at a single location''. Cite an authoritative source for a fact that is not yet widely known but key to your point. +Don't haunt readers. As people read, they will have questions. Some questions may trouble them if not answered soon. I call such questions the haunting questions. You must anticipate such questions and answer them before a reader starts to wonder. If you can't answer them immediately, do acknowledge their existence and assure the reader you will answer them later with a forward reference. +Be consistent. Consistency in naming and style makes your writing easier to understand. Call the same thing the same name. If one figure shows the software stack vertically and another horizontally, you make readers' job more difficult. When figures of similar plots are grouped together, e.g., speed vs. number of processors for several benchmarks, make their axes consistent in terms of unit and range. When it is impossible, e.g., one benchmark has higher speed by orders of magnitude than others, highlight that inconsistency, e.g., pointing it out in the caption or using a different font size for the inconsistent axis. +Specialized techniques +Now I describe tips for important structural components of system papers. +Title +A good title captures both the problem addressed by the paper and the novel solution contributed. To avoid a title too long, it often suffices to capture the problem only. + +Abstract and Introduction +Many readers only read the abstract and introduction. Many a reviewer makes up their mind after reading them. The introduction is so important that even the most hands-off advisors will read, revise, and sometimes rewrite it. I suspect that's a major source of surprises and inconsistencies. The introduction should summarize the entire paper crisply and bring out major points you want readers to walk away with. + +Motivation. There is a delicate balance one must achieve here with intended readers in mind. A problem well-known to your intended readers needs no motivation but a simple statement of its importance, e.g., ``Energy efficiency is a critical design concern for battery-powered mobile devices''. An emerging problem anticipated by you or a small part of the community requires more explanation. However, the motivation should never be more than a small paragraph, simply because readers are impatient: they want to get to your problem statement quick. If you have a rather long motivation, either it contains a lot of obvious, well-known materials or you are solving a trivial problem. +Problem statement. Don't wait too long to state the problem. If the motivation is a small paragraph, the problem statement can be the first sentence of the second paragraph. The problem statement must be obvious, prominent, explicit. For example, ``This paper answers the following question:...", "Our goal is to support concurrent computer vision applications efficiently", and "This paper presents our design and implementation of a system that supports concurrent computer vision applications efficiently." A common mistake is to place the problem statement at the end of a paragraph that elaborates the problem; instead, the statement should be placed at the opening of the paragraph. Remember to write top-down. +Prior work. After the problem statement, you need to concisely explain why prior work is inadequate. There is no need to discuss related papers one by one here. +Key ideas. Without reading the rest of the paper, a reader must appreciate why you are able to solve the problem while prior work falls short. Therefore, you need to explain your key ideas, secret sauce, explicitly and prominently in the introduction. For example, "Our design is based on two novel insights. First,...", "We solve the problem with a clever synthesis of technique A and technique B." +Evaluation results. Summarize how you evaluate your work and the main results. Be quantitative. Be upfront about negative results. +Contribution summary. In order to help the reader, especially the reviewer, provide an itemized summary of your contributions. There should be no more than three or four items. +Background +Unless you are writing for a very narrow community, it is very likely your readers do not have all the background to appreciate your work. So educate them. A nicely written background section is always enjoyable. Indeed, learning something new so quickly is one of the perks of doing reviews. The challenge is: the background section must be self-contained, concise, clear. This often requires a deep understanding of who your intended readers are. For a paper submission, it often helps to take a look at the program committee and imagine which members are likely to review your paper. + +Design +For a systems paper, it is often nice to describe design and implementation separately. The design section often presents ideas that are agnostic to any specific implementation. These include the overall architecture, principles and algorithms. + +A common mistake of the design section is lack of structure, e.g., a section of seven or more subsections each describing a component of the design. As mentioned above, the lack of structure often results from the lack of focus or abstraction. For example of lack of focus, the authors simply describe each system components, regardless of how interesting/novel they are. If the system has a lot of components, this strategy leads to a flat section with many subsections. One naturally solution is to use a subsection to provide an overview of the design and then use subsections to explain only important components that are novel and interesting. Presenting the design by component is often a sign of lack of abstraction. System component is just one possible dimension along which you can present the design. There are other dimensions, e.g., design objectives and principles/invariants followed. Discovering these dimensions require you to think about your design in an abstract way. + +Another common mistake is failure to place a design choice in the context, e.g., simply describing it. There are two types of context that add depth. First, how is your choice related to prior work? Perhaps no one has solved the exactly same problem but it is highly likely others have faced similar challenges in solving a related one. It is important to be aware of how others made the choice and explain how their choice affected your choice. This is also a perfect place to give prior work credit. Second, what alternatives have you considered? It is highly unlikely your choice is straightforward or obvious (if it is, you should not spend a lot of space highlighting it). Share with readers what other choices you have considered and why you passed them in favor of the one ended up in the design. + +Implementation +The section of implementation should not be a boring description. It should present interesting things about your implementation. If there is nothing interesting, the section should be very brief, providing the necessary information for readers to assess the quality of your evaluation, which is presumably based on the implementation. + +What are the interesting things about the implementation? As the implementers, you should know. For example, any nontrivial challenges have you had to overcome? How does the implementation have to deviate from the design due to practical limitations? Any platform-specific optimization? + +It is important to let the readers know how significant the implementation effort is. You can quantify it with the number of lines of code and the number of man-hours. + +Evaluation +Since this article is about writing, instead of research itself, I will focus on writing, assuming you have done all the right things in evaluating your work. + +Retain a neutral tone toward your solution. It is important to remember that the objective of evaluation is NOT to show that your solution works. Instead, it is to show how well it works, importantly when it works and when it does not. +Be explicit upfront about what questions you want to answer. These questions are often related to the contributions you claim. For example, if you claim a novel optimization A that saves a lot of energy, the question to answer via evaluation would be "how much energy does A save? How does the saving depends on operational parameters?" +Acknowledge the limitations of your evaluation. And explain how it may affect the conclusions you draw. When you acknowledge them, readers find you honest, trustworthy; when you don't, they are more likely to consider the evaluation as contrived. +Share negative results. A very common mistake is to only show positive results. Unless your solution always works, it is your responsibility to tell readers when it works and when it does not. In other words, you have to specify the scope under which your solution works. Sharing negative results not only adds depth to your paper but also impresses readers as being honest: too many times I have heard reviewers complaining the evaluation setup appears to be contrived, the benchmarks appear to be cherry-picked. To the contrary of the belief by many young researchers, sharing negative results actually strengthens your paper and earns respect from readers. +Provide necessary information for reproducibility. The key of scientific and engineering research is reproducibility. That is, others should be able to follow your writing and reproduce your results. For software, it is nice to make it openly available, open-source or not. When machine configurations matter, you should consider providing a virtual machine with the same configuration as used in the reported evaluation. Where the physical setup matters, e.g., wireless channels, be specific about the physical parameters, e.g., providing a floor map with node locations precisely marked. +Related work +The related work section is important in many ways. It sets your work in the larger intellectual context and explains how it advances the state of the art. For novice readers, it points out a quality collection of good works that they can follow up. For expert readers, it is the perfect place to check the authors' mastery of the topic area. For the authors of the cited work, it is where their own work is discussed, credited and criticized. A well-written section of related work always wins respect from its readers. Below are a few tips. + +Use structure. Don't discuss related works simply one by one. Discuss them in groups. For each group, start with the common things of the group and then important ones one by one. Of course this requires you to understand how related works are related to each other and to your own work. +Be generous. First, when discussing a related work, don't just focus on how it is different: start with how it is similar. Don't just criticize its shortcomings: start with its strength and value. Second, give benefit of doubt to a related work. Don't interpret it in favor of your work. For example, when a related work did not mention whether its idea works for iOS, you should not interpret that as the idea will not work for iOS. A general way to examine if your discussion is generous is to imagine if you were the author of the discussed related work, would you like the way it is discussed? +Be insightful. Don't be content with superficial similarity and dissimilarity. Only when you point out something non-obvious, you earn readers' respect for your insight. +Recommended books +I ask all my students to buy and read the following two books: + +The elements of style by William Strunk Jr. and E. B. White (any edition). +Style: lessons in clarity and grace by Joseph Willims and Gregory G. Colomb (any edition, I have 10th). +The following book is more specialized for Computer Science but I have not read it as carefully as the above two: +Writing for computer science by Justin Zobel. + +*** + +How to write a technical paper or a research paper +by Michael Ernst +April, 2005 +Last updated: July 1, 2024 +Contents: + +Know your message, and stay on message +Which details to include +Make the organization and results clear +Getting started: overcoming writer's block and procrastination +Brevity +Writing style +Figures +Computer program source code +Naming +Numbers and measurements +Processing data +Related work +Feedback +When to submit your paper for publication +Responding to conference reviews +Rejection +Norman Ramsey's advice +Other resources +This document describes several simple, concrete ways to improve your writing, by avoiding some common mistakes. The end of this document contains more resources for improving your writing. + +Some people believe that writing papers, giving talks, and similar “marketing” activities are not part of research, but an adjunct to it or even an undesirable distraction. This view is inaccurate. The purpose of research is to increase the store of human knowledge, and so even the very best work is useless if you cannot effectively communicate it to the rest of the world. If a paper is poorly written, then readers might conclude you spent as little effort on the research that it describes. + +Equally importantly, writing papers and giving talks will clarify your thinking and thereby improve your research. You may be surprised how difficult it is to clearly communicate your ideas and contributions; doing so will force you to understand them more deeply and enable you to improve them. + +Know your message, and stay on message +The goal of writing a paper is to change people's behavior: for instance, to change the way they think about a research problem or to convince them to use a new approach. Determine your goal (also known as your thesis), and focus the paper around that goal. + +As a general rule, your paper needs to convince the audience of three key points. If any of these is missing or unclear, the paper will not be compelling. + +The problem is important. The problem has a significant impact and consequences. You can buttress your argument by showing that others consider the problem important. +The problem is hard. Explain that obvious techniques and existing approaches do not suffice. Showing what others have tried can be effective here. +You have solved the problem. This is often demonstrated via experiments. Keep in mind how you expect the behavior of readers to change once they appreciate your contributions. +You'll also need to convince readers that your contributions are novel. When expressing this, it is helpful to explain why no one else thought of your approach before (or why, if they thought of it, they would have rejected the approach) , and whether similar insights apply to other problems. +Before you write your paper, you need to understand your audience. Who will read your paper? What are their backgrounds, motivations, interests, and beliefs? What are the key points you want a reader person to take away from your paper? Once you know the thesis and audience, you can determine what points your document should make to achieve its purpose. + +For each point in your paper, you need to explain both what and why. Start with what, but don't omit why. For example, it is not enough to state how an algorithm works; you should explain why it works in that way, or why another way of solving the problem would be different. Similarly, it is not sufficient to present a figure or facts. You must also ensure that reader understands the significance or implications of the figure and what parts of it are most important. + +Which details to include +Your purpose is to communicate specific ideas, and everything about your paper should contribute to this goal. If any part of the paper does not support your main point, then delete or change that part. You must be ruthless in cutting every irrelevant detail, however true it may be. Everything in your paper that does not support your main point distracts from it. + +Write for the readers, rather than writing for yourself. In particular, think about what matters to the intended audience, and focus on that. It is not necessarily what you personally find most intriguing. + +A common mistake is to focus on what you spent the most time on. Do not write your paper as a chronological narrative of all the things that you tried, and do not devote space in the paper proportionately to the amount of time you spent on each task. Most work that you do will never show up in any paper; the purpose of infrastructure-building and exploration of blind alleys is to enable you to do the small amount of work that is worth writing about. Another way of stating this is that the purpose of the paper is not to describe what you have done, but to inform readers of the successful outcome or significant results, and to convince readers of the validity of those conclusions. + +Likewise, do not dwell on details of the implementation or the experiments except insofar as they contribute to your main point. This is a particularly important piece of advice for software documentation, where you need to focus on the software's benefits to the user, and how to use it, rather than how you implemented it. However, it holds for technical papers as well — and remember that readers expect different things from the two types of writing! + +The audience is interested in what worked, and why, so start with that. If you discuss approaches that were not successful, do so briefly, and typically only after you have discussed the successful approach. Furthermore, the discussion should focus on differences from the successful technique, and if at all possible should provide general rules or lessons learned that will yield insight and help others to avoid such blind alleys in the future. + +Whenever you introduce a strawman or an inferior approach, say so upfront. A reader will (and should) assume that whatever you write in a paper is something you believe or advocate, unless very clearly marked otherwise. A paper should never first detail a technique, then (without forewarning) indicate that the technique is flawed and proceed to discuss another technique. Such surprises confuse and irritate readers. This mistake is often called “leading the reader down the garden path”. + +When there are multiple possible approaches to a problem, it is preferable to give the best or successful one first. Oftentimes it is not even necessary to discuss the alternatives. If you do, they should generally come after, not before, the successful one. Your paper should give the most important details first, and the less important ones afterward. Its main line of argument should flow coherently rather than being interrupted. It can be acceptable to state an imperfect solution first (with a clear indication that it is imperfect) if it is a simpler version of the full solution, and the full solution is a direct modification of the simpler one. Less commonly, it can be acceptable to state an imperfect solution first if it is an obvious solution that every reader will assume is adequate; but use care with this rationalization, since you are usually wrong that every reader will jump to the given conclusion. + +Make the organization and results clear +A paper should communicate the main ideas of your research (such as the techniques and results) early and clearly. Then, the body of the paper can expand on these points; a reader who understands the structure and big ideas can better appreciate the details. Another way of saying this is that you should give away the punchline. A technical paper is not a joke or a mystery novel. The reader should not encounter any surprises, only deeper explanations of ideas that have already been introduced. It's particularly irritating when an abstract or introduction states, “We evaluated the relationship between baldness and beekeeping”, with the key results buried pages later. A better abstract would say, “Male beekeepers are 25% more likely to be bald (p=.04), but there is no statistically significant correlation for female beekeepers.” + +The same advice applies at the level of sections and paragraphs. It is a bad approach to start with a mass of details and only at the end tell the reader what the main point was or how the details related to one another. Instead, state the point first and then support it. The reader is more likely to appreciate which evidence is important and why, and is less likely to become confused or frustrated. + +For each section of the paper, consider writing a mini-introduction that says what its organization is, what is in each subpart, and how the parts relate to one another. For the whole paper, this is probably a paragraph. For a section or sub-section, it can be as short as a sentence. This may feel redundant to you (the author), but readers haven't spent as much time with the paper's structure as you have, so they will truly appreciate these signposts that orient them within your text. + +Some people like to write the abstract, and often also the introduction, last. Doing so makes them easier to write, because the rest of the paper is already complete and can just be described. However, I prefer to write these sections early in the process (and then revise them as needed), because they frame the paper. If you know the paper's organization and outlook, then writing the front matter will take little effort. If you don't, then it is an excellent use of your time to determine that information by writing the front matter. To write the body of the paper without knowing its broad outlines will take more time in the long run. Another way of putting this is that writing the paper first will make writing the abstract faster, and writing the abstract first will make writing the paper faster. There is a lot more paper than abstract, so it makes sense to start with that and to clarify the point of the paper early on. + +It is a very common error to dive into the technical approach or the implementation details without first appropriately framing the problem and providing motivation and background. Readers need to understand what the task is before they are convinced that they should pay attention to what you are saying about it. You should first say what the problem or goal is, and — even when presenting an algorithm — first state what the output is and probably the key idea, before discussing steps. Avoid providing information that isn't useful to readers/users. It just distracts from the important content. + +Getting started: overcoming writer's block and procrastination +Some writers are overwhelmed by the emptiness of a blank page or editor buffer, and they have trouble getting started with their writing. Don't worry! Here are some tricks to help you get started. Once you have begun, you will find it relatively easier to revise your notes or first draft. The key idea is to write something, and you can improve it later. + +Start verbally. Explain what the paper needs to say to another person. After the conversation is over, write down what you just said, focusing on the main points rather than every word you spoke. Many people find it easier to speak than to write. Furthermore, getting feedback and giving clarifications will help you discover problems with your argument, explanation, or word choice. + +Outline. You may not be ready to write full English paragraphs, but you can decide which sections your paper will have and give them descriptive titles. Once you have decided on the section structure, you can write a little outline of each section, which indicates the subsection titles. Now, expand that into a topic sentence for each paragraph. At this point, since you know the exact topic of each paragraph, you will find the paragraph easy to write. + +Stream-of-consciousness notes. Write down everything that you know, in no particular order and with no particular formatting. Afterward, organize what you wrote thematically, bringing related points together. Eventually, convert it into an outline and proceed as above. While writing notes, use phrases/keywords, not complete sentences. The phrases are quicker to write and less likely to derail your brainstorming; they are easier to organize; and you will feel less attached to them and more willing to delete them. + +Divide and conquer. Rather than trying to write your entire document, choose some specific part, and write just that part. Then, move on to another part. + +Re-use. Find other text that you have written on the topic and start from that. An excellent source is your progress reports — you are writing them, aren't you? This can remind you what was hard or interesting, or of points that you might otherwise forget to make. You will rarely want to re-use text verbatim, both because you can probably convey the point better now, and also because writing for different audiences or in different contexts requires a different argument or phrasing. For example, a technical paper and a technical talk have similar aims but rather different forms. + +You must be willing to delete and/or rewrite your notes and early drafts. If you wrote something once, you can write it again (probably better!). Early on, the point is to organize your ideas, not to create finished sentences. + +Brevity +Be brief. Make every word count. If a word does not support your point, cut it out, because excess verbiage and fluff only make it harder for the reader to appreciate your message. Use shorter and more direct phrases wherever possible. + +Make your writing crisp and to the point. Eliminate any text that does not support your point. Here is one way you might go about this; it is time-consuming but extremely effective. First, examine each section of the paper in turn and ask what role it serves and whether it contributes to the paper's main point. If not, delete it. Next, within each section, examine each paragraph. Ask whether that paragraph has a single point. If not, rewrite the paragraph. Also ask whether that point contributes to the goals of the section. If not, then delete the paragraph. Next, within each paragraph, examine each sentence. If it does not make a single, clear point that strengthens the paragraph, delete or rewrite it. Finally, within each sentence, examine each word, and delete or replace those that do not strengthen their point. You will need to repeat this entire process multiple times, keeping a fresh perspective on the paper. + +Some people find it easier to follow this approach bottom-up, first cutting/rewriting words, then sentences, etc. + +Writing style +Passive voice has no place in technical writing. It obscures who the actor was, what caused it, and when it happened. Use active voice and simple, clear, direct phrasing. + +First person is rarely appropriate in technical writing. + +First person is appropriate when describing something that the author of the paper did manually. Recall that your paper should not be couched as a narrative. +Do not use “we” to mean “the author and the reader” or “the paper”. For example, do not write “In this section, we ...”. +Do not use “we” to describe the operation of a program or system. “We compute a graph” makes it sound like the authors did it by hand. As a related point, do not anthropomorphize computers: they hate it. Anthropomorphism, such as “the program thinks that ...”, is unclear and vague. +Avoid puffery, self-congratulation, superlatives, and subjective or value judgments: give the objective facts and let the reader judge. Avoid vague terms like “sizable” and “significant” (which are also subjective). Don't overuse the word “novel”. When I see a paper that is full of these, my rule of thumb is that the paper is trying too hard to cover up for scanty evidence. + +Do not use words like “clearly”, “easily”, “obviously”, and “trivially”, as in “Obviously, this Taylor series sums to π.” If the point is really obvious, then you are just wasting words by pointing it out. And if the point is not obvious to readers who are not intimately familiar with the subject matter the way you are, then you are offending readers by insulting their intelligence, and you are demonstrating your own inability to communicate the intuition. + +Prefer singular to plural number. In “sequences induce graphs”, it is not clear whether the two collections are in one-to-one correspondence, or the set of sequences collectively induces a set of graphs; “each sequence induces a graph” avoids this confusion. Likewise, in “graphs might contain paths”, it is unclear whether a given graph might contain multiple paths, or might contain at most one path. + +When describing an experiment or some other event or action that occurred in the past, use past tense. For example, the methodology section might say “We ran the program”. It would be ungrammatical and confusing to use present tense, as in “We run the program”. Present tense is for ongoing events (“I write this letter to inform you...”) or regular events (“I brush my teeth each day”), but not past events (“Yesterday, I eat dinner with my family”). It is also correct to say “Our methodology was to run the program”, where you use past tense “was” and the infinitive “to run”. + +When describing the paper itself, use present tense. “This paper shows that ...”. The reason for this is that the reader is experiencing the paper in real time. + +Avoid gratuitous use of the future tense “will ...”, as in, “switching the red and green wires will cause the bomb to explode”. It is unclear when the action will occur. If it is an immediate effect, use the shorter and more direct “switching the red and green wires causes the bomb to explode”. + +Use “previous work” instead of “existing work”. Your work exists, so “existing work” would refer to it as well. + +In a list with 3 or more elements list, put a serial comma between each of the items (including the last two). As a simple example of why, consider this 3-element grocery list written without the clarifying last comma: “milk, macaroni and cheese and crackers”. It's not clear whether that means { milk, macaroni and cheese, crackers } or { milk, macaroni, cheese and crackers }. As another example, “I would like to thank my parents, Rene Descartes and Ayn Rand,” suggests rather unusual parentage, whereas “I would like to thank my parents, Rene Descartes, and Ayn Rand,” shows a debt to four people. I've seen real examples that were even more confusing than these. + +In English, compound adjectives are hyphenated (except those whose first words end with “ly”, in some style guides) but compound nouns are not. Consider “the semantics provide name protection” versus “the name-protection semantics”. + +Prefer unambiguous words to ambiguous ones. Do not use “as” or “since” to mean “because”. Do not use “if” to mean “whether”. + +Use quotations sparingly. A clear paraphrase of the points that are relevant to your own work (along with a proper citation) is usually better than a long quotation from a previous publication. + +Avoid third-person pronouns when you can. The old standard was “he”, which is masculine chauvinist. The new standard is “he or she”, which can be viewed as heteronormative and which some people find clumsy. An emerging standard is “they” as a first-person singular pronoun, which is inclusive but grammatically incorrect and confusing (see comments above about singular vs. plural number). + +Some of the suggestions in this document are about good writing, and that might seem secondary to the research. But writing more clearly will help you think more clearly and often reveals flaws (or ideas!) that had previously been invisible even to you. Furthermore, if your writing is not good, then either readers will not be able to comprehend your good ideas, or readers will be (rightly) suspicious of your technical work. If you do not (or cannot) write well, why should readers believe you were any more careful in the research itself? The writing reflects on you, so make it reflect well. + +Figures +Use figures! Different people learn in different ways, so you should complement a textual or mathematical presentation with a graphical one. Even for people whose primary learning modality is textual, another presentation of the ideas can clarify, fill gaps, or enable the reader to verify his or her understanding. Figures can also help to illustrate concepts, draw a skimming reader into the text (or at least communicate a key idea to that reader). Figures make the paper more visually appealing. + +It is extremely helpful to give an example to clarify your ideas: this can make concrete in the reader's mind what your technique does (and why it is hard or interesting). A running example used throughout the paper is also helpful in illustrating how your algorithm works, and a single example permits you to amortize the time and space spent explaining the example (and the reader's time in appreciating it). It's harder to find or create a single example that you re-use throughout the paper, but it is worth it. + +A figure should stand on its own, containing all the information that is necessary to understand it. Good captions contain multiple sentences; the caption provides context and explanation. For examples of good, informative captions, see the print editions of magazines such as Scientific American and American Scientist. The caption should state what the figure illustrates or what conclusion a reader should draw from it. Don't write an obvious description of what the figure is, such as "Code example". Never write a caption like “The Foobar technique”; the caption should also say what the Foobar technique is, what it is good for, or how it works. The caption may also need to explain the meaning of columns in a table or of symbols in a figure. However, it's even better to put that information in the figure proper; for example, use labels or a legend. When the body of your paper contains information that belongs in a caption, there are several negative effects. The reader is forced to hunt all over the paper in order to understand the figure. The flow of the writing is interrupted with details that are relevant only when one is looking at the figure. The figures become ineffective at drawing in a reader who is scanning the paper — an important constituency that you should cater to! + +As with naming, use pictorial elements consistently. Only use two different types of arrows (or boxes, shading, etc.) when they denote distinct concepts; do not introduce inconsistency just because it pleases your personal aesthetic sense. Almost any diagram with multiple types of elements requires a legend (either explicitly in the diagram, or in the caption) to explain what each one means; and so do many diagrams with just one type of element, to explain what it means. + +Some writers label all the types of figures differently — some as “figure”, others as “table” or “graph” or “picture”. This differentiation has no benefits, but it does have a drawback: it is very hard for a reader to find “table 3”, which might appear after “figure 7” but before “freehand drawing 1”. You should simply call them all figures and number them sequentially. The body of each figure might be a table, a graph, a diagram, a screenshot, or any other content. + +Put figures at the top of the page, not in the middle or bottom. If a numbered, captioned figure appears in the middle or at the bottom of a page, it is harder for readers to find the next paragraph of text while reading, and harder to find the figure from a reference to it. + +Avoid bitmaps, which are hard to read. Export figures from your drawing program in a vector graphics format. If you must use a bitmap (which is only appropriate for screenshots of a tool), then produce them at very high resolution. Use the biggest-resolution screen you can, and magnify the portion you will capture. + +Don't waste text in the paper (and tax the reader's patience) regurgitating information that is expressed more precisely and concisely in a figure. For example, the text should not repeat the numbers from a table or graph. Text in the paper should add insight or explanations, or summarize the conclusions to be drawn from the data in the figure. + +Computer program source code +Your code examples should either be real code, or should be close to real code. Never use synthetic examples such as procedures or variables named foo or bar. Made-up examples are much harder for readers to understand and to build intuition regarding. Furthermore, they give the reader the impression that your technique is not applicable in practice — you couldn't find any real examples to illustrate it, so you had to make something up. + +Any boldface or other highlighting should be used to indicate the most important parts of a text. In code snippets, it should never be used to highlight syntactic elements such as “public” or “int”, because that is not the part to which you want to draw the reader's eye. (Even if your IDE happens to do that, it isn't appropriate for a paper.) For example, it would be acceptable to use boldface to indicate the names of procedures (helping the reader find them), but not their return types. + +Naming +Give each concept in your paper a descriptive name to make it more memorable to readers. Never use terms like “approach 1”, “approach 2”, or “our approach”, and avoid acronyms when possible. If you can't think of a good name, then quite likely you don't really understand the concept. Think harder about it to determine its most important or salient features. + +It is better to name a technique (or a paper section, etc.) based on what it does rather than how it does it. + +Use terms consistently and precisely. Avoid “elegant variation”, which uses different terms for the same concept to avoid boredom on the part of the reader or to emphasize different aspects of the concept. While elegant variation may be appropriate in poems, novels, and some essays, it is not acceptable in technical writing, where you should clearly define terms when they are first introduced, then use them consistently. If you switch wording gratuitously, you will confuse the reader and muddle your point. A reader of a technical paper expects that use of a different term flags a different meaning, and will wonder what subtle difference you are trying to highlight. Thus, don't confuse the reader by substituting “program”, “library”, “component”, “system”, and “artifact”, nor by conflating “technique”, “idea”, “method” and “approach”, nor by switching among “program”, “code”, and “source”. Choose the best word for the concept, and stick with it. + +Do not use a single term to refer to multiple concepts. If you use the term “technique” for every last idea that you introduce in your paper, then readers will become confused. This is a place that use of synonyms to distinguish concepts that are unrelated (from the point of view of your paper) is acceptable. For instance, you might always use “phase” when describing an algorithm but “step” when describing how a user uses a tool. + +When you present a list, be consistent in how you introduce each element, and either use special formatting to make them stand out or else state the size of the list. Don't use, “There are several reasons I am smart. I am intelligent. Second, I am bright. Also, I am clever. Finally, I am brilliant.” Instead, use “There are four reasons I am smart. First, I am intelligent. Second, I am bright. Third, I am clever. Fourth, I am brilliant.” Especially when the points are longer, this makes the argument much easier to follow. Some people worry that such consistency and repetition is pedantic or stilted, or it makes the writing hard to follow. There is no need for such concerns: none of these is the case. It's more important to make your argument clear than to achieve “elegant variation” at the expense of clarity. + +Choose good names not only for the concepts that you present in your paper, but for the document source file. Don't name the file after the conference to which you are submitting (the paper might be rejected) or the year. Even if the paper is accepted, such a name won't tell you what the paper is about when you look over your files in later years. Instead, give the paper or its folder/directory a name that reflects its content. Another benefit is that this will also lead you to think about the paper in terms of its content and contributions. + +Here is a piece of advice that is specific to computing: do not use the vague, nontechnical term “bug”. Instead, use one of the standard terms fault, error, or failure. A fault is an underlying defect in a system, introduced by a human. A failure is a user-visible manifestation of the fault or defect. In other circumstances, “bug report” may be more appropriate than “bug”. + +Numbers and measurements +Digits of precision: + +Don't report more digits of precision than the measurement process reliably and reproducibly produces. The 3rd or 4th digit of precision is rarely accurate and generalizable; if you don't have confidence that it is both repeatable and generalizable to new experiments, omit it. Another way to say this is that if you are not confident that a different set of experiments would produce all the same digits, then don't report so much precision. +Don't report more digits of precision than needed to convey your message. If the difference between 4.13 and 4 will not make a difference in convincing readers, then don't report the extra digits. Reporting extra digits can distract readers from the larger trends and the big picture. Including an inappropriate number of digits of precision can cast suspicion on all of your results, by giving readers the impression that you are statistically naive. +Use a consistent number of digits of precision. If the measured data are 1.23, 45.67, and 891.23, for example, you might report them as 1.23, 45.7, and 891, or as 1.2, 46, and 890, or as 1, 50, and 900. (An exception is when data are known to sum to a particular value; I would report 93% and 7% rather than either 93% and 7.4% or 90% and 7%. Often it's appropriate to report percentages as whole numbers rather than using the same precision.) +If you do any computations such as ratios, your computations should internally use the full precision of your actual measurements, even though your paper reports only a limited number of digits of precision. +If a measurement is exact, such as a count of items, then it can be acceptable to give the entire number even if it has many digits; by contrast, timings and other inexact measurements should always be reported with a limited number of digits of precision. +Do not confuse relative and absolute measurements. For instance, suppose your medicine cures 30% of patients, and the placebo cures 25% of patients. You could report that your medicine's cure rate is .3, the placebo's cure rate is .25, and your medicine's cure rate is either .05 greater (an equivalent way to say this is that it is 5 percentage points higher) or 20% greater. (Other correct, but less good, ways to say the same thing are that it cures 20% more, 120% as many, or 1.2 times as many patients.) It would be inaccurate to state that your medicine cures 5% more patients or your medicine cures 120% more patients. Just as you need to correctly use “120% more” versus “120% as many”, you need to correctly use “3 times faster than” versus “3 times as fast as”. A related, also common, confusion is between “3 times faster than and 3 times as fast as”. And, “2 times fewer” makes absolutely no sense. I would avoid these terms entirely. “Half as many” is a much better substitute for “2 times fewer”. + +Given the great ease of misunderstanding what a percentage means or what its denominator is, I try to avoid percentages and focus on fractions whenever possible, especially for base measurements. For comparisons between techniques, percentages can be acceptable. Avoid presenting two different measurements that are both percentages but have different denominators. + +Processing data +Your paper probably includes tables, bibliographies, or other content that is generated from external data. Your paper may also be written in a text formatting language such as LaTeX. In each of these cases, it is necessary to run some external command to create some of the content or to create the final PDF. + +All of the steps to create your final paper should be clearly documented — say, in comments or in a notes file that you maintain with the paper. Preferably, they should be automated so that you only have to run one command that collects all the data, creates the tables, and generates the final PDF. + +If you document and automate these steps, then you can easily regenerate the paper when needed. This is useful if you re-run experiments or analysis, or if you need to defend your results against a criticism by other researchers. If you leave some steps manual, then you or your colleagues are highly likely to make a mistake (leading to a scientific error) or to be unable to reproduce your results later. + +One good way to automate these tasks is by writing a program or creating a script for a build system such as Ant, Gradle, Make, Maven, etc. + +Related work +A related work section should not only explain what research others have done, but in each case should compare and contrast that to your work and also to other related work. After reading your related work section, a reader should understand the key idea and contribution of each significant piece of related work, how they fit together (what are the common themes or approaches in the research community?), and how your work differs. Don't write a related work section that is just a list of other papers, with a sentence about each one that was lifted from its abstract, and without any critical analysis nor deep comparison to other work. + +Unless your approach is a small variation on another technique, it is usually best to defer the related work to the end of the paper. When it comes first, it gives readers the impression that your work is rather derivative. (If this is true, it is your responsibility to convey that clearly; if it is not true, then it's misleading to intimate it.) You need to ensure that readers understand your technique in its entirety, and also understand its relationship to other work; different orders can work in different circumstances. + +Just as you should generally explain your technique first, and later show relationships with other work, it is also usually more effective to defer a detailed discussion of limitations to a later section rather than the main description of your technique. You should be straightforward and honest about the limitations, of course (do mention them early on, even if you don't detail them then), but don't destroy the coherence of your narrative or sour the reader on your technique. + +Feedback +Get feedback! Finish your paper well in advance, so that you can improve the writing. Even re-reading your own text after being away from it can show you things that you didn't notice. An outside reader can tell you even more. + +When readers misunderstand the paper, that is always at least partly the author's fault! Even if you think the readers have missed the point, you will learn how your work can be misinterpreted, and eliminating those ambiguities will improve the paper. + +Be considerate to your reviewers, who are spending their time to help you. Here are several ways to do that. + +As with submission to conferences, don't waste anyone's time if there are major flaws. Only ask someone to read (a part of) your paper when you think you will learn something new, because you are not aware of serious problems. If only parts are ready, it is best to indicate this in the paper itself (e.g., a TODO comment that the reader will see or a hand-written annotation on a hardcopy) rather than verbally or in email that can get forgotten or separated from the paper. + +It is most effective to get feedback sequentially rather than in parallel. Rather than asking 3 people to read the same version of your paper, ask one person to read the paper, then make corrections before asking the next person to read it, and so on. This prevents you from getting the same comments repeatedly — subsequent readers can give you new feedback rather than repeating what you already knew, and you'll get feedback on something that is closer to the final version. If you ask multiple reviewers at once, you are de-valuing their time — you are indicating that you don't mind if they waste their time saying something you already know. You might ask multiple reviewers if you are not confident of their judgment or if you are very confident the paper already is in good shape, in which case there are unlikely to be major issues that every reviewer stumbles over. + +It usually best not to email the document, but to provide a location from which reviewers can obtain the latest version of the paper, such as a version control repository or a URL you will update. That way, you won't clutter inboxes with many revisions, and readers can always get the most recent copy. + +Be generous with your time when colleagues need comments on their papers: you will help them, you will learn what to emulate or avoid, and they will be more willing to review your writing. + +Some of your best feedback will be from yourself, especially as you get more thoughtful and introspective about your writing. To take advantage of this, start writing early. One good way to do this is to write a periodic progress report that describes your successes and failures. The progress report will give you practice writing about your work, oftentimes trying out new explanations. + +Whereas you should start writing as early as possible, you don't need to put that writing in the form of a technical paper right away. In fact, it's usually best to outline the technical paper, and get feedback on that, before you start to fill in the sections with text. (You might think that you can copy existing text into the paper, but it usually works out better to write the information anew. With your knowledge of the overall structure, goals, and audience, you will be able to do a much better job that fits with the paper's narrative.) When outlining, I like to start with one sentence about the paper; then write one sentence for each section of the paper; then write one sentence for each subsection; then write one sentence for each paragraph (think of this as the topic sentence); and at that point, it's remarkably easy just to flesh out the paragraphs. + +When to submit your paper for publication +You should not submit your paper too early, when it does not reflect well on you and a submission would waste the community's reviewing resources. You should not submit your paper too late, because then the community is deprived of your scientific insights. In general, you should err on the side of submitting too late rather than too early. + +A rule of thumb is to submit only if you are proud for the world to associate your name with the work, in its current form. If you know of significant criticisms that reviewers might raise, then don't submit the paper. + +Submitting your paper prematurely has many negative consequences. + +You will waste the time of hard-working reviewers, who will give you feedback that you could have obtained in other ways. +You will get a reputation for shoddy work. +You will make the paper less likely to be accepted in the future. Oftentimes the same reviewers may serve two different venues. Reviewing a paper again puts a reviewer in a negative state of mind. I have frequently heard reviewers say, “I read an earlier version of this paper, it was a bad paper, and this version is similar.” (This is unethical because reviewers are not supposed to talk about papers they have reviewed, but nonetheless it is very common.) Now the paper will likely be rejected again, and the whole committee gets a bad impression of you. A reviewer who has read a previous version of the paper may read the resubmission less carefully or make assumptions based on a previous version. To sum up: it's harder to get a given paper accepted on its second submission, than it would have been to get the identical paper accepted on its first submission. +Here are some bad reasons to submit a paper. + +You need feedback on the paper. +It's true that the feedback from reviewers is extraordinarily valuable to you and will help you improve the paper. However, you should get feedback from other scientists (your friends and colleagues) before submitting for publication. + +Submitting the paper in its current form means more papers on your c.v. and more opportunities for others to learn about your work. +Those are true facts, and some people do “salami-slice” their research into as many papers as possible — such papers are called a “least publishable unit”. However, doing so leads to less impact than publishing fewer papers, each one with more content. If a paper contains few contributions, it is less likely to make a big impression, because it is less exciting. In addition, readers won't enjoy reading many pages to learn just a few facts. + +Note: This point refers to taking a single research idea or theme and splitting it into multiple publications. When there are multiple distinct research contributions, it can be appropriate to describe them in different papers. + +To roll the dice for acceptance on a paper that is better than some other published papers. +The reviewing process can be frustrating, because it contains a great deal of randomness: the same paper would be rejected by some reviewers and accepted by others. However, all great papers are accepted and all bad papers are rejected. For mediocre papers, luck plays a role. Your goal should not be to write great papers, not mediocre ones. Find a way to improve your paper. Recognize the great value of reviews: they provide a valuable perspective on your work and how to improve it, even if you feel that the reviewer should have done a better job. + +You are tired of the paper and need a break from it. +If you aren't excited about the paper, it is unlikely that other people will be. Furthermore, the period after submitting the paper is not a time to take a break, but an opportunity to further improve it. + +After you submit a paper, don't stop working on it! You can always improve the research. For instance, you might expand the experiments, improve the implementation, or make other changes. Even if your paper is accepted, you want the accepted version to be as impressive as possible. And if the paper is rejected, you need to have a better paper to submit to the next venue. + +Responding to conference reviews +(This section is most relevant to fields like computer science where conferences are the premier publication venue. Responding to journal reviews is different.) + +Many conferences provide an author response period: the authors are shown the reviews and are given limited space (say, 500 words) to respond to the reviews, such as by clarifying misunderstandings or answering questions. The author response is sometimes called a “rebuttal”, but I don't like that term because it sets an adversarial tone. + +Your paper will only be accepted if there is a champion for the paper: someone who is excited about it and will try to convince the rest of the committee to accept the paper. Your response needs to give information to your champion to overcome objections. If there isn't a champion, then the main goal of your response is to create that champion. Your response should also give information to detractors to soften their opposition. + +After reading the reviews, you may be disappointed or angry. Take a break to overcome this, so that you can think clearly. + +For every point in the reviews, write a brief response. Do this in email-response style, to ensure that you did not miss any points. You will want to save this for later, so it can be better to do this in the paper's version control repository, rather than in a WYSIWYG editor such as Google Docs. (This assumes you have a version control repository for the paper, which you should!) Much of this text won't go in your response, but it is essential for formulating the response. + +Summarize (in 5 or so bullet points, however many make sense) the key concerns of the reviewers. Your review needs to focus on the most important and substantive critiques. The authors of the paper should agree on this structure before you start to write the actual response. + +Your response to each point will be one paragraph in your response. Start the paragraph with a brief heading or title about the point. Do not assume that the reviewers remember everything that was written by every reviewer, nor that they will re-read their reviews before reading your response. A little context will help them determine what you are talking about and will make the review stand on its own. This also lets you frame the issues in your own words, which may be clearer or address a more relevant point than the reviews did. + +Organize your responses thematically. Group the paragraphs into sections, and have a small heading/title for each section. If a given section has just one paragraph, then you can use the paragraph heading as the section heading. Order the sections from most to least important. + +This is better than organizing your response by reviewer, first addressing the comments of reviewer 1, then reviewer 2, and so forth. Downsides of by-reviewer organization include: + +It can encourage you not to give sufficient context. +It does not encourage putting related information together nor important information first. +You want to encourage all reviewers to read the entire response, rather than encouraging them to just look at one part. +When multiple reviewers raised the same issue, then no matter where you address it, it's possible for a reviewer to overlook it and think you failed to address it. +You don't want to make glaringly obvious which issues in a review you had to ignore (for reasons of space or other reasons). +You don't want to make glaringly obvious that you spent much more time and space on one reviewer than another. +In general, it's best not to mention reviewer names/numbers in your response at all. Make the response be about the science, not about the people. + +In your responses, admit your errors forthrightly. Don't ignore or avoid key issues, especially ones that multiple reviewers brought up. + +Finally, be civil and thankful the reviewers. They have spent considerable time and energy to give you feedback (even if it doesn't seem to you that they have!), and you should be grateful and courteous in return. + +Rejection +If you submit technical papers, you will experience rejection. In some cases, rejection indicates that you should move on and begin a different line of research. In most cases, the reviews offer an opportunity to improve the work, and so you should be very grateful for a rejection! It is much better for your career if a good paper appears at a later date, rather than a poor paper earlier or a sequence of weak papers. + +Even small flaws or omissions in an otherwise good paper may lead to rejection. This is particularly at the elite venues with small acceptance rates, where you should aim your work. Referees are generally people of good will, but different referees at a conference may have different standards, so the luck of the draw in referees is a factor in acceptance. + +The wrong lesson to learn from rejection is discouragement or a sense of personal failure. Many papers — even papers that later win awards — are rejected at least once. The feedback you receive, and the opportunity to return to your work, will invariably improve your results. + +Don't be put off by a negative tone in the reviews. The referees are trying to help you, and the bast way to do that is to point out how your work can be improved. I often write a much longer review, with more suggestions for improvement, for papers that I like; if the paper is terrible, I may not be able to make as many concrete suggestions, or my high-level comments may make detailed comments moot. + +If a reviewer didn't understand something, then the main fault almost always lies with your writing. If you blame a lazy or dumb reviewer, you are missing the opportunity to improve. Reviewers are not perfect, but they work hard to give you helpful suggestions, so you should give them the benefit of the doubt. Remember that just as it is hard to convey technical ideas in your paper (and if you are getting a rejection, that is evidence that you did not succeed!), it is hard to convey them in a review, and the review is written in a few hours rather than the weeks you spent on the paper (not to mention months or years of understanding the concepts). You should closely attend to both the explicit comments, and to underlying issues that may have led to those comments — it isn't always easy to capture every possible comment in a coherent manner. Think about how to improve your research and your writing, even beyond the explicit suggestions in the review — the prime responsibility for your research and writing belongs with you. + +Norman Ramsey's advice +Norman Ramsey's nice Teach Technical Writing in Two Hours per Week espouses a similar approach to mine: by focusing on clarity in your writing, you will inevitably gain clarity in your thinking. + +Don't bother to read both the student and instructor manuals — the student one is a subset of the instructor one. You can get much of the benefit from just one part, his excellent “principles and practices of successful writers”: + +Principles + +Correctness. Write correct English, but know that you have more latitude than your high-school English teachers may have given you. +Consistent names. Refer to each significant character (algorithm, concept, language) using the same word everywhere. Give a significant new character a proper name. +Singular. To distinguish one-to-one relationships from n-to-m relationships, refer to each item in the singular, not the plural. +Subjects and verbs. Put your important characters in subjects, and join each subject to a verb that expresses a significant action. +Information flow. In each sentence, move your reader from familiar information to new information. +Emphasis. For material you want to carry weight or be remembered, use the end of a sentence. +Coherence. In a coherent passage, choose subjects that refer to a consistent set of related concepts. +Parallel structure. Order your text so your reader can easily see how related concepts are different and how they are similar. +Abstract. In an abstract, don't enumerate a list of topics covered; instead, convey the essential information found in your paper. +Practices + +Write in brief daily sessions. Ignore the common myth that successful writing requires large, uninterrupted blocks of time — instead, practice writing in brief, daily sessions. +Focus on the process, not the product. Don't worry about the size or quality of your output; instead, reward yourself for the consistency and regularity of your input. +Prewrite. Don't be afraid to think before you write, or even jot down notes, diagrams, and so on. +Use index cards. Use them to plan a draft or to organize or reorganize a large unit like a section or chapter. +Write a Shitty First Draft™. Value a first draft not because it's great but because it's there. +Don't worry about page limits. Write the paper you want, then cut it down to size. +Cut. Plan a revision session in which your only goal is to cut. +Other resources +Norman Ramsey's advice, excerpted immediately above. +“Hints on writing an M.Eng. thesis”, by Jeremy Nimmer +my notes on reviewing a technical paper, which indicate how to recognize — and thus produce — quality work +my notes on choosing a venue for publication +my notes on giving a technical talk: a talk has the same goal as a paper, namely to convey technical ideas +my notes on making a technical poster +Ronald B. Standler's advice on technical writing +Dave Patterson's Writing Advice +Advice on SIGPLAN conference submissions (at bottom of page) +The Elements of Style, William Strunk Jr. and E. B. White, is classic book on improving your writing. It focuses at a low level, on English usage. +Style: Toward Clarity and Grace, by Joseph M. Williams, is another general-purpose writing guide, with a somewhat higher-level focus than that of Strunk & White. +The Sense of Style: The Thinking Person's Guide to Writing in the 21st Century, by Steven Pinker, is an excellent guide to writing. It gives reasons (from psychology and other scientific fields) for its advice, making it more authoritative than someone's opinion. \ No newline at end of file diff --git a/paper/related-works/alpaserve.pdf b/related-works/alpaserve.pdf similarity index 100% rename from paper/related-works/alpaserve.pdf rename to related-works/alpaserve.pdf diff --git a/paper/related-works/cachegen.pdf b/related-works/cachegen.pdf similarity index 100% rename from paper/related-works/cachegen.pdf rename to related-works/cachegen.pdf diff --git a/paper/related-works/distserve.pdf b/related-works/distserve.pdf similarity index 100% rename from paper/related-works/distserve.pdf rename to related-works/distserve.pdf diff --git a/paper/related-works/efficient-inference-survey.pdf b/related-works/efficient-inference-survey.pdf similarity index 100% rename from paper/related-works/efficient-inference-survey.pdf rename to related-works/efficient-inference-survey.pdf diff --git a/paper/related-works/fastserve.pdf b/related-works/fastserve.pdf similarity index 100% rename from paper/related-works/fastserve.pdf rename to related-works/fastserve.pdf diff --git a/paper/related-works/flexgen.pdf b/related-works/flexgen.pdf similarity index 100% rename from paper/related-works/flexgen.pdf rename to related-works/flexgen.pdf diff --git a/paper/related-works/funsearch.pdf b/related-works/funsearch.pdf similarity index 100% rename from paper/related-works/funsearch.pdf rename to related-works/funsearch.pdf diff --git a/paper/related-works/infercept.pdf b/related-works/infercept.pdf similarity index 100% rename from paper/related-works/infercept.pdf rename to related-works/infercept.pdf diff --git a/paper/related-works/infersim-repo.pdf b/related-works/infersim-repo.pdf similarity index 100% rename from paper/related-works/infersim-repo.pdf rename to related-works/infersim-repo.pdf diff --git a/paper/related-works/llm-inference-roofline-survey.pdf b/related-works/llm-inference-roofline-survey.pdf similarity index 100% rename from paper/related-works/llm-inference-roofline-survey.pdf rename to related-works/llm-inference-roofline-survey.pdf diff --git a/paper/related-works/llm-serving-survey.pdf b/related-works/llm-serving-survey.pdf similarity index 100% rename from paper/related-works/llm-serving-survey.pdf rename to related-works/llm-serving-survey.pdf diff --git a/paper/related-works/loongserve.pdf b/related-works/loongserve.pdf similarity index 100% rename from paper/related-works/loongserve.pdf rename to related-works/loongserve.pdf diff --git a/paper/related-works/megatron-lm.pdf b/related-works/megatron-lm.pdf similarity index 100% rename from paper/related-works/megatron-lm.pdf rename to related-works/megatron-lm.pdf diff --git a/paper/related-works/memserve.pdf b/related-works/memserve.pdf similarity index 100% rename from paper/related-works/memserve.pdf rename to related-works/memserve.pdf diff --git a/paper/related-works/mooncake.pdf b/related-works/mooncake.pdf similarity index 100% rename from paper/related-works/mooncake.pdf rename to related-works/mooncake.pdf diff --git a/paper/related-works/phantora.pdf b/related-works/phantora.pdf similarity index 100% rename from paper/related-works/phantora.pdf rename to related-works/phantora.pdf diff --git a/paper/related-works/s3-gpu-utilization.pdf b/related-works/s3-gpu-utilization.pdf similarity index 100% rename from paper/related-works/s3-gpu-utilization.pdf rename to related-works/s3-gpu-utilization.pdf diff --git a/paper/related-works/sarathi-serve.pdf b/related-works/sarathi-serve.pdf similarity index 100% rename from paper/related-works/sarathi-serve.pdf rename to related-works/sarathi-serve.pdf diff --git a/paper/related-works/sarathi.pdf b/related-works/sarathi.pdf similarity index 100% rename from paper/related-works/sarathi.pdf rename to related-works/sarathi.pdf diff --git a/paper/related-works/servegen.pdf b/related-works/servegen.pdf similarity index 100% rename from paper/related-works/servegen.pdf rename to related-works/servegen.pdf diff --git a/paper/related-works/serverlessllm.pdf b/related-works/serverlessllm.pdf similarity index 100% rename from paper/related-works/serverlessllm.pdf rename to related-works/serverlessllm.pdf diff --git a/paper/related-works/sglang.pdf b/related-works/sglang.pdf similarity index 100% rename from paper/related-works/sglang.pdf rename to related-works/sglang.pdf diff --git a/paper/related-works/slora.pdf b/related-works/slora.pdf similarity index 100% rename from paper/related-works/slora.pdf rename to related-works/slora.pdf diff --git a/paper/related-works/splitwise.pdf b/related-works/splitwise.pdf similarity index 100% rename from paper/related-works/splitwise.pdf rename to related-works/splitwise.pdf diff --git a/paper/related-works/vidur.pdf b/related-works/vidur.pdf similarity index 100% rename from paper/related-works/vidur.pdf rename to related-works/vidur.pdf diff --git a/paper/related-works/vllm.pdf b/related-works/vllm.pdf similarity index 100% rename from paper/related-works/vllm.pdf rename to related-works/vllm.pdf