Vacuous Truth in Computer Science

In discrete mathematics, a universally quantified statement has the form:

This means that for every element in set , some property holds.

But what happens if is empty?

The statement is still true! Since there are no elements in , there can’t be any counterexamples. This is called vacuous truth.

For example, “Every unicorn in my apartment is blue” is technically true if there are no unicorns in my apartment.

This idea appears in programming, too:

all([])  # True

Python’s all() returns True for an empty iterable because no element violates the condition.

What’s interesting is that the truth comes entirely from the absence of objects that we are making claims about.

Buddhism Through Vacuous Truth

Conclusion

Of course, vacuous truth is a logical property, not a proof of Buddhist philosophy. But I find this analogy particularly useful as it frames challenging Buddhist concepts using vacuous truth, a concept that I am already familiar with.

The problem

Recently I’ve been building a lot of services on my homelab, with each one needing resources from some number of LLMs. Each service may use a different model, since each model has its own strengths and weaknesses. For example, in p80, my language learning tool, I wanted the flexibility of choosing a model that specializes in a certain language — German, which I am learning right now. However, my job-tracker service needs the largest model I can fit on my DGX Spark for its nightly runs. The issue is that I would have to manually switch between models like this if I wanted both the biggest model possible and a smaller tailored model like my German translation model. If I didn’t want to manually switch between the two, then I would have to sacrifice and cut down the size of the big model to make room for the smaller ones.

The naive fix is to just run a vLLM server per model, and that breaks down fast. The models collectively don’t fit. Every service ends up owning inference-server lifecycle, which isn’t its job. Whoever swaps last wins. And a low-traffic model can sit blocked behind a busy one forever. Serving inference was never the hard part here as vLLM already does that well. The hard part is deciding which model gets to be loaded right now, and that decision needs one owner.

A key observation I made was that my services, of which I had a lot, all didn’t require real-time chat with an LLM. For example, job-tracker needs an LLM for job posting ranking, matching, and prefills, all of which can run async (I’m fine waiting minutes or hours for a response). This pattern seemed to be repeated across all my homelab services.

Tools I tried and considered, and why they weren’t enough for my case

I considered multiple existing tools to solve my problem, but ultimately found none that fit my specific situation: a single-user homelab setup that needs an intelligent queue.

First I found llama-swap, which seemed to fit my problem nicely. llama-swap is an open-source proxy that dynamically starts, stops, and switches between local LLM inference servers based on the model a client asks for, so several models can share one GPU without all of them staying resident. That is most of what I wanted, and it can use an inference server like vLLM underneath. The problem is that llama-swap swaps in response to whichever request happens to arrive next, but my whole premise is that this work is async and can be queued — so the interesting decision is which of the waiting jobs should run next, not which one showed up first. Since my system already owns the queue, adding llama-swap would be a second orchestration layer that still left the scheduling to me. I kept vLLM as the inference backend and implemented model lifecycle management directly alongside the queue and scheduler.

Something I considered for quite a while was NVIDIA Triton. Triton loads and unloads models too, but it goes about it differently than llama-swap. You can run it with --model-control-mode explicit and one process can serve generation on its OpenAI port while exposing a load/unload API on a separate port. This allowed for not needing a container to stop and start, or no process to respawn. You just had to tell it to drop one model and bring up another. That is very close to the shape I wanted.

Where it fell over was structured output. Two of my services ask for JSON back using response_format, and Triton’s OpenAI frontend accepts that field and then drops it before it ever reaches vLLM. This issue appeared exactly twice in the whole frontend: once in the request schema that accepts it, and once in the list of things excluded on the way through. So job-tracker gets a 422 back, and my translation service gets unconstrained prose that fails every JSON parse downstream. I considered patching it myself, however that would leave me re-applying and re-verifying on every Triton upgrade, just to get back a behaviour that the vLLM already does natively.

My solution

The solution I came to was to build a scheduler, which I called SIR (stupid-inference-router). It exposes an endpoint on :8000 and listens for LLM requests. When it hears a request for inference (in the standard OpenAI-compatible shape), it appends it to an internal queue. The scheduler then picks the best item on the queue to run, based on which model is currently loaded and what else is waiting, so it doesn’t thrash, and runs inference. SIR does not know who needs inference or why, it only schedules and sends the answer back.

  Service A   Service B   Service C   Service D
      └───────────┴─────┬─────┴───────────┘
                        ▼
              ┌──────────────────┐
              │       sir        │   API layer  ─ validate, enqueue, stream back
              │      :8000       │   Scheduler  ─ which model runs next?
              └────────┬─────────┘   Registry   ─ what models exist, what they cost
                       ▼
              Backend Manager (pluggable)
                       ▼
                  vLLM / Mock
                       ▼
                   DGX Spark

The line I cared most about drawing is the one under the scheduler. The scheduler decides what should run, and the backend manager decides how to run it, so the scheduler never touches a vLLM process directly. That separation is what allowed me try Triton and then decide not to move forward with it without touching any of the scheduling code, and it’s let me run the whole test suite runs against a mock backend.

The scheduling algorithm I went with initially

One thing I definitely wanted to avoid for the scheduling algorithm was plain FIFO. Suppose my queue is:

A A B A C C B A

A FIFO scheduler might cause:

load A
load B
load A
load C
load B
load A

However, I want my scheduler to consider model-switch cost and produce something closer to:

load A → A A A A
load B → B B
load C → C C

Serving the model that is already loaded costs nothing, however serving any other model costs about ~5 minutes on my box. So the main goal of my scheduler is deciding when a waiting model has become worth 5 minutes of not generating anything

So the algorithm I chose for the first version scores models, not requests. Every model with pending work gets a score:

demand(m) = age_weight * oldest_wait(m) + depth_weight * depth(m)
score(m)  = priority(m) * demand(m) - switch_cost(m)

switch_cost is zero for whatever model is already loaded, and switch_cost_weight * estimated_load_seconds for everyone else.

One nice consequence of this is that hysteresis falls out of it for free. If a load costs 8 seconds, a single fresh request for model B can’t take the GPU away from A immediately; it has to sit there accumulating age until its demand outgrows the load it would cause.

For scoring I defined 2 main rules:

• Minimum residency. A freshly loaded model keeps the GPU for min_residency_seconds while it still has work, so alternating A B A B traffic buys one swap instead of one swap per request. A one issue here is what counts as “still wants the GPU”. Queue depth alone is not good enough here, since a model serving four requests a second has an empty queue most of the time, and only reading those gaps as “done” gives the GPU away mid-workload which is what we are trying to prevent. So the hold only lapses early once the model has been continuously idle for longer than the swap.
• A starvation ceiling. Past max_wait_seconds, a waiting model becomes mandatory, outranking priority, hysteresis, and switch cost. I wanted fairness to be a core feature instead of just a weight since it’s the last line of defense for when the rest of the weights might be wrong.

To make sure it worked, I built a test to simulate switching between 2 models, shown below:

switch cost only, no minimum residency
  chat        wait p50   9.8s   p99  15.9s   served 10/10
  translate   wait p50   9.0s   p99  15.2s   served 10/10
  swaps 7   loads 8   loading 56.0s

as shipped (min_residency 30s)
  chat        wait p50   0.0s   p99  33.0s   served 10/10
  translate   wait p50  11.5s   p99  41.5s   served 10/10
  swaps 2   loads 3   loading 22.0s

Seven swaps was reduced to two, and the time spent loading weights instead of generating tokens dropped by more than half in the test. In the test, the p99 wait gets noticeably worse (15.9s to 33.0s for chat) but thats exactly the tradeoff I made in making SIR. I traded tail latency for GPU time intentionally, and now I can see the size of the trade instead of just assuming it.

The numbers above come from the mock backend, so they tell me the policy is reasonable, not what it will do on the Spark. Each of my decisions also got logged with the score breakdown that produced it, which helps me answer “why did it just do that” from one log line. If I have no visability into what is happening SIR would quickly become hard to trust.

Issue 1: HTTP timeout

My first thought was to have all the services think they were sending to a vLLM or SGLang inference provider (using the default OpenAI request shapes) on port :8000, but actually send to SIR. I loved this idea, because it meant I could swap SIR out for a standard SGLang or vLLM server whenever I wanted, and nothing on the service side would need to change or break. The one issue is that with something like vLLM, the service expects an HTTP response in seconds, not minutes or hours. On a synchronous path, every service waiting for SIR to get to its request would block. Furthermore, a connection held open that long isn’t a durable thing: any proxy or idle timeout between the service and SIR can drop it. And if the service crashed or restarted, SIR would have no way of knowing whether anyone still wanted the answer, so it would happily spend GPU time generating a response into a socket nobody was reading.

Issue 2: services are no longer provider agnostic

One issue the polling approach above exposed is that the services are no longer LLM-provider agnostic. If I still wanted SIR to be swappable with vLLM or SGLang, each service would have to handle two different response shapes: a queued shape, which vLLM and SGLang never send, and a normal completion. Making every service branch on that was exactly what I didn’t want. The principle I was reaching for is the Adapter pattern — encapsulate what varies behind a stable interface, so the thing that changes changes in one place instead of six.

Next steps

I proved everything above against a mock backend and one real vLLM. What is still unproven in this design is that the deployment currently only runs a single model, and a single model never swaps. So my scheduling algorithm hasn’t actually made a decision under load yet. However, SIR is designed for it now, and all my services already talk to the SIR client package, so adding a second model requires no code change in any of them. The case where I would use smaller LLMs is my p80 language learning tool, and per-language models there are still a TODO.

Some further steps could be adding something like a simple prediction algorithm, which could use historical data (which models were requested, and when) and SIR could preemptively load and unload models based on the probability of needing them at that time.