For a long time my selection logic was a weighted sum. Several factors, each with a coefficient I had chosen, multiplied and added into a single score, highest score wins. It is the obvious design and it has an obvious problem: I was the one setting the weights. Every coefficient was a small, untestable opinion about how much one thing mattered relative to another, and I had no honest way to know if any of them were right. When the picks were bad I could always re-tune. That is not calibration. It is me moving knobs until the output flatters my prior.
So I threw the rubric out and built a tournament instead. Here is the whole design.
Every morning a pool of candidates arrives, on a typical day around 94 of them, each one a single option contract with its flow, greeks, technicals, and news folded into one JSON blob. The selector runs three independent brackets over that pool. Each bracket shuffles the pool with its own fixed seed, splits it into batches of at most ten, and asks the model a single question per batch: which of these would you buy. The top two from each batch advance. Shuffle, batch, advance, repeat. Roughly 94 collapses to 20, then to 4, then to 1. Three brackets, three winners.
Two structural choices carry the whole thing.
First, top two advance, not top one. Single-elimination is the intuitive bracket, and it is wrong here, because the batches are random. A genuinely strong candidate can draw into a batch with an even stronger one and get knocked out on luck of the draw rather than on merit. Advancing two per batch gives the second-best in any unlucky grouping a path forward. It costs a little more compute and it stops the randomness from eating good candidates.
Second, the three brackets are the point, not redundancy. Because each bracket reshuffles independently, the same pool gets explored along three different paths. When all three brackets land on the same contract, that agreement happened despite the randomness, not because of it. When they split three ways, the pool genuinely had no standout and the method is telling me so. So I read the consensus directly as confidence. Three of three is high. Two of three is medium. One of three is low. I did not design that confidence scale and then calibrate it. It falls out of the method. The disagreement between identical judges is the signal.
The prompt is almost aggressively plain. No rubric, no weights, no scoring schema, no memory of past picks. It states the goal in one line, hands over the day's market report as shared context, and shows the contracts. That plainness is deliberate. The old weighted scorer encoded my beliefs about importance into numbers; the tournament refuses to encode any. It asks a capable model the same simple question many times under reshuffled conditions and lets the structure, not my coefficients, produce the ranking and the confidence.
There are two failure modes worth calling out, because a teardown that skips them is selling something. The first is transport. If a batch call fails and returns nothing, naively that silently eliminates up to ten candidates, which would let an outage masquerade as a verdict. So if more than half the batches in a round come back empty, that entire bracket is aborted and the remaining brackets vote without it. A broken bracket abstains rather than lying. The second is cost. Each daily decision runs around $0.85, dominated almost entirely by serializing every contract to JSON three times over, roughly 408 thousand input tokens against 3.5 thousand out. For one decision a day that is nothing. If you fanned this pattern across thousands of items an hour, the serialization, not the reasoning, is what you would optimize first.
Does it pick better than the weighted rubric did? I am still accumulating the closed sample to answer that with a number, and I will not pretend a tournament is automatically smarter than a sum. But it is honest in a way the rubric never was. The rubric gave me a single score and an unearned feeling of precision. The tournament gives me a pick and a confidence I did not author, one that gets quieter on its own when the field is mediocre. For a one-shot daily decision where I care more about knowing when not to trust the output than about squeezing out the last basis point, I will take the method that argues with itself.
If you are building selection or ranking on top of an LLM and your confidence number is really just a knob you tuned, there is a better way to earn it. That is the kind of system I build: /services.