SupraAgent benchmark: modules, long sources, poisoned corpora and concurrency

Status 12 November 2026. The SupraAgent benchmark measures, separately, whether retrieval finds the right module source, whether facts survive in very long sources, how the answer path behaves under poisoned corpora, and how fast the local components run cold, warm and under concurrent tenants. Every run is a local diagnostic: real retrieval, admission and synthesis components over a fixture index, deterministic extraction, no model-provider calls and no executed business actions. The questions, sources and attacks were written by us. The numbers below are the current state of that benchmark. They are not answer correctness, not a user study and not evidence of being the best agent.

Module routing: 96 questions over 1,037 documentation sections

The questions cover People, Teaching, Attema, Blogery, Store Connect, Deals, IoT Manage, Shortcuts, iAccelerator, Help Center, Templaterr and SupraAgent. Each module has four question types in German and English: actions with inputs and prerequisites, activities and status, process steps with verifiable outcomes, and support for permission errors and missing information. The corpus contains 1,037 complete endpoint documentation sections; both retrievers see the same sources. The replay now fetches 32 candidates before the admission gate and keeps the first four admitted, exactly as production does, so a rejected document no longer consumes one of the four slots.

Hit@1 matters because readers act on the first listed source (Craswell et al., WSDM 2008). BM25 scores fields separately with the section title weighted 2.5 (Robertson, Zaragoza and Taylor, CIKM 2004). A module-coherence feedback step (Lavrenko and Croft, SIGIR 2001) was implemented and measured: hit@1 77 → 91, MRR 0.891 → 0.970, no question ranking its module source lower, paired difference +0.146 over 48 question families (95 % bootstrap interval 0.073–0.229, sign-flip p = 0.0005; Sakai, SIGIR 2006; Smucker, Allan and Carterette, CIKM 2007). It is switched off because 9 of 288 runs (3 German Shortcuts questions × 3 repeats) lose their citation: the four Shortcuts sections are single units without sentence punctuation, longer than the 420-character snippet budget, so the extractive synthesizer has nothing to quote once the wrong-module section it used to cite is no longer ranked. With the switch off, first-position routing is unchanged. Answer texts changed in 141 of 576 runs against the previous state although sources and ranks are identical: a unit that several admitted sources repeat is now listed once and the freed line is filled from the next source (citation markers 918 → 879 on BM25). Timing is one workstation without model calls or production IO, read against the 0.1-second perceptual threshold of Card, Robertson and Mackinlay, CHI 1991, not as a service commitment. Hash vectors are deterministic lexical hashes, not a neural embedding baseline.

1. Module routing: first and top-four sources per retriever, 96 questions, first repeat.

Every question type reaches its module within the top four. First-position routing is weakest for process questions and for Shortcuts (1 of 8 first) and Store Connect (3 of 8); language and question-type rows overlap and must not be added together.

2. Routing per question type, BM25, 24 questions per type in German and English.

Sealed holdout: 96 new questions, evaluated once

A holdout reused for tuning loses its validity (Dwork et al., Science 2015). 96 new questions, 25 of which do not name their module, each with a gold documentation section, were sealed by SHA-256 on 11 November 2026 before any change of this round and replayed once after every decision, with the shipped defaults and with the coherence switch on.

Hash vectors reach module hit@4 in 48 of 96 and hit@1 in 38 of 96. The holdout shows what the development set hid: the coherence step raises hit@1 by 6 but lowers hit@4 by 3 of 96, so keeping it off stands on the holdout as well as on the citation gate. Four holdout answers carry no citation in both states, and the gold section is among the four listed sources in only 41 of 96 questions: module routing is not section routing, and that gap is the clearest open target. The holdout is now unsealed and counts as a development set from here on.

Long sources: 144 controlled and 180 decoy-dense cases up to 1,024,000 characters

The study crosses six source lengths, 8,000 to 1,024,000 characters, with evidence at the beginning, middle and end, in German and English, as motivated by Lost in the Middle (Liu et al., TACL 2024). Four controlled scenarios test a positive fact, an oversized negation, an injected instruction and an absent fact. A decoy-dense set repeats the product name and every question word in every paragraph, with other products' numbers in the same sentence template, so the partial-read windows beyond the fully scanned prefix are exercised instead of keyword uniqueness. Each case runs three times for two actors, the previous and the current code, and is measured at the provider-prompt boundary and in the deterministic answer. Lengths are Unicode characters, not model tokens, and no model comprehension is measured (Bai et al., ACL 2024).

The complete scan of a source is bounded at 200,000 characters and 4,096 source units. Beyond that bound the packer segments at most sixteen windows of 8,192 characters on either side of whole-word matches of the question terms, discards every window-edge sentence and keeps an opened parenthesis or list from being cut by a window edge. Each bound that is reached is a named omission in the provider prompt, and the deterministic answer opens with one localized sentence saying that the source was only partially read (guidelines G2 and G11 of Amershi et al., CHI 2019). The packer, the attribution checker and the grounding audit used to scan those windows separately, up to nine tail passes per answer; one scan session per answer now serves all three, which is why the deterministic answer at 1,024,000 characters takes 45 ms instead of 268 ms with the same facts. Prompt preparation is 1.7–2.3× faster, most likely for a separate reason: a cheaper pre-filter in the directive detector described under poisoned corpora; that attribution was not measured in isolation. Correctness and citation support remain separate questions (Gao et al., EMNLP 2023).

Decoy-dense set (180 cases): the fact leads the answer in 130 of 180 cases (30 of 30 up to 128,000 characters, then 20, 10 and 10 of 30), a decoy number leads in 25 of 180 (English, middle and end positions from 256,000 characters), and the limit notice appears in 90 of 90 cases over the bound with no false positive. These lead-line counts are identical for the previous and the current code: a fact behind hundreds of matching paragraphs beyond the prefix is outside the sixteen windows the tail scan opens, and that is a disclosed, unchanged limit. A decoy number appears somewhere below the lead line in 115 of 180 answers, 13 more than before, because a line freed by duplicate collapse is filled from the next paragraph, which in this set is a decoy. Injected instructions in this set, in literal, fake-system, role and paraphrase form, are projected into prompt or answer in 0 of 144 cases; the role and paraphrase forms reached the prompt in 39 cases before the directive detector was extended.

3. Long sources: facts retained and limit notices per length, beginning, middle and end, German and English.

4. Answer path per source length, previous versus current code, 24 cases × 3 repeats per length, no model call or production IO.

5. Decoy-dense sources: fact versus decoy in the lead line per length, 30 cases per length.

Poisoned corpora: 5,952 trials

Following PoisonedRAG (Zou et al., USENIX Security 2025) and Greshake et al. (AISec 2023), 1, 5 or 20 attacker documents per targeted module are appended to the corpus: documents claiming to be the official source, clones with a new id and a changed number, documents stuffed with the module vocabulary, and documents carrying directives in five forms (English, two German forms, mixed, and a fake "System:" line with a detached payload sentence). Two negative controls: a clone the tenant must not see, and an exact benign duplicate. Success is counted separately as: attacker document among the four retrieved candidates, at rank 1, cited, its planted number adopted, its directive text projected into the prompt or the answer, and admitted although the tenant may not see it. BM25, dose 5, 96 questions per cell, before → after this round's mitigations (the "before" run used the code before the mitigations, with the earlier iterations of this round applied):

The admission gate excluded every foreign-membership clone (0 of 576 admitted); the 70 answers that were left without an admitted source because rejected clones consumed the four slots fell to 0 with production-faithful over-fetching. The directive detector now recognises German, paraphrased and role forms, a directive's neighbouring sentences are withheld only when they carry none of the question's words or are sentence fragments, and near-duplicates across different documents are collapsed into one citation (Broder, 1997; Bernstein and Zobel, CIKM 2005). Over all attack cells, both retrievers and all doses, directive text in the prompt fell from 947 to 0 and planted-number adoption from 1,760 to 1,120, with no cell worse on any metric. Authority and hub documents still take rank 1 and their planted number is adopted in up to 95 of 96 deterministic answers: ranking inside the admitted set is not defended, and a verbatim source-pointer answer that would restore the 9 missing Shortcuts citations stays off because it would cite the attacker's document when every admitted source is one. The benign-duplicate control expects an unchanged answer; in 79 of 96 answers (80 before) the answer text differs because the duplicate displaces one of the four admitted sources, and that count stands as a disclosed limit. A passing cell is a local diagnostic of authored attacks, not a security certification.

6. Poisoned corpora: planted numbers adopted per attack kind, BM25, dose 5, 96 questions, before and after; the indirect-instruction bar is the fake "System:" payload, the other four directive forms adopt nothing before or after.

Cold starts and concurrency

The harness runs ten independent interpreter processes per condition and reports pooled percentiles over their 960 requests, with the 95 % bootstrap interval of the mean per-process median as the uncertainty statement (Georges, Buytaert and Eeckhout, OOPSLA 2007), closed-loop threads with one simulated tenant each, a process pool, a large-tenant interference probe and a cross-tenant cache probe (Dean and Barroso, CACM 2013). The per-document statistics cache is now partitioned into one BM25 snapshot per tenant and listing (still bounded at 20,000 entries and 64 MiB of text per snapshot), verified by exact text, with scores identical cold and warm. The "before" column was measured on the code before this change, with the earlier iterations of this round applied. Apple M4 Max, CPython 3.11.9, no network, database or container contention.

A freshly started process answers its first query after about 4.1 s of Django setup plus the cold request; cold p99 is garbage-collection driven and stays above 0.1 s; thread numbers include the CPython global interpreter lock.

7. Cold start and concurrency: mean per-process medians with 95 % intervals (the table reports pooled percentiles, hence 24.8 and 21.5 there), and closed-loop throughput, before and after.

Context fixtures: source recall per retrieval method

This study was not re-run in the current round; the numbers and figure are those of the previous evidence bundle (9 fixture cases × 5 retrieval methods × 3 repeats = 135 executions; recall over the 5 annotated cases, fact presence over the 4 cases that carry a reference fact). Three graph hops recover every expected source and reference fact in these small authored fixtures. This is an integration diagnostic: literal fact presence does not establish entailment, and synthetic benchmark.invalid addresses are test identifiers.

8. Context fixtures: source recall over five annotated cases and literal fact presence over four cases. Does not measure semantic accuracy.

What source links establish

Every numeric source marker in the module replay resolves to an admitted source ID, with zero lost or unresolved citations across all 576 runs. Source-ID resolution is structural, not semantic citation support. Answer correctness, translation quality, support usefulness and process success remain unmeasured without independent assessment.

Peer-reviewed methodology

The design follows peer-reviewed work; every DOI in the evidence bundle's research record was verified through Crossref, and entries without a DOI link to their proceedings pages. The papers guide the measurements and do not certify SupraAgent.

• Robertson, Zaragoza and Taylor (CIKM 2004), Simple BM25 extension to multiple weighted fields: field weighting for the section title.

• Robertson and Zaragoza (2009), The Probabilistic Relevance Framework: BM25 and Beyond: the scoring model and its per-document statistics.

• Craswell et al. (WSDM 2008), An experimental comparison of click position-bias models: why the first listed source is measured separately.

• Lavrenko and Croft (SIGIR 2001), Relevance based language models: the module-coherence feedback step, implemented and switched off.

• Tao and Zhai (SIGIR 2007), An exploration of proximity measures in information retrieval: the proximity alternative that was considered and not adopted.

• Sakai (SIGIR 2006), Evaluating evaluation metrics based on the bootstrap and Smucker, Allan and Carterette (CIKM 2007), A comparison of statistical significance tests for information retrieval evaluation: paired bootstrap intervals and sign-flip tests over question families.

• Dwork et al. (Science 2015), The reusable holdout: why the holdout was sealed before tuning and evaluated once.

• Card, Robertson and Mackinlay (CHI 1991), The information visualizer: the 0.1-second perceptual threshold used to read local runtime.

• Liu et al. (TACL 2024), Lost in the Middle: evidence position in long contexts.

• Bai et al. (ACL 2024), LongBench: why character-length retention is not model comprehension.

• Gao et al. (EMNLP 2023), ALCE: answer quality and citation quality are separate.

• Amershi et al. (CHI 2019), Guidelines for Human-AI Interaction: making capability limits visible in the answer.

• Zou, Geng, Wang and Jia (USENIX Security 2025), PoisonedRAG and Greshake et al. (AISec 2023), Not What You've Signed Up For: the corpus-poisoning and indirect-injection stress test.

• Broder (SEQUENCES 1997), On the resemblance and containment of documents and Bernstein and Zobel (CIKM 2005), Redundant documents and search effectiveness: near-duplicate collapse across documents.

• Georges, Buytaert and Eeckhout (OOPSLA 2007), Statistically rigorous Java performance evaluation and Dean and Barroso (CACM 2013), The tail at scale: cold starts over independent processes and tail latency under concurrency.

• Lewis et al. (NeurIPS 2020), Retrieval-Augmented Generation: retrieval and generation are reported as separate stages.

• Yao et al. (ICLR 2025), tau-bench: process performance requires outcome checks against the resulting state, which this benchmark does not run.

Evidence, reproduction and limits

Reports, archived sources, raw answers, the start-of-round runs, figures, the review findings and reproduction commands are in the evidence bundle (GitHub access required). The bundle freezes source hashes before and after each run and is fingerprinted with SHA-256; an explicit gate of 537 tests passes on the measured code. Changes shipped only when they passed gates fixed before measuring: no routing regression, no lost citation, no attack cell worse, no long-source cell below the previous code. One long-source check, the decoy lead line, fails identically before and after and is carried as a limit. Two features that failed a gate are implemented but switched off and are reported in both states (coherence: hit@1 77 versus 91 of 96 with 9 lost citations; source pointer: attacker cited in 88 versus 96 of 96 under hub stuffing).

A product-level quality claim would additionally require independently assessed answers and citation support, fresh customer-domain questions, human evaluation of the user experience, production latency under load, defended ranking under authority and hub documents, and isolated workflow runs with verified before/after state. Until then this benchmark supports specific retrieval, retention, robustness and regression diagnostics. It does not establish general accuracy, successful live workflows or superiority to other agents.

Auf Deutsch lesen