# AI search study: technical methods and evidence

Evidence cutoff: November 13, 2026. The matched study has not run. These are proposed methods, historical limitations and source notes, not measured current results.

## Research workflow described in the article

The article describes product capabilities, not an observed trace from the example Alzheimer's research prompt. That prompt is an illustration and is excluded from the scored cohort. Search tools, source-reading access and citation links do not guarantee that an answer found every relevant paper or correctly interpreted it.

The paper-search tool supports relevance search over indexed PubMed records and open-access PubMed Central passages, with publication-year and publication-type filters. Its separate systematic mode runs exact PubMed and Europe PMC queries and saves search logs and results. Full-text access depends on the source. Libraries and citation grounding let researchers work with their own documents and inspect supporting passages.

These descriptions were verified at product revision `bd551407d87ea11124bb5ed789ea164980694334`:

- [Paper-search modes, filters and full-text fields](https://github.com/arcainc/arca/blob/bd551407d87ea11124bb5ed789ea164980694334/apps/web/lib/ai/tools/search-papers.ts)
- [Libraries and source grounding](https://github.com/arcainc/arca/blob/bd551407d87ea11124bb5ed789ea164980694334/docs/content/architecture/systems/knowledge.md)

## Coverage receipt

The machine-readable evidence at `/data/ai-search-study.json` contains the exact read-only SQL, query hash, frozen counter receipt and historical chart aggregates.

Snapshot: `2026-11-13T16:04:21.525843Z`.

| Maintained counter | Value |
| --- | ---: |
| Mirrored public-source record rows | 104,284,404 |
| Indexed chunks | 176,710,718 |
| Total chunks | 176,807,937 |
| Nondeleted source families | 125 |
| Enabled source families | 123 |

These are maintained metadata counters, not a fresh recount, unique-paper count or live availability audit. Nondeleted sources and namespaces are included, whether enabled or disabled. Each source joins one namespace through its primary key, so the join does not multiply source rows. Future execution produces a new snapshot, not the frozen values above.

The separate website-directory census found 227 entries, 225 unique source names, 211 distinct domain strings and 14 editorial categories. It counts entries in the source arrays, not metadata such as `expectedDomainCount`. Configured provider routes and domain filters are not a census of upstream holdings. Provider collections overlap and have not been added together.

## Historical regulatory results

LifeSci Diamond's November 4 release has 55 tasks per system in each requested-model pair. Strict passes require every positive rubric criterion fully met and no penalty. Weighted partial credit is a different measure. The published paired differences and 95% intervals are preserved in the article, chart and JSON download.

The plain comparison received task documents and six Exa results, each capped at 2,500 characters. It could not read full external pages or PDFs. Arca used its chat workflow and life-sciences tools. One of five Arca Opus 5 strict passes routed through Opus 5.5 and GPT-6.1 Sol. Complete per-step routing is unavailable, so requested models do not guarantee same-model execution.

The release used GPT-6.1 Sol as judge at medium effort. Complete generation-time reasoning effort, temperature, top-p, seed and output limits were not recorded. Gateway caching was `auto`.

The release contains 1,210 graded answers across 22 systems. Of these, 102 were separately dated new answers replacing missing or ungradeable archive cells. A final unavailable count of zero does not imply a simultaneous fresh run without failures. The older 35-task shared archive intersection is not the denominator of this 55-task chart.

Task review is uneven: 26 historically reviewed, 18 contributor-only and 11 never reviewed. Only 17 meet the later ready-and-reviewed diagnostic standard. Independent regulatory-expert approval of the revised grading policy remains outstanding. Eight tasks were developed around plain-model over-filing/category errors. Original generation-time library and rubric hashes were not recovered. This is not a representative random sample of regulatory research.

Recorded generation costs exclude other serving components. Arca's recorded generation cost was higher in the Opus 5.5 pair. This evidence does not establish total-cost savings or general search superiority.

## Historical literature results

ResearchQA used 50 questions, 377 rubric criteria and an AI jury of GPT-6 Astra, Claude Opus 5 and Gemini 3.1 Pro Preview. The score measures rubric adherence and completeness, not independently validated factual accuracy. Scientific key review is partial.

| Configuration | Date | Score /100 | Routing or instruction change |
| --- | --- | ---: | --- |
| Production Auto | November 4, 2026 | 74.48 | 28 Luna, 21 Terra, 1 Sol; no pinned tier |
| E1 evidence/completeness instruction | November 5, 2026 | 85.17 | Appended instruction; 43 Terra, 6 Sol, 1 Gemini 4 Argon |
| E2 Max | November 5, 2026 | 84.50 | Unchanged questions; all 50 Sol |
| E3 instruction plus Max | November 5, 2026 | 89.435 | Appended instruction; 48 Sol, 2 Gemini 4 Argon |
| E4 later Auto | November 6, 2026 | 81.44 | Changed routing/prompt; 47 Terra, 3 Luna |

Each row represents one attempt per question in a different configuration. Competitor answers were not rerun alongside every Arca change. One rubric line had two responding jurors rather than three. Failed and recovery judging calls counted toward spend. Missing final-token timing and unreconciled tool/infrastructure fees prevent a speed or total-cost comparison.

Medical LitQA3 varied Paperclip access inside Arca: a component ablation, not Arca versus standard web search. Legal primary-source cases, LegalBench configurations and reasoning-effort diagnostics do not supply a completed current legal-search comparison.

## Proposed matched pilot

Use 20 newly authored questions, five per domain, with two systems and three fresh repeats: 120 planned attempts, 60 per system, over 20 independent question clusters. Questions and source-validated scoring keys are not yet written. Select from independent publisher, regulator and court inventories, not only questions or mirrored sources Arca already handles. Licenses must permit the intended use. Keep protected questions, answer keys, customer content and raw captures out of public releases.

Proposed root model: `openai/gpt-6.1-sol`, medium reasoning effort, in both systems. Provider route and model revision remain unfrozen. Confirm supported controls, disable Auto routing and root-model fallback, and retain returned model identifiers if only a mutable alias is available.

Keep task instructions, answer format, clock date, general-web provider and full webpage/PDF reader policy identical. Arca adds its declared specialized search tools. Disable private libraries, customer connectors and unrelated action tools. Record differing tool descriptions. Optional paid tools, including Paperclip, require rights, prices and telemetry; declare any exclusions before the run.

| Common limit | Proposed value |
| --- | --- |
| Root-model steps | 12 maximum |
| Visible characters per full read | 20,000; counted pagination allowed |
| Evidence characters per attempt | 120,000 Unicode characters |
| Root input tokens per request | 64,000 maximum |
| Total billed root input tokens | 500,000 maximum |
| Total billed root output and reasoning tokens | 32,000 maximum |
| Final visible answer tokens | 4,000 maximum |
| Deadline | 600 seconds after accepted submission |

These limits are not a spending cap. Internal retrieval, auxiliary models and retries must be bounded, priced and included in the configuration and cost ledger before execution. No paid execution is authorized by this protocol.

Use fresh chats without deliberate task prewarming or shared answers/search payloads between systems. Log unavoidable provider and mirror caches. Randomize system order within each question/repeat pair and run each pair close in time. Freeze code, instructions, source cutoffs, reader versions, tools and schedule. Record corpus receipts or source hashes where a live index cannot be frozen. Preflight full-page/PDF readers on unrelated fixtures before the scored run.

### Scoring and failure accounting

Two blinded domain experts validate claims and required evidence, with disagreements adjudicated. Fixed GPT-6.1 Sol and Claude Opus 5.5 judges may assist using independent curated source packets and recorded supported controls; they do not replace human validation.

Accuracy is the fraction of adjudicated material claims that are correct and supported. Disclose claim counts and answer coverage. Completeness is weighted coverage of frozen applicable evidence items. Report question-level averages and pooled item counts. Citation resolution and quotation matching are separate diagnostics. Track abstention and required-item coverage so short or empty answers do not appear successful.

Useful task completion is a separate measure: answer delivered by the deadline, all critical items, at least 80% weighted supported coverage, no material error, and appropriate scope or abstention. Its denominator includes all planned attempts. It is not retrieval accuracy.

Retain all 120 planned IDs. Distinguish unstarted, cap-stopped, complete, partial, timeout, tool/provider failure, generation failure, unjudged and invalid-key outcomes. Missing answers have no factual-accuracy score. Grade partial answers separately. Report judged/complete/submitted/planned denominators, unresolved counts and missing-outcome bounds. Do not discard failed attempts or score judge failures as false answers.

Measure submission to first token and final complete answer with a monotonic clock, including queueing and validation. Report completed-answer median and p95, completion rates and right-censored 600-second timeouts.

Total serving cost includes root/helper models, searches, full reads, reranking, embeddings, cache categories, retries, failed calls and disclosed infrastructure/index allocations. Unknown charges remain unknown. Study judging and expert labor are separate. Do not compare subscription allocations against model-only spend.

Retain every repeat in paired question clusters. Use 10,000 domain-stratified cluster-bootstrap resamples with seed `20261113` and percentile 95% intervals. Conditional quality differences use paired adjudicated repeats only, with coverage disclosed. Show repeat spread rather than a best run. Five questions per domain cannot support broad superiority claims. A fresh confirmatory set is required after the pilot.

### Before execution and publication of results

The pilot requires licensed frozen questions, independently validated keys, reader parity, frozen model/provider settings, approved spending and enforceable nested limits. Publish results only after complete paired timing/cost ledgers, blinded scoring and disclosure review. Only aggregates and approved licensed examples belong in the public release.

## Source notes

Website evidence revision: `fe6bb4e0105cdff76ff12b071406b38c358879e3`. Product evidence revision: `bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4`. Operational schema revision: `bd551407d87ea11124bb5ed789ea164980694334`.

LifeSci Diamond aggregate: `2026-11-03-gdp-v6`, released November 4, 2026, source fingerprint `ea23abfc202177b563475b30e0b9a996ae160a317ed4654b7a4704181784b0f7`. Its file hash is recorded in the JSON download. Private repository links are reviewer provenance, not a public release of protected evidence.

- [Website directory](https://github.com/arcainc/science-website/blob/fe6bb4e0105cdff76ff12b071406b38c358879e3/lib/data-sources.ts)
- [LifeSci Diamond aggregate](https://github.com/arcainc/science-website/blob/fe6bb4e0105cdff76ff12b071406b38c358879e3/content/data/lifesci-diamond.json)
- [Public LifeSci Diamond run notes](https://github.com/arcainc/lifesci-diamond/blob/main/RUN_NOTES.md), mutable; the article uses the frozen aggregate above.
- [Regrade methods and limitations](https://github.com/arcainc/arca/blob/bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4/evals/reg_affairs_bench/publish/regrades/2026-11-03/README.md), an older incomplete archive rather than the 55-task chart denominator.
- [ResearchQA production survey](https://github.com/arcainc/arca/blob/bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4/evals/life_science_lit_bench/baseline/production-survey-2026-11-04/README.md)
- [ResearchQA configuration experiments](https://github.com/arcainc/arca/blob/bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4/evals/life_science_lit_bench/baseline/survey-improvement-experiments-2026-11-05/README.md)
- [LitQA3 report](https://github.com/arcainc/arca/blob/bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4/evals/litqa3_search_medical/README.md)
- [Legal primary-source runner](https://github.com/arcainc/arca/blob/bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4/evals/legal_primary_source/run.ts), [LegalBench runner](https://github.com/arcainc/arca/blob/bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4/evals/legalbench/run.py), and [reasoning-effort diagnostics](https://github.com/arcainc/arca/blob/bb65bdf0f1320d0f344b0f1ef68a301e9dfd01a4/evals/chat_effort/README.md)