WP-05 — Market Intelligence Super-Swarm: 10-Pass Pipeline

The Market Intelligence Super-Swarm is the pipeline that produces structured intelligence reports inside the Treeova platform. It is designed around three commitments: hermetic isolation per pass, quality gating between passes, and recursive webhook orchestration so the pipeline never depends on a single long-lived process.

A single, monolithic prompt that "does everything" is hard to debug, hard to retry, and impossible to gate selectively. The Super-Swarm decomposes the work into ten passes, each with a narrow contract — for example: source acquisition, semantic deduplication, classification, regime contextualization, scenario reasoning, conviction scaffolding, and final synthesis.

The decomposition makes each pass replaceable, individually testable, and individually quality-gated. A weak intermediate result becomes a repairable defect instead of a silent contaminant in the final report.

Each pass runs in isolation. It receives only the structured inputs it is contractually allowed to see, returns only the structured output it is contractually expected to produce, and has no access to the runtime state of any other pass. Cross-pass coupling — the source of most pipeline regressions — is structurally prevented.

This isolation is what makes the pipeline auditable. Any pass can be replayed against its recorded inputs and compared to its recorded outputs, without standing up the rest of the pipeline.

Every pass produces, alongside its structured output, a self-evaluated quality score. A pass is considered complete only when its score clears the configured threshold. Passes that fall below the threshold are retried or repaired before downstream passes are permitted to consume them.

Quality gating is the mechanism that prevents low-confidence intermediate output from silently degrading the final report. The downstream conviction methodology (see WP-01) consumes only quality-gated intelligence.

When a pass errors out or fails its quality gate, the orchestrator triggers a targeted recovery routine for that pass — not for the whole pipeline. State already produced by upstream passes is preserved in the run record, so recovery resumes from the failure point rather than from the beginning.

Rather than holding a single process open for the entire 10-pass run, each pass invokes the next via webhook callback. State lives in the database between hops, identified by a stable run record.

This pattern has three practical benefits. It avoids long-lived process limits and timeouts. It gives every pass a clean execution boundary, with its own logs and resource budget. And it lets the pipeline be paused, resumed, or audited at any pass boundary without bespoke infrastructure.

Source acquisition routinely returns near-duplicates from overlapping providers. A dedicated deduplication pass uses semantic similarity to collapse these near-duplicates into a single canonical item with merged provenance, before any downstream pass reasons over the corpus. The similarity threshold itself is withheld, but the architectural commitment — dedup before reasoning — is public and stable.

• The per-pass prompt strings.
• The specific model assigned to each pass.
• The exact quality scoring rubric.
• The semantic deduplication similarity threshold.
• The orchestrator's retry and backoff policies.

• The Super-Swarm produces intelligence reports, not trading advice. Downstream consumers — including the conviction methodology — apply their own evaluation before any report influences a decision.
• Quality gating prevents below-threshold passes from being consumed; it does not guarantee the threshold itself is the right one for every market regime. The pipeline is conservative by design, which can produce more retries during volatile regimes.
• Recursive webhook orchestration depends on the database as a shared substrate; database degradation degrades the pipeline.
• Source quality is upstream of the pipeline. The Super-Swarm can improve a noisy corpus through dedup and gating, but it cannot create signal that was never present in its inputs.