# AGENTS.md — operating manual for emergence.fact.ngo

You are working in the mechanism repo of **emergence**, the live-events sensor net of
fact.ngo (a coherence.ngo subproject). Mission: continuously ingest news reports from
registered sources, distill each one through a fixed protocol (neutral framing, principle
isolation, topic mapping, geo tags) with a cheap Cloudflare Workers AI model, and roll the
results into fractal day-level aggregates the website renders as "many lenses."

This repo is the mechanism (sources registry, prompts, schemas, scripts). Data never
lives here — it lives in the `emergence-data` repo, which is append-only.

## The two lanes

- **unverified** (active). The second-order heap: every ingested report is machine-
  distilled, onboarded, and labeled `status: unverified` at birth. It has some validity
  (it comes from real newsrooms) but no verified status. It is never presented as fact.
- **verified** (built, inactive). The fact-checking pipe: an AI council (glm-5.3 seats,
  multi-round deliberation) that turns a claim into a verdict record. `council.js` refuses
  to run without `--activate`. Nothing feeds the verified lane yet; the site renders it
  as in-development.

## The pipeline (one cycle)

```bash
# everything, in order (what cron runs):
node scripts/pipeline.js --commit          # scrape -> distill -> cluster -> aggregate -> gis, then commit data repo

# or stage by stage:
node scripts/scrape.js                     # fetch sources -> raw/<date>/articles.jsonl
node scripts/distill.js                    # new articles -> distilled/<date>/essence.jsonl (spends money)
node scripts/cluster.js                    # essences -> events + facets (spends money; dedupe layer)
node scripts/aggregate.js                 # fractal topics + day rollup -> aggregates/<date>/
node scripts/gis.js                        # country points     -> aggregates/<date>/gis.json

# the inactive lane:
node scripts/council.js --claim "..." --activate
```

The event layer: every ok essence belongs to exactly one event; the day portrait ranks
topics by summed event weight (corroboration x complexity), while raw report counts stay
visible. Duplicate detection is two-phase — deterministic similarity candidates, then
AI resolution into events + facets — and failures degrade to the deterministic grouping,
never to data loss.

Data repo: `../emergence-data` (override with `EMERGENCE_DATA`). Days are UTC.
One commit per pipeline run; push after every run.

## Hard rules

1. **Never touch credentials.** `scripts/lib.js` resolves the Cloudflare token (env or
   `~/.local/share/opencode/auth.json`). Never print, copy, or commit tokens.
2. **The engine stores no data.** Everything the pipeline writes goes to `emergence-data`.
   Never edit records in the data repo by hand; regeneration flows through these scripts.
3. **Budget guard is law.** distill stops when the day's ledger reaches the cap
   (default $1.00/day; `scripts/config.json`). Never bypass it with a loop.
   Distill model: `@cf/zai-org/glm-5.3-flash`. Council model: `@cf/zai-org/glm-5.3`.
4. **Neutral framing only.** The distillation prompt forbids editorializing, speculation,
   and asserting unverified specifics. Parse failures are recorded, never silently
   retried into different content; a deliberate retry runs only with `--retry`.
5. **Unverified is a property of the record.** `status: "unverified"` is set at birth by
   the mechanism. The only script that may write `status: "verified"` is `council.js`,
   and only while activated.
6. **Sources are teaser-depth.** We take feeds at their offered depth. No full-text
   scraping, no bypassing robots, paywalls, or bot-gates, no third-party relays whose
   terms forbid non-personal use. Reuters is registered but deferred (feeds retired,
   site bot-gated) until a legitimate route exists. Add or change sources only in
   `sources/sources.yaml`.
7. **Dedupe is content-addressed** (hash of source + normalized title). Never force-add.
8. **Uptime over perfection.** A failed source is recorded as failed and the cycle
   continues; a failed distill is recorded as a failure record. Honest gaps, no fillers.
9. Commit messages: imperative, one line ("Cycle 2026-09-29: 61 articles, 58 essences").
10. Push both repos to their the self-hosted remote bare remotes after every finished work unit
    (`remote:remotes/fact.ngo/emergence.git`, `remote:remotes/fact.ngo/emergence-data.git`).

## Environments

- Mono-folder: `~/Documents/fact.ngo/emergence` (Mac), `~/coherence/fact.ngo/emergence`
  (the self-hosted remote). The data repo sits next to it.
- The live cycle runs on **Cloudflare** (Worker `emergence-net`, cron every 2h,
  R2 as live primary). the self-hosted remote and the Mac are dev/archive machines only:
  archive sync via `scripts/sync-from-r2.sh`, local runs via the Node scripts
  (same pipeline cores). Never install the old the self-hosted remote cycle cron.
- The site consumes only `emergence-data/aggregates/` and `distilled/`; cross-repo
  effects go through the build, never through shared files.