AGENTS.md
# AGENTS.md — operating manual for emergence.fact.ngo
You are working in the mechanism repo of **emergence**, the live-events sensor net of
fact.ngo (a coherence.ngo subproject). Mission: continuously ingest news reports from
registered sources, distill each one through a fixed protocol (neutral framing, principle
isolation, topic mapping, geo tags) with a cheap Cloudflare Workers AI model, and roll the
results into fractal day-level aggregates the website renders as "many lenses."
This repo is the mechanism (sources registry, prompts, schemas, scripts). Data never
lives here — it lives in the `emergence-data` repo, which is append-only.
## The two lanes
- **unverified** (active). The second-order heap: every ingested report is machine-
distilled, onboarded, and labeled `status: unverified` at birth. It has some validity
(it comes from real newsrooms) but no verified status. It is never presented as fact.
- **verified** (built, inactive). The fact-checking pipe: an AI council (glm-5.3 seats,
multi-round deliberation) that turns a claim into a verdict record. `council.js` refuses
to run without `--activate`. Nothing feeds the verified lane yet; the site renders it
as in-development.
## The pipeline (one cycle)
```bash
# everything, in order (what cron runs):
node scripts/pipeline.js --commit # scrape -> distill -> cluster -> aggregate -> gis, then commit data repo
# or stage by stage:
node scripts/scrape.js # fetch sources -> raw/<date>/articles.jsonl
node scripts/distill.js # new articles -> distilled/<date>/essence.jsonl (spends money)
node scripts/cluster.js # essences -> events + facets (spends money; dedupe layer)
node scripts/aggregate.js # fractal topics + day rollup -> aggregates/<date>/
node scripts/gis.js # country points -> aggregates/<date>/gis.json
# the inactive lane:
node scripts/council.js --claim "..." --activate
```
The event layer: every ok essence belongs to exactly one event; the day portrait ranks
topics by summed event weight (corroboration x complexity), while raw report counts stay
visible. Duplicate detection is two-phase — deterministic similarity candidates, then
AI resolution into events + facets — and failures degrade to the deterministic grouping,
never to data loss.
Data repo: `../emergence-data` (override with `EMERGENCE_DATA`). Days are UTC.
One commit per pipeline run; push after every run.
## Hard rules
1. **Never touch credentials.** `scripts/lib.js` resolves the Cloudflare token (env or
`~/.local/share/opencode/auth.json`). Never print, copy, or commit tokens.
2. **The engine stores no data.** Everything the pipeline writes goes to `emergence-data`.
Never edit records in the data repo by hand; regeneration flows through these scripts.
3. **Budget guard is law.** distill stops when the day's ledger reaches the cap
(default $1.00/day; `scripts/config.json`). Never bypass it with a loop.
Distill model: `@cf/zai-org/glm-5.3-flash`. Council model: `@cf/zai-org/glm-5.3`.
4. **Neutral framing only.** The distillation prompt forbids editorializing, speculation,
and asserting unverified specifics. Parse failures are recorded, never silently
retried into different content; a deliberate retry runs only with `--retry`.
5. **Unverified is a property of the record.** `status: "unverified"` is set at birth by
the mechanism. The only script that may write `status: "verified"` is `council.js`,
and only while activated.
6. **Sources are teaser-depth.** We take feeds at their offered depth. No full-text
scraping, no bypassing robots, paywalls, or bot-gates, no third-party relays whose
terms forbid non-personal use. Reuters is registered but deferred (feeds retired,
site bot-gated) until a legitimate route exists. Add or change sources only in
`sources/sources.yaml`.
7. **Dedupe is content-addressed** (hash of source + normalized title). Never force-add.
8. **Uptime over perfection.** A failed source is recorded as failed and the cycle
continues; a failed distill is recorded as a failure record. Honest gaps, no fillers.
9. Commit messages: imperative, one line ("Cycle 2026-09-29: 61 articles, 58 essences").
10. Push both repos to their the self-hosted remote bare remotes after every finished work unit
(`remote:remotes/fact.ngo/emergence.git`, `remote:remotes/fact.ngo/emergence-data.git`).
## Environments
- Mono-folder: `~/Documents/fact.ngo/emergence` (Mac), `~/coherence/fact.ngo/emergence`
(the self-hosted remote). The data repo sits next to it.
- The live cycle runs on **Cloudflare** (Worker `emergence-net`, cron every 2h,
R2 as live primary). the self-hosted remote and the Mac are dev/archive machines only:
archive sync via `scripts/sync-from-r2.sh`, local runs via the Node scripts
(same pipeline cores). Never install the old the self-hosted remote cycle cron.
- The site consumes only `emergence-data/aggregates/` and `distilled/`; cross-repo
effects go through the build, never through shared files.