This document describes how WoT Oracle discovers follow relationships and relays, how it determines the source of truth for follow lists, and how it batches requests for efficient network usage.
- Overview
- Current Implementation
- Proposed Sync Architecture
- Relay Discovery Strategy
- Follow Event Resolution
- Batched Subscriptions
- Progressive Depth Crawling
- Data Flow Diagrams
- Profile Ingestion (kind:0)
- Configuration
WoT Oracle builds a global follow graph by ingesting kind:3 (contact list / NIP-02) events from the Nostr network and caches user profile metadata by ingesting kind:0 (profile / NIP-01) events. The core challenges are:
- Finding follow lists — Where do we fetch a user's kind:3 event from?
- Relay discovery — How do we know which relays a user publishes to?
- Source of truth — How do we ensure we have the latest follow list, not a stale one?
- Efficiency — How do we minimize relay connections and redundant requests?
| Aspect | Status | Details |
|---|---|---|
| Kind:3 ingestion | ✅ | Subscribes to kind:3 events across configured relays |
| Kind:0 ingestion | ✅ | Subscribes to kind:0 events for profile caching alongside kind:3 |
| Deduplication | ✅ | LRU cache (100k entries) by pubkey bytes, skips older events |
| Timestamp validation | ✅ | Graph only accepts events newer than the stored created_at |
| Batch persistence | ✅ | 100 events per SQLite transaction, 5-second flush timeout |
| Batch API queries | ✅ | /distance/batch supports up to 100 targets per request |
| Aspect | Status | Impact |
|---|---|---|
| NIP-65 relay discovery | ❌ | Can't find user-specific relays; misses events from users on non-default relays |
| Relay hints from p-tags | ❌ | Kind:3 p-tags contain relay URLs at position 2 — currently discarded |
| Multi-source verification | ❌ | Takes first event seen; no waiting for multiple relays to confirm latest |
| Dynamic relay connections | ❌ | Only connects to statically-configured relays from RELAYS env var |
| Outbox model | ❌ | No NIP-65-aware routing; all relays queried uniformly |
| Indexer API usage | ❌ | No use of nostr.band API or purplepag.es for bootstrapping |
Static Relays (env var)
│
▼
Subscribe kind:3 (all authors, no filter)
│
▼
Event arrives ──▶ LRU dedup check ──▶ Parse p-tags ──▶ Graph update ──▶ Persist queue
(skip if older) (pubkey only) (if newer) (batch of 100)
Problem: This is a passive, firehose approach. We connect to 3 relays and hope to see every user's latest kind:3 event flow by. Users who publish only to relays we don't monitor are invisible.
- Respect user sovereignty — Users choose their relays via NIP-65. Honor those choices.
- Outbox model — To read a user's events, connect to their write relays.
- Multiple sources — Never trust a single relay for "latest." Wait for corroboration.
- Progressive discovery — Start from seed relays, discover more relays as we crawl.
- Batch aggressively — Group users by relay, subscribe in bulk, minimize connections.
- Degrade gracefully — If NIP-65 is unavailable, fall back to relay hints, then seed relays.
┌──────────────────────────────────────┐
│ Sync Orchestrator │
└──────────┬───────────────────────────┘
│
┌────────────────┼────────────────────┐
▼ ▼ ▼
┌────────────────┐ ┌────────────┐ ┌───────────────────┐
│ Relay Discovery│ │ Follow │ │ Event Resolution │
│ Service │ │ Fetcher │ │ Service │
└────────────────┘ └────────────┘ └───────────────────┘
│ │ │
▼ ▼ ▼
┌────────────────┐ ┌────────────┐ ┌───────────────────┐
│ Relay Routing │ │ Batched │ │ Multi-Source │
│ Table │ │ Subs Pool │ │ Conflict Resolver│
└────────────────┘ └────────────┘ └───────────────────┘
To find where a user publishes, we use multiple sources in order of reliability:
| Priority | Source | Event Kind | How |
|---|---|---|---|
| 1 | NIP-65 relay list | kind:10002 |
Author-signed declaration of read/write relays. Canonical source. |
| 2 | Kind:3 p-tag relay hints | kind:3 |
Relay URL at position 2 of ["p", pubkey, relay_url, petname]. Hints from the follower about where they find this user. |
| 3 | Seed/indexer relays | — | Bootstrap relays like purplepag.es, relay.nostr.band that aggregate events from across the network. |
NIP-65 events declare a user's relay preferences:
{
"kind": 10002,
"tags": [
["r", "wss://relay.damus.io"],
["r", "wss://my-relay.com", "write"],
["r", "wss://inbox.nostr.com", "read"]
]
}- No marker = both read and write
"write"= outbox — user publishes here (fetch their events from here)"read"= inbox — user reads here (send them mentions/DMs here)
For WoT Oracle, we care about write relays — that's where we'll find a user's kind:3 event.
Kind:3 p-tags can carry relay hints:
["p", "ab12...cd34", "wss://relay.example.com", "alice"]Position 2 (wss://relay.example.com) is a hint from the follow-list author about where to find the followed user's events. These are less authoritative than NIP-65 (they're from a third party) but widely available.
Current code discards these. The ingestion should extract and store them as fallback relay hints.
These special-purpose relays aggregate data from across the network:
| Relay | Purpose | Event Kinds |
|---|---|---|
wss://purplepag.es |
Directory relay for relay discovery | kind:0, kind:10002 only |
wss://relay.nostr.band |
Full indexer relay | All kinds, supports NIP-45 COUNT |
Usage:
- Query
purplepag.esforkind:10002events to bootstrap relay discovery - Query
relay.nostr.bandforkind:3events as a fallback aggregator - These should be used as bootstrap sources, not as the sole data source
The system should maintain an in-memory routing table:
pubkey → RelaySet {
write_relays: Vec<String>, // From NIP-65 (outbox)
read_relays: Vec<String>, // From NIP-65 (inbox)
hint_relays: Vec<String>, // From kind:3 p-tag hints
source: RelaySource, // How we discovered these
updated_at: u64, // When this entry was last refreshed
}
Lookup order:
- Check routing table for user's write relays
- If missing → query
purplepag.es+relay.nostr.bandforkind:10002 - If still missing → use hint relays from kind:3 p-tags
- If still missing → use seed relays (configured defaults)
Kind:3 is a replaceable event — only the latest version per pubkey is valid. But relays don't gossip among themselves. Relay A may have a version from January while Relay B has the real latest from March.
For each user whose kind:3 we want:
- Query multiple relays in parallel with
{ kinds: [3], authors: [pubkey], limit: 1 } - Collect responses with a timeout window (e.g., 3-5 seconds, or until N relays respond)
- Select the winner using NIP-01 conflict resolution:
- Primary: highest
created_attimestamp - Tiebreaker: lowest event
id(lexicographic)
- Primary: highest
- Update graph only with the winning event
Relay A ──────┐
│ ┌──────────────┐ ┌────────────────┐
Relay B ──────┼────▶│ Collect │────▶│ Select latest │────▶ Graph Update
│ │ (timeout) │ │ (created_at) │
Relay C ──────┘ └──────────────┘ └────────────────┘
Wait for:
- All relays respond, OR
- 3-5 second timeout, OR
- N responses received
The current implementation processes events immediately as they arrive. This means:
- If a stale relay responds first, we accept a stale follow list
- When the real latest arrives later, we update again (wasted work)
- In the worst case, we might miss the latest entirely if it's on a relay we don't monitor
Waiting for multiple sources:
- Increases confidence we have the actual latest event
- Reduces unnecessary graph updates from stale events
- Worth the latency cost at sync time (queries can still be served from cache/existing graph)
After initial sync, the system should still process kind:3 events as they stream in (the current firehose approach), but with the same timestamp-based conflict resolution. The waiting strategy is primarily for:
- Initial bootstrap
- Targeted re-fetches of specific users
- Periodic freshness checks
Subscribing to kind:3 for each user individually creates N subscriptions. With 500k users, that's unmanageable.
Single subscription:
{ kinds: [3, 10002], authors: [seed_pubkey] }
→ Sent to: purplepag.es + seed relays
→ Wait: until responses from ≥2 relays or 5s timeout
Batch subscription (grouped by relay):
For each relay R that multiple followed users publish to:
{ kinds: [3], authors: [pubkey1, pubkey2, ..., pubkeyN] }
→ Sent to: R
→ Wait: until responses from ≥2 relays per user or 10s timeout
If user has no known relays:
{ kinds: [3, 10002], authors: [pubkeyA, pubkeyB, ...] }
→ Sent to: purplepag.es + relay.nostr.band
Same as Level 1 but with more users. The relay routing table is now populated, so more requests go to user-specific relays rather than indexers.
To minimize connections, group users by their write relays:
Step 1: Collect all pubkeys needing kind:3 fetch
Step 2: Look up each pubkey's write relays in routing table
Step 3: Group: relay_url → [pubkey1, pubkey2, ...]
Step 4: For each group, create one subscription with all pubkeys
Step 5: Subscribe, collect responses, resolve conflicts
Example:
relay.damus.io → [alice, bob, dave] → 1 subscription, 3 authors
nos.lol → [alice, carol, eve] → 1 subscription, 3 authors
my-relay.com → [bob] → 1 subscription, 1 author
Alice appears on two relays — we'll get her kind:3 from both, then select the latest.
Most relays limit authors per filter to ~1000. If a batch exceeds this:
- Split into chunks of 500-1000 authors
- Send as separate subscriptions to the same relay
The first level (seed user's follows) takes the longest because:
- We don't know anyone's relays yet
- We must bootstrap via indexer relays
- We must wait for multi-source verification
Each subsequent level is faster because:
- The relay routing table is populated from previous levels
- Relay hints from kind:3 p-tags are available
- Subscriptions are better targeted (fewer indexer queries)
Level 0: Fetch seed user's kind:3 + kind:10002
├── Extract follows (e.g., 500 pubkeys)
├── Extract relay hints from p-tags
└── Store in relay routing table
│
Level 1: For seed's 500 follows:
├── Batch-fetch kind:10002 from purplepag.es
├── Update relay routing table
├── Batch-fetch kind:3 grouped by relay
├── Wait for multi-source verification
├── Extract their follows (~50k unique pubkeys)
└── Extract relay hints from their p-tags
│
Level 2: For ~50k unique pubkeys:
├── Many already have relay info (from Level 1 hints)
├── Batch-fetch kind:10002 for unknowns
├── Batch-fetch kind:3 grouped by relay
└── Continue...
| Level | Users | Expected Duration | Why |
|---|---|---|---|
| 0 | 1 | 2-5 seconds | Single user, wait for multiple relays |
| 1 | ~500 | 30-60 seconds | Relay discovery needed, batch subscriptions, multi-source wait |
| 2 | ~50k | 5-15 minutes | Most relay info known, large batches, some unknowns |
| 3+ | ~500k | 15-30 minutes | Routing table well-populated, mostly targeted fetches |
After initial sync, the continuous firehose subscription maintains freshness.
┌─────────────────────────────────────────────────────────────────────────┐
│ Sync Orchestrator │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ │
│ │ 1. Bootstrap │ Connect to seed relays + purplepag.es │
│ │ Relays │ Subscribe kind:10002 for target pubkeys │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 2. Build Relay │ Parse kind:10002 r-tags (read/write) │
│ │ Routing │ Store write relays per pubkey │
│ │ Table │ Fall back to p-tag hints if no kind:10002 │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 3. Batch Fetch │ Group pubkeys by write relay │
│ │ kind:3 │ Subscribe { kinds:[3], authors:[...] } │
│ │ Events │ per relay │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 4. Multi-Source │ Collect kind:3 from multiple relays per user │
│ │ Resolution │ Wait for timeout or N responses │
│ │ │ Select highest created_at (lowest id tiebreak) │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 5. Process & │ Parse p-tags → extract follows + relay hints │
│ │ Extract │ Update relay routing table with hints │
│ │ │ Update in-memory graph (diff-based) │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 6. Batch │ Queue updates → batch of 100 per transaction │
│ │ Persist │ 5-second flush timeout │
│ └────────┬─────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ 7. Next Level │ Enqueue newly discovered pubkeys │
│ │ Crawl │ Repeat from step 2 for next depth level │
│ └──────────────────┘ │
│ │
│ ┌──────────────────┐ │
│ │ 8. Continuous │ After initial crawl, maintain firehose │
│ │ Firehose │ subscription for real-time updates │
│ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Need events from user X
│
▼
┌─────────────┐
│ Check relay │
│ routing │──── Found write relays ────▶ Use them
│ table │
└──────┬──────┘
│ Not found
▼
┌─────────────┐
│ Query │
│ purplepag.es│──── Found kind:10002 ──────▶ Parse r-tags
│ + nostr.band│ Update routing table
└──────┬──────┘ Use write relays
│ Not found
▼
┌─────────────┐
│ Check p-tag │
│ relay hints │──── Found hints ───────────▶ Use hint relays
└──────┬──────┘
│ Not found
▼
┌─────────────┐
│ Use seed │──── Always available ──────▶ Use configured defaults
│ relays │
└─────────────┘
| Variable | Default | Description |
|---|---|---|
RELAYS |
wss://relay.damus.io,wss://nos.lol,wss://relay.primal.net/,wss://relay.mostr.pub/ |
Seed relays for initial connections |
| Variable | Default | Description |
|---|---|---|
DISCOVERY_RELAYS |
wss://purplepag.es,wss://relay.nostr.band |
Relays queried for kind:10002 (relay lists) |
SYNC_TIMEOUT_SECS |
5 |
How long to wait for multi-source verification per batch |
SYNC_MIN_SOURCES |
2 |
Minimum relay responses before accepting an event |
SYNC_BATCH_SIZE |
500 |
Max authors per subscription filter |
SYNC_MAX_DEPTH |
3 |
How many levels deep to crawl from seed |
SEED_PUBKEY |
(none) | Starting pubkey for targeted WoT crawl |
| Aspect | Current | Proposed |
|---|---|---|
| Relay selection | Static 3 relays | Dynamic per-user via NIP-65 + hints |
| Follow discovery | Passive firehose | Active crawl + continuous firehose |
| Event freshness | First seen wins | Multi-source verification with timeout |
| Relay hints | Discarded | Extracted and stored in routing table |
| NIP-65 | Not used | Core of relay discovery strategy |
| Indexer usage | relay.nostr.band as relay only | purplepag.es + nostr.band for bootstrapping |
| Subscription model | 1 global filter for all kind:3 | Batched per-relay, grouped by author |
| Depth control | None (all events) | Progressive level-by-level crawl |
| Network efficiency | Low (broad firehose) | High (targeted, batched, grouped) |
WoT Oracle ingests kind:0 (user metadata / NIP-01) events to populate the profile cache. Profile data (display name, picture URL, NIP-05 identifier, about text, etc.) is made available through the /profiles endpoint and via the include_profiles parameter on other endpoints.
Profiles are ingested through two mechanisms:
-
Relay subscription (continuous): The same relay connections used for kind:3 events also subscribe to kind:0 events. As profiles stream in from connected relays, they are processed and cached alongside follow lists.
-
purplepag.es batch fetch (bootstrap): During initial sync and progressive depth crawling,
purplepag.esis queried in batch for kind:0 events of discovered pubkeys. This is particularly effective becausepurplepag.esserves as a directory relay that aggregates kind:0 and kind:10002 events from across the network.
Profile events follow the same deduplication pattern as kind:3 events:
- LRU dedup cache: An LRU cache keyed by pubkey bytes stores
(created_at, event_id)tuples. Incoming kind:0 events are checked against this cache and rejected if already seen or older than the cached entry. - Timestamp validation: The profile cache only accepts events with a
created_atnewer than the currently stored profile for that pubkey, ensuring stale profiles never overwrite fresher ones. - Replaceable event semantics: Kind:0 is a replaceable event (per NIP-01), so only the latest version per pubkey is retained. The same conflict resolution rules apply: highest
created_atwins, lowest eventidbreaks ties.
Relay subscription (kind:0)
│
▼
Event arrives ──▶ LRU dedup check ──▶ Parse JSON content ──▶ Profile cache update ──▶ Persist queue
(skip if older) (name, picture, etc.) (if newer) (batch of 100)
purplepag.es batch fetch
│
▼
{ kinds: [0], authors: [pubkey1, pubkey2, ...] }
│
▼
Responses ──▶ Same pipeline as above
| Variable | Default | Description |
|---|---|---|
PROFILE_CACHE_SIZE |
50000 |
Maximum number of profiles in the in-memory LRU cache |
PROFILE_CACHE_TTL_SECS |
3600 |
Time-to-live for cached profile entries (1 hour) |
The current process_event function in src/sync/ingestion.rs only extracts pubkeys from p-tags. To also extract relay hints:
// Current: only extracts pubkey (with validation)
let follows: Vec<String> = event.tags.iter()
.filter_map(|tag| {
let tag_vec = tag.as_slice();
if tag_vec.len() >= 2 && tag_vec[0] == "p" {
let pk = &tag_vec[1];
if pk.len() == 64 && pk.bytes().all(|b| b.is_ascii_hexdigit()) {
Some(pk.to_string())
} else {
None
}
} else { None }
}).collect();
// Proposed: extract pubkey + relay hint
struct FollowEntry {
pubkey: String,
relay_hint: Option<String>,
}
let follows: Vec<FollowEntry> = event.tags.iter()
.filter_map(|tag| {
let tag_vec = tag.as_slice();
if tag_vec.len() >= 2 && tag_vec[0] == "p" {
let relay_hint = tag_vec.get(2)
.filter(|r| r.starts_with("wss://") || r.starts_with("ws://"))
.map(|r| r.to_string());
Some(FollowEntry {
pubkey: tag_vec[1].to_string(),
relay_hint,
})
} else { None }
}).collect();fn select_latest_event(events: &[Event]) -> Option<&Event> {
events.iter().max_by(|a, b| {
a.created_at.cmp(&b.created_at)
.then_with(|| b.id.cmp(&a.id)) // Lower ID wins on tie
})
}// Group pubkeys by their known write relays
fn group_by_relay(
pubkeys: &[String],
routing_table: &RelayRoutingTable,
fallback_relays: &[String],
) -> HashMap<String, Vec<String>> {
let mut groups: HashMap<String, Vec<String>> = HashMap::new();
for pubkey in pubkeys {
let relays = routing_table
.get_write_relays(pubkey)
.unwrap_or_else(|| fallback_relays.to_vec());
for relay in relays {
groups.entry(relay).or_default().push(pubkey.clone());
}
}
groups
}