Skip to content
BigBrotr
GitHubRSS

Architecture Overview

BigBrotr’s architecture follows a diamond DAG (Directed Acyclic Graph) dependency structure. Every import flows strictly downward through well-defined layers, preventing circular dependencies and ensuring clean separation of concerns.

Diamond DAG

Section titled “Diamond DAG”
              services          Orchestration
             /   |   \
          core  nips  utils    Infrastructure
             \   |   /
              models           Pure domain types

Five packages, one rule: imports only flow downward.

LayerPackagesResponsibility
TopservicesBusiness logic. Six independent services that orchestrate discovery, monitoring, and archiving.
MiddlecoreConnection pool, database facade, base service, logging, metrics.
MiddlenipsProtocol-aware I/O. NIP-11 relay information, NIP-66 health monitoring. Depends on utils and models.
MiddleutilsNetwork primitives. DNS resolution, Nostr key management, WebSocket/HTTP transport, SOCKS5 proxy.
BottommodelsPure frozen dataclasses. Zero I/O, zero bigbrotr imports. Uses only stdlib.

Import Rules

Section titled “Import Rules”
  • Same package: relative imports (from .logger import Logger)
  • Cross-package: absolute imports (from bigbrotr.core.logger import Logger)
  • models layer: stdlib only (import logging)
  • No upward imports: models never imports from core; core never imports from services.
  • No parent-relative imports: enforced by ruff rule ban-relative-imports = "parents".

Design Principles

Section titled “Design Principles”

Independent Services

Section titled “Independent Services”

All six services run as independent processes. They communicate exclusively through the shared PostgreSQL database. There are no message queues, no gRPC calls, no inter-service APIs. Each service can start, stop, scale, and fail independently without affecting the others.

Immutable Domain Models

Section titled “Immutable Domain Models”

Every model is a frozen dataclass with __slots__. Instances are immutable after construction. Validation happens in __post_init__ — invalid data never escapes the constructor. This fail-fast approach means bugs surface immediately rather than propagating through the system.

Content-Addressed Deduplication

Section titled “Content-Addressed Deduplication”

Metadata objects are hashed with SHA-256. Same data always produces the same hash. The composite primary key (id, metadata_type) means deduplication operates within each metadata type. The type is NOT included in the hash — deduplication is per-type. This eliminates duplicates at the data layer without application-level coordination.

Database as Integration Point

Section titled “Database as Integration Point”

The PostgreSQL database is the single source of truth and the only communication channel between services. All mutations go through stored functions with bulk array parameters. Materialized views provide pre-computed analytics. This design is simple, reliable, and eliminates distributed system failure modes.

Never Trust Stored Data

Section titled “Never Trust Stored Data”

When reconstructing objects from the database, BigBrotr re-validates everything. Relay.from_db_params() re-parses the URL completely. Metadata.from_db_params() recomputes and verifies the hash. This defensive approach catches data corruption at the boundary rather than deep in business logic.

System Topology

Section titled “System Topology”

A complete BigBrotr deployment consists of:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Seeder    │     │   Finder    │     │  Validator   │
│  (one-shot) │     │ (continuous)│     │ (continuous) │
└──────┬──────┘     └──────┬──────┘     └──────┬──────┘
       │                   │                   │
       ▼                   ▼                   ▼
┌────────────────────────────────────────────────────┐
│                    PgBouncer                       │
│            (transaction pooling)                   │
├────────────────────────────────────────────────────┤
│                   PostgreSQL                       │
│    6 tables · 25 functions · 11 materialized views  │
└────────────────────────────────────────────────────┘
       ▲                   ▲                   ▲
       │                   │                   │
┌──────┴──────┐     ┌──────┴──────┐     ┌──────┴──────┐
│   Monitor   │     │  Refresher  │     │Synchronizer │
│ (continuous)│     │ (scheduled) │     │ (continuous) │
└─────────────┘     └─────────────┘     └─────────────┘

Each service connects to PgBouncer, which provides connection pooling in transaction mode. Writer services (all six) use the writer database user; read-only consumers (APIs, DVMs, monitoring dashboards) use the reader user.

Next Steps

Section titled “Next Steps”