Architecture Overview
BigBrotr follows a three-layer architecture that separates concerns and enables maximum flexibility. This design allows multiple deployments from the same codebase, easy testing through dependency injection, and configuration-driven behavior without code changes.
Three-Layer Architecture
Section titled “Three-Layer Architecture”┌─────────────────────────────────────────────────────────────────────┐│ IMPLEMENTATION LAYER ││ ││ implementations/bigbrotr/ implementations/lilbrotr/ ││ ├── yaml/ ├── yaml/ ││ ├── postgres/init/ ├── postgres/init/ ││ ├── data/seed_relays.txt ├── data/seed_relays.txt ││ └── docker-compose.yaml └── docker-compose.yaml ││ ││ Purpose: Define HOW this specific deployment behaves │└───────────────────────────────┬─────────────────────────────────────┘ │ Uses ▼┌─────────────────────────────────────────────────────────────────────┐│ SERVICE LAYER ││ ││ src/services/ ││ ├── initializer.py Database bootstrap and verification ││ ├── finder.py Relay URL discovery ││ ├── monitor.py Relay health monitoring (NIP-11/NIP-66) ││ ├── synchronizer.py Event collection and sync ││ ├── api.py REST API (planned) ││ └── dvm.py Data Vending Machine (planned) ││ ││ Purpose: Business logic, service coordination, data transformation │└───────────────────────────────┬─────────────────────────────────────┘ │ Leverages ▼┌─────────────────────────────────────────────────────────────────────┐│ CORE LAYER ││ ││ src/core/ ││ ├── pool.py PostgreSQL connection pooling ││ ├── brotr.py Database interface + stored procedures ││ ├── base_service.py Abstract service base class ││ └── logger.py Structured logging ││ ││ Purpose: Reusable foundation, zero business logic │└─────────────────────────────────────────────────────────────────────┘Layer Responsibilities
Section titled “Layer Responsibilities”| Layer | Responsibility | Changes When |
|---|---|---|
| Core | Infrastructure, utilities, abstractions | Rarely - foundation is stable |
| Service | Business logic, orchestration | Feature additions, protocol updates |
| Implementation | Configuration, customization | Per-deployment or environment |
Design Benefits
Section titled “Design Benefits”1. Multiple Deployments
Section titled “1. Multiple Deployments”The same codebase supports different deployment configurations:
- BigBrotr: Full event storage (tags, content) with Tor support
- LilBrotr: Lightweight indexing without tags/content (~60% disk savings)
- Custom: Your own implementation for specific needs
2. Testability
Section titled “2. Testability”Dependency injection enables comprehensive unit testing:
# Production codeservice = MyService(brotr=real_brotr)
# Test codemock_brotr = MagicMock(spec=Brotr)service = MyService(brotr=mock_brotr)3. Configuration-Driven
Section titled “3. Configuration-Driven”All behavior is configurable through YAML files:
concurrency: max_parallel: 10 max_processes: 10tor: enabled: true4. Clear Separation
Section titled “4. Clear Separation”Each layer has a single responsibility:
- Core: “How do we connect to the database?”
- Service: “What do we do with the data?”
- Implementation: “How is this deployment configured?”
Data Flow
Section titled “Data Flow”Event Synchronization Flow
Section titled “Event Synchronization Flow”┌─────────────┐ ┌─────────────┐ ┌─────────────┐│ Finder │ │ Monitor │ │ Synchronizer│└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ Discover │ Check health │ Collect events │ relay URLs │ NIP-11/NIP-66 │ from relays ▼ ▼ ▼┌─────────────────────────────────────────────────────┐│ PostgreSQL ││ ┌─────────┐ ┌──────┐ ┌─────────────────┐ ││ │ relays │ │events│ │ relay_metadata │ ││ └─────────┘ └──────┘ └─────────────────┘ ││ │ │ │ ││ └───────────┴──────────────┘ ││ events_relays │└─────────────────────────────────────────────────────┘Metadata Deduplication Flow
Section titled “Metadata Deduplication Flow”┌──────────────────────────────────────────────────────────────┐│ Monitor Service ││ ││ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ││ │ Fetch NIP-11│────▶│Compute Hash │────▶│Check if exists│ ││ └─────────────┘ └─────────────┘ └──────────────┘ ││ │ ││ ┌─────────────┴──────────┐││ │ │││ ▼ ▼││ ┌──────────────┐ ┌────────┐││ │Insert new rec│ │Reuse ID│││ └──────────────┘ └────────┘││ │ │││ └─────────────┬──────────┘││ │ ││ ▼ ││ ┌─────────────────────────┐││ │ Insert relay_metadata │││ │ (links relay to nip11/ │││ │ nip66 by hash ID) │││ └─────────────────────────┘│└──────────────────────────────────────────────────────────────┘Concurrency Model
Section titled “Concurrency Model”Async I/O
Section titled “Async I/O”All I/O operations are async using:
asyncpgfor database operationsaiohttpfor HTTP requestsaiohttp-socksfor SOCKS5 proxy (Tor)
Connection Pooling
Section titled “Connection Pooling”Application PGBouncer PostgreSQL │ │ │ ├── asyncpg pool ─────────▶├── connection pool ──▶│ │ (20 connections) │ (25 pool size) │ (100 max) │ │ │ ├── Service 1 ────────────▶│ │ ├── Service 2 ────────────▶│ │ ├── Service 3 ────────────▶│ │ └── Service 4 ────────────▶│ │Multicore Processing (Synchronizer)
Section titled “Multicore Processing (Synchronizer)”┌─────────────────────────────────────────────────────────────────┐│ Main Process ││ ││ ┌────────────────────────────────────────────────────────┐ ││ │ aiomultiprocess Pool │ ││ │ │ ││ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ ││ │ │ Worker 1 │ │ Worker 2 │ │ Worker N │ │ ││ │ │ │ │ │ │ │ │ ││ │ │ relay batch │ │ relay batch │ │ relay batch │ │ ││ │ │ │ │ │ │ │ │ │ │ │ ││ │ │ ▼ │ │ ▼ │ │ ▼ │ │ ││ │ │ events │ │ events │ │ events │ │ ││ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ ││ │ │ │ │ │ ││ └─────────┴───────────────┴───────────────┴───────────────┘ ││ │ ││ ▼ ││ ┌────────────────┐ ││ │ Aggregate and │ ││ │ insert to DB │ ││ └────────────────┘ │└─────────────────────────────────────────────────────────────────┘Design Patterns
Section titled “Design Patterns”Dependency Injection
Section titled “Dependency Injection”Services receive their dependencies via constructor:
# Brotr is injected, not created internallyservice = MyService(brotr=brotr, config=config)
# Enables testing with mocksmock_brotr = MagicMock(spec=Brotr)service = MyService(brotr=mock_brotr)Composition
Section titled “Composition”Brotr HAS-A Pool (rather than IS-A):
class Brotr: def __init__(self, pool: Pool | None = None, ...): self._pool = pool or Pool(...)
@property def pool(self) -> Pool: return self._poolTemplate Method
Section titled “Template Method”BaseService.run_forever() calls abstract run():
class BaseService: async def run_forever(self, interval: float) -> None: while not self._shutdown_requested: await self.run() # Template method if await self.wait(interval): break
@abstractmethod async def run(self) -> None: """Implemented by subclasses.""" passFactory Method
Section titled “Factory Method”Services provide multiple construction paths:
# From YAML fileservice = MyService.from_yaml("config.yaml", brotr=brotr)
# From dictionaryservice = MyService.from_dict(config_dict, brotr=brotr)
# Direct constructionservice = MyService(brotr=brotr, config=MyServiceConfig(...))Context Manager
Section titled “Context Manager”Resources are automatically managed:
async with brotr: # Connect on enter, close on exit async with service: # Load state on enter, save on exit await service.run_forever(interval=3600)Graceful Shutdown
Section titled “Graceful Shutdown”# Signal handler (sync context)def handle_signal(signum, frame): service.request_shutdown() # Sets flag, doesn't await
# Service main loopasync def run_forever(self, interval: float) -> None: while not self._shutdown_requested: await self.run() if await self.wait(interval): # Returns early if shutdown break # Cleanup happens in context manager __aexit__Next Steps
Section titled “Next Steps”- Learn about the Core Layer
- Explore the Service Layer
- Understand Configuration