Skip to main content

System Architecture & Technical Specifications

This document outlines the distributed architecture, component boundaries, data flow, vector search pipeline, and deployment topology of the GitHub Backup Automation & Observatory System.

1.1 Go Service Layout​

Each Go service uses a canonical command entrypoint and keeps startup wiring separate from application behavior:

<service>/
├── cmd/<service-name>/main.go  # executable entrypoint
├── internal/app/               # dependency wiring and lifecycle
├── config/                     # service configuration
└── domain-specific packages/   # transport, services, repositories, and models

The backend, backup worker, and MCP server expose their commands under cmd/ (the old root main.go wrappers were removed). make backup, make setup, the Dockerfiles and GoReleaser all build or run those cmd/ packages.

Shared Go packages (src/internal/)​

Infrastructure used by more than one Go service lives in small packages under src/internal/, importable only from inside src/:

PackageResponsibilityUsed by
envfileLoads repo-relative .env files (root found from the working directory or the executable); ENV_FILE selects one explicit file; never overrides variables already in the environment; test binaries only honour ENV_FILEAPI, MCP, worker
loggingslog JSON logger (RFC 3339 time, level, msg, service), LOG_LEVEL parsing, request IDs carried in the contextAPI, MCP, worker
pgpoolpgxpool constructor with Neon-friendly defaults and per-service application_nameAPI, MCP, worker
dbmigrateEmbedded schema migrations and the runner (one transaction, advisory lock)API, worker
gitexecgit runner: argv only, own session and process group killed on timeout, bounded output, non-interactive environment, credentials as GIT_CONFIG_* via Runner.Config, transient retry, redactionworker, MCP (git bundles)
githubapiGitHub REST client: pagination, typed errors, rate-limit pauses shared by the client, per-request token source, bounded 5xx retriesworker
pgtestIntegration-test helpers gated on TEST_DATABASE_URLtests

1. High-Level Architecture Topology​

graph TD
    subgraph Client Layer
        Browser[Web Browser / User]
    end

    subgraph Hosting: Vercel Serverless Edge
        Frontend[Next.js 16 App Router UI]
        Observatory[Python FastAPI AI Observatory]
    end

    subgraph Hosting: Render Docker Web Service
        Backend[Go Fiber v2 API Server]
    end

    subgraph Hosting: Local / Scheduled Cron
        Worker[Go CLI Backup Worker Engine]
    end

    subgraph Cloud Storage & Intelligence
        NeonDB[(Neon PostgreSQL + pgvector)]
        GitCentral[Central Backup Git Repository]
        OpenRouter[OpenRouter AI / Free Embedding Registry]
    end

    Browser -->|HTTPS / SSE, same origin only| Frontend

    Frontend -->|"/api/proxy/agent: REST / SSE + session JWT (Bearer)"| Observatory
    Frontend -->|"/api/proxy/backup and server components: REST / SSE + X-Internal-Secret"| Backend

    Observatory -->|Async SQLAlchemy / pgvector| NeonDB
    Observatory -->|HTTP Multi-Key Failover| OpenRouter
    Observatory -->|X-Internal-Secret| Backend

    Backend -->|pgxpool / SQL| NeonDB

    Worker -->|Incremental Archiving| GitCentral
    Worker -->|Log / Metrics Ingest| NeonDB

2. Core Service Components & Responsibilities​

Tier 1: Frontend Dashboard (Next.js 16 App Router)​

  • Hosting: Vercel (Turbopack, Serverless Functions).
  • Role: Single-pane-of-glass operations console behind a public landing page (/); the overview is /dashboard.
  • Design system: the Observatory design system, two packages installed from a GitHub release: @mishrashardendu22/observatory-tokens (CSS custom properties, a Tailwind v4 theme and a typed object; dark by default, light when the device prefers it, no toggle) and @mishrashardendu22/observatory-ui (React components and their stylesheet). The dashboard keeps only its shell, sidebar chrome and page-specific styles in src/app/styles/.
  • Key Features:
    • Sign-in & Sessions: "Sign in with GitHub" (OAuth web flow with PKCE, limited to ALLOWED_GITHUB_USERS) or the Observatory password login, both done by dashboard route handlers under /api/auth/*. The session is an HS256 JWT (shared JWT_SECRET, iss/aud github-backup) in an httpOnly cookie; the Observatory proxy sends it as a Bearer token, ignores browser-supplied Authorization headers and refuses cross-origin writes. DASHBOARD_REQUIRE_LOGIN=true gates every page and /api/proxy/* (src/proxy.ts). Users can connect their own OpenRouter key (PKCE; the Observatory stores it). Setup: AUTH_SETUP.md.
    • Runtime Backend Proxy: The browser only talks to the dashboard. Route handlers under /api/proxy/backup/* and /api/proxy/agent/* forward allowlisted requests to BACKUP_API_URL and AGENT_API_URL, which the server reads at request time; INTERNAL_SECRET is attached on the server for Go API calls and never reaches the browser.
    • Real-time Live Logs: SSE from the Go backend (/api/logs/stream) through the proxy, with batched rendering and bounded reconnects; falls back to polling /api/logs while the stream endpoint is not deployed.
    • Search Playground: Live hybrid query testing with model selection, generation status badges, and score breakdowns.
    • AI Chat & Investigation Console: Multi-turn agentic chat with SSE token streaming, thinking visualization, and Human-in-the-Loop (HITL) approval modals.
    • Analytics & Metrics: Visual charts for repository sizes, run durations, success rates, and commit trends.

Tier 2: AI Agentic Observatory (FastAPI + LangChain)​

  • Hosting: Vercel (Python Serverless Runtime).
  • Role: Cognitive telemetry, vector embeddings, and autonomous incident investigation.
  • Key Features:
    • Tool-Calling RAG Pipeline: Multi-turn agent with dynamic tool invocation (hybrid_search_knowledge_base, fetch_backup_metrics, list_backup_runs, send_report_email).
    • Dynamic OpenRouter Model Registry: Fetches 100% free embedding and reranking models dynamically with in-memory caching and fallback definitions.
    • Multi-Key OpenRouter Failover: Per-request rotation across the comma-separated server keys in OPENROUTER_API_KEY, only on 401, 402 or 429; other errors are reported to the user. A user who connected their own OpenRouter key (PKCE, stored encrypted in ai_openrouter_credentials) is served only with that key, never another user's.
    • Bounded LLM Calls: 30 second request timeout, no SDK retries, one cached client per model and key, and a 280 second deadline per chat turn; /chat/stream streams the model's tokens as they arrive.
    • Vector Lifecycle Engine: Blue-Green style index generation (BUILDING -> READY -> ACTIVE -> RETIRED). Only a READY generation can be activated; the generation it replaces is retired, and only RETIRED and FAILED generations are deleted, by an explicit prune.
    • Human-In-The-Loop (HITL): Sensitive actions yield confirmation tokens (confirm_id) requiring user approval before execution. Confirmations are rows in ai_tool_confirmations, so any instance can record the decision; only the user who started the turn can decide, and after 120 seconds the action counts as rejected.
    • No startup DDL: the service only checks the database connection when it starts. Its tables are created by the Go migrations (see section 5).

Tier 3: High-Performance Backend API (Go 1.25 + Fiber v2)​

  • Hosting: Render (Docker web service built from src/backend/github-backup-api/Dockerfile, see render.yaml).
  • Role: Read-only REST API over the data the worker writes to PostgreSQL itself, plus live log streams (SSE /api/logs/stream for the dashboard, /ws/live for non-browser clients), both fed by the shared livefeed poller.
  • Key Features:
    • Connection Pooling: pgxpool built by src/internal/pgpool: DB_MAX_CONNS (default 10) and DB_MIN_CONNS (default 0, so an idle API holds no connections and Neon can scale to zero), idle connections closed after 1m, 30m lifetime, 10s connect timeout, application_name=github-backup-api.
    • Versioned Migrations: Embedded SQL migrations and runner in src/internal/dbmigrate, applied at startup for zero-downtime idempotent schema updates.
    • Structured Logging & Metrics: log/slog JSON logging via src/internal/logging and native Prometheus metrics (/metrics).
    • Live Logs: one livefeed poller that runs only while someone subscribes, feeding the SSE stream the dashboard reads through its proxy and the /ws/live WebSocket, which needs the internal secret and so serves non-browser clients.

Tier 4: Autonomous Backup Worker (Go CLI — src/services/github-backup-worker/)​

  • Hosting: Local Machine / Scheduled Timer. make setup (scripts/setup.sh; scripts/setup.ps1 on Windows, untested) builds ~/.local/bin/github-backup-worker and schedules it per user: systemd user timers on Linux, launchd agents on macOS, crontab elsewhere; weekly run --scheduled on Mondays 00:00 and an hourly run --pending-only. make backup runs go run ./cmd/github-backup-worker in the worker directory; --dry-run, --only owner/repo, --limit N, --force; subcommands are a table in internal/app (run is the default, plus login, logout, auth, github-app, version).
  • Logins (credentials, githubauth, githubapp, openrouter): a read-only GitHub App registered from githubapp/manifest.json, device-flow user tokens refreshed automatically, the GitHub CLI's token, and an OpenRouter PKCE key, stored in the OS keychain (go-keyring) or a 0600 file, indexed in credentials.json. Without a login the environment tokens are used. Guide: CLI_AUTH_SETUP.md.
  • Role: Repository discovery, change detection, full archives, pluggable storage, telemetry. Details and the configuration reference: src/services/github-backup-worker/README.md.
  • Pipeline (engine):
    1. Discovery (discovery, src/internal/githubapi): organization, user and private listings, or with a GitHub App login the repositories of each app installation (/user/installations), every page, 30 s per request; any listing error, and finding no installation, aborts the run before anything is written or deleted.
    2. Change detection: unchanged when pushed_at and the archive format match the recorded state (status ok, empty or too_large) and the sink still holds the recorded archive; no git call for unchanged repositories.
    3. Archiving (archive, src/internal/gitexec): BACKUP_CONCURRENCY workers clone bare (branches and tags, no pull request refs) and write a git bundle (--branches --tags, verified, SHA-256) or a git archive tarball, named <owner>__<repo>.<ext> in lowercase. Empty repositories are recorded as empty; LFS objects are not included.
    4. Storage (sink): one uploader writes to git (a clone of BACKUP_REPO_PATH, one commit and push per batch, an https:// destination pushed with the stored destination login through a scoped header, files over 100 MiB recorded as too_large, private repositories only into a destination the API confirms private), local (atomic files with .sha256 sidecars) or s3 (any S3-compatible bucket, streaming uploads, SHA-256 metadata). State is recorded only after the upload or push containing the archive succeeded.
    5. Retention: repositories GitHub no longer lists keep their archives and are marked missing; pruning needs BACKUP_PRUNE_DELETED=true and is refused above BACKUP_MAX_DELETIONS per run. Legacy <repo>.tar.gz files are removed from the git sink after the new archives are pushed.
  • State: with DATABASE_URL, worker_repo_state in PostgreSQL (migration 000008, applied by the worker at startup), shared by every machine; without it the same table in a local SQLite file (DB_PATH, pure-Go modernc.org/sqlite) for one machine. app.db is no longer committed to the repository.
  • Coordination (runctl): the backup-run row of worker_run_lease (atomic upsert, 10 minute expiry, one-minute heartbeat) lets one machine run at a time; --scheduled skips a timer run within BACKUP_MIN_INTERVAL of the last completed run unless requests are pending; pending backup_run_requests are claimed with FOR UPDATE SKIP LOCKED, forced and marked done/failed; --pending-only serves them from a frequent timer.
  • PostgreSQL Telemetry (monitor): backup_runs, backup_results + backup_result_errors, backup_run_errors, execution_logs, analytics_snapshots (skipped when no run was recorded), written through a buffered single-writer queue (15 s per write, one retry) that is flushed on shutdown.

3. Vector Search & Tool-Calling RAG Pipeline​

sequenceDiagram
    autonumber
    actor User as User / Admin
    participant UI as Frontend (Next.js)
    participant Agent as Python Observatory
    participant DB as PostgreSQL + pgvector
    participant LLM as OpenRouter AI

    User->>UI: Submit inquiry ("Analyze clone failures")
    UI->>Agent: POST /chat/stream (SSE)
    
    rect rgb(240, 245, 255)
        Note over Agent,DB: Pre-Turn Context Enrichment
        Agent->>DB: Query top relevance chunks (pgvector + FTS)
        DB-->>Agent: Ingest initial context
    end

    Agent->>LLM: Turn 1: System prompt + User question + Injected context
    LLM-->>Agent: Emit tool call: hybrid_search_knowledge_base("clone timeout")
    
    rect rgb(240, 255, 240)
        Note over Agent,DB: Dynamic Tool-Calling RAG
        Agent->>DB: Execute FTS + pgvector Cosine Search + RRF
        alt Chunks not indexed
            Agent->>DB: Query raw live tables directly
        end
        DB-->>Agent: Ranked source chunks
    end

    Agent->>LLM: Turn 2: Feed tool results back to LLM
    LLM-->>Agent: Emit final response tokens
    Agent-->>UI: Stream SSE tokens to UI
    UI-->>User: Display markdown-formatted answer

4. Hybrid Search Architecture​

The search engine implements a robust 3-stage retrieval pipeline:

  1. Full-Text Search (FTS) & Trigram Matching:
    • Utilizes PostgreSQL tsvector (content_tsv) with ts_rank_cd scoring for keyword accuracy.
    • Includes ILIKE substring fallback for technical symbols and error codes.
  2. pgvector Semantic Similarity:
    • Uses cosine distance operator <=> against generation-specific embedding vectors.
  3. Reciprocal Rank Fusion (RRF):
    • Merges FTS and semantic results using reciprocal rank scoring: $$\text{Score}(d) = \frac{w_{\text{fts}}}{k + r_{\text{fts}}(d) + 1} + \frac{w_{\text{sem}}}{k + r_{\text{sem}}(d) + 1}$$
    • Smooths ranking discrepancies and eliminates threshold tuning dependencies.
  4. Live Table Fallback:
    • If vector index generations are pending or unindexed, automatically falls back to live SQL queries across execution_logs, backup_results, investigations, backup_fixes, and ai_chat_messages.

5. Database Schema & Migration Architecture​

PostgreSQL serves as the central data store across all services. The Go backend runs the idempotent versioned migrations at startup; the Python Observatory runs no DDL and expects these tables to exist:

The Go migrations live in src/internal/dbmigrate/migrations/ and are embedded in the binaries; the API applies them at startup (db.RunMigrations, 60s timeout), and so does the backup worker when DATABASE_URL is set (2 minute timeout). A run is a single read-committed transaction that first takes pg_advisory_xact_lock(dbmigrate.LockKey), so instances starting together queue up instead of racing, and a failed run keeps nothing. Every lock a migration takes lasts until the run commits and CREATE INDEX CONCURRENTLY is not possible; CREATE INDEX locks its table even when the index exists, so guard it on tables other services write to (as 000000 does). The lock is transaction-scoped because Neon's pooled endpoint (PgBouncer in transaction mode) cannot hold a session-level lock reliably. Applied versions are recorded in schema_migrations (version, name, dirty, applied_at); any unrecorded version is applied, even below the highest recorded one. Version numbers are reserved per workstream: 000008 worker, 000009 API and MCP, 000010 observatory.

Core Relational & Telemetry Models​

  1. backup_runs & backup_run_errors: Execution batches, totals, durations, status, and dedicated error tracking.
  2. backup_results & backup_result_errors: Individual repository outcomes, archive sizes, commit hashes, and dedicated error tracking.
  3. execution_logs: Structured step-by-step logs with run associations and timestamp indices.
  4. analytics_snapshots: 1-to-1 repository Git analytics and blob metrics per backup run (run_id PK).
  5. backup_fixes, backup_run_fixes, & backup_fix_commits: Incident resolutions, run associations, and optional commit tags.
  6. ai_chat_sessions & ai_session_metadata: Multi-turn chat sessions with dedicated normalized metadata.
  7. ai_chat_messages, ai_tool_calls, ai_tool_call_args, & ai_tool_call_errors: Chat history and granular tool execution telemetry with relational argument and failure tables.
  8. investigations & investigation_errors: Saved agent investigation traces, questions, answers, tool calls, and dedicated failure table.
  9. embedding_generations, embedding_chunks, embedding_chunk_metadata, embedding_jobs, & embedding_job_errors: pgvector vector store and durable work queue with dedicated chunk metadata and failure tables.
  10. worker_repo_state, worker_run_lease, & backup_run_requests: backup worker state, the run lease and queued backup requests (written by the worker; requests are inserted by other services).
  11. ai_openrouter_credentials & ai_tool_confirmations: users' own OpenRouter keys (Fernet-encrypted) and HITL confirmations (owner, tool, arguments, status, decision time).

Versioned Migration History​

  • 000000_observatory_tables: The observatory tables (ai_chat_sessions, ai_session_metadata, ai_chat_messages, ai_tool_calls, ai_tool_call_args, ai_tool_call_errors, investigations, investigation_errors, ai_reports) exactly as SQLAlchemy's create_all() creates them from data/db.py, with IF NOT EXISTS. It lets a fresh database migrate before the observatory has ever run; on databases that already recorded 1-6 it is a no-op apart from creating a missing index.
  • 000001_initial_schema: Core telemetry tables (backup_runs, backup_results, execution_logs, analytics_snapshots, backup_fixes).
  • 000002_embeddings_and_search: Extensions vector & pg_trgm, tables embedding_generations, embedding_chunks, embedding_jobs, embedding_indexing_checkpoints.
  • 000003_normalize_schema_and_metadata: Normalized ai_session_metadata table, unique index on analytics run_id, empty string cleanup to SQL NULL.
  • 000004_cleanup_stale_embeddings_and_errors: Obsolete generation pruning and failed job cleanup; the investigations.error cleanup only runs where that legacy column exists.
  • 000005_deterministic_embedding_lifecycle: Dedicated ai_session_metadata, backup_run_errors, backup_result_errors, backup_fix_commits, embedding_chunk_metadata, embedding_job_errors, investigation_errors, ai_tool_call_args, and ai_tool_call_errors tables, primary key deduplication on analytics_snapshots, partial indexes for fast failure/error lookups (idx_backup_results_status_failed, idx_backup_runs_status_failed, idx_investigations_status_failed, idx_ai_tool_calls_success_false), commit hash indexes (idx_backup_results_commit, idx_backup_fix_commits_hash), and transactional blue-green promotion.
  • 000006_prune_stale_embedding_generations: Deletes every non-active embedding generation (cascading to chunks and jobs), keeping the most complete one when none is active.
  • 000008_worker_state: Backup worker tables: worker_repo_state (per-repository state: pushed_at and format of the last backup, archive key, SHA-256 and size, status ok/empty/too_large/failed/missing/withheld), worker_run_lease (the cross-machine run lease) and backup_run_requests (queued backups; NULL target means every repository; partial index on pending rows).
  • 000010_observatory_hardening (in the shared src/internal/dbmigrate/migrations/): ai_openrouter_credentials, ai_tool_confirmations with its session and pending indexes, and the partial HNSW index idx_embedding_chunks_hnsw_1024 on (embedding::vector(1024)) for rows with 1024 dimensions. The index is skipped with a NOTICE when embedding_chunks already has more than 200000 rows; create it CONCURRENTLY by hand then.

6. Security, Deployment Model & Future SaaS Architecture​

  • Containerized Deployment Architecture: Go backend deploys as a Docker container on Render; Frontend and AI Observatory deploy on Vercel; Backup Worker CLI runs as an isolated container or native scheduled job.
  • Non-Destructive Database Migrations: All migrations must be idempotent (CREATE TABLE IF NOT EXISTS, CREATE INDEX IF NOT EXISTS) and preserve historical data on Neon PostgreSQL.
  • Failover Resilience: Multi-key OpenRouter rotation ensures continuous uptime against LLM provider rate limits.
  • Enterprise SaaS Transformation & MCP Roadmap: Full technical specification for UI Connector Hubs, AES-256 encrypted credential vaults, pluggable multi-cloud storage (S3/R2/GCS/Drive), self-hosted distribution, and Autonomous Model Context Protocol (MCP) servers is documented in docs/SAAS_ARCHITECTURE_AND_MCP_ROADMAP.md.