GoLakeScanner: Case Study | Shourya Upadhyaya

The problem

Scanning large unstructured data lakes full of nested PDFs, scanned images, and corrupt archives fails when a single hung file parser blocks the entire pipeline or when unchanged files are re-indexed on every run.

Re-running deep magic-byte and PDF text-stream probes on unchanged terabyte directories wastes CPU and I/O.

A single malformed PDF or stalled network mount can hang a worker goroutine indefinitely without per-file deadlines.

How it works

01

Phase 1: Streaming SHA-256 state diffing

Walked directory trees and computed streaming 64 KB SHA-256 digests against the PostgreSQL catalog to classify every file as New, Modified, or Unchanged before dispatching heavy probes.

02

Phase 2: Generic bounded worker pool

Implemented pool.Run[In, Out](ctx, workers, jobs, fn) using Go generics and bounded channels to cap concurrency while propagating cancellation cleanly.

03

Per-file 30s context timeouts & magic-byte probes

Wrapped each worker invocation in context.WithTimeout(ctx, 30*time.Second) while inspecting 512-byte magic headers and PDF /Font & /FlateDecode streams to distinguish native PDFs from scanned images.

04

Signal-safe pgx/v5 PostgreSQL catalog

Persisted file state transitions via pgx/v5 connection pools and handled SIGINT and SIGTERM so interrupted scans resume from the exact unindexed file.

Architecture

Two-phase incremental SHA-256 walker and generic bounded goroutine worker pool.

01
Walk nested directories
Streams files from the target data lake without loading full files into memory.
02
Incremental SHA-256 diffing
Unchanged files are skipped immediately; only new or modified files enter Phase 2.
03
Generic bounded worker pool
pool.Run[In, Out] dispatches jobs across N worker goroutines with channel backpressure.
04
30s timeout-isolated classification
Magic-byte and PDF structure probes classify files without risking a pipeline hang.
05
Resumable pgx/v5 catalog
Results commit atomically to PostgreSQL with clean SIGINT/SIGTERM resume support.

Engineering guarantees

phases: incremental SHA-256 diffing then bounded deep probes: 2
hard per-file context timeout preventing worker starvation: 30 s
streaming 64 KB buffer memory footprint per worker: O(1)
resumable progress across SIGINT/SIGTERM interrupts: 100%

Technical deep dive

Complete system specification, schemas, and implementation notes.

A fault-tolerant Go CLI and worker-pool pipeline that walks, hashes, classifies, and indexes unstructured data lakes into PostgreSQL.

The Problem

Unstructured data lakes accumulate thousands of PDFs, scanned invoices, audio recordings, videos, and corrupted files across deeply nested directories. Re-scanning an entire directory tree from scratch on every run wastes CPU and I/O, while a single malformed PDF or hung external probe (ffprobe/pdftotext) can stall a naive single-threaded scanner indefinitely.

The Outcome

A fast, incremental Go ingestion pipeline (Go + pgx/v5 + Bounded Worker Pool + PostgreSQL) that detects new, changed, and unchanged files via SHA-256 content addressing, isolates per-file classification behind strict context timeouts, and catalogs rich media/document metadata idempotently.

How It Works (The Mechanism)

Incremental Content-Addressed Walking: Streams the filesystem tree, computing SHA-256 hashes and file sizes to upsert records into PostgreSQL as new, changed, or unchanged—ensuring unchanged files are never re-classified on subsequent runs.
Bounded Generic Worker Pool (pool.Run[In, Out]): Dispatches pending unclassified files across N concurrent goroutine workers using Go generics ([In, Out any]), channels, and sync.WaitGroup with per-item context.WithTimeout deadlines (default 30s).
Deep Magic-Byte & Subprocess Probing: Inspects file headers and invokes bounded external probes to classify files beyond simple extensions—distinguishing searchable text PDFs from scanned image-only PDFs, detecting corrupted/broken archives, and extracting audio/video/image stream metadata.
Graceful SIGINT/SIGTERM Resumability: Traps OS interrupt signals via signal.NotifyContext, cancelling in-flight workers cleanly while persisting partial progress in PostgreSQL so interrupted scans resume seamlessly on the next run.

Architecture & System Design

Key Engineering Decisions

Two-Phase Walk-Then-Classify Pipeline: Separating fast filesystem hashing (Phase 1) from CPU/subprocess-bound classification (Phase 2) ensures the directory walk completes rapidly and records an exact snapshot in PostgreSQL before spinning up heavy classification workers.
Per-File Context Deadlines & Fault Isolation: If a corrupted file causes a classification probe to error or time out after 30 seconds, the worker catches the context error, records the failure reason in the PostgreSQL catalog (SetError), and continues processing the queue without crashing the pipeline.
Zero-ORM pgx/v5 Connection Pooling: Uses native jackc/pgx/v5 for low-overhead binary protocol communication with PostgreSQL.

The Value Stack

Language & Concurrency: Go (Golang) 1.24, Generics, Goroutines, Channels, context cancellation, os/signal.
Database: PostgreSQL, jackc/pgx/v5.
Output & Observability: Structured JSONL (stdout) streaming + summary telemetry (stderr).

The Proof

Built a generic concurrent worker pool (pool.Run[In, Out]) in Go with per-task timeouts and graceful SIGINT/SIGTERM cancellation.
Engineered idempotent, content-addressed incremental scanning that skips unchanged files and automatically retries failed or interrupted classifications on the next run.