SKIP TO CONTENT
All projects
Real-Time Audio ML & Streaming SystemsSmart India Hackathon · VIT Vellore College Finalist

VoiceShield AI

Low-latency 48 kHz PCM16 audio gateway and multi-model deepfake & scam-risk engine built for Smart India Hackathon (VIT Vellore Finalist).

Timeline
Sep 2025 — Dec 2025
Stack
Node.js · TypeScript · Python · FastAPI · PyTorch · OpenAI Whisper · ONNX Runtime · SpeechBrain · WebSockets · PostgreSQL · Prisma · Next.js · Tailwind CSS

The problem

Voice-cloning deepfakes and high-pressure social-engineering phone scams happen mid-call. Detecting them requires analyzing raw audio and conversational intent within seconds without blocking the live audio stream.

  • WebSocket transport frames arrive in arbitrary sizes, whereas acoustic deepfake models require contiguous 3-second 48 kHz mono PCM16 windows (288,000 bytes).
  • Synchronous PostgreSQL writes or slow model calls on the ingest thread cause buffer underruns and dropped call frames.
  • Acoustic models alone miss scripted human scams, while STT + LLM rules alone miss synthetic voice clones.

How it works

  1. 01

    Byte-conserving 48 kHz PCM16 ring buffer

    Built a Node.js/TypeScript media gateway that ingests arbitrary WebSocket binary frames, reassembles them into exact 3-second (288 KB) chunks, and verifies a byte-conservation invariant on every emission.
  2. 02

    Persistent WebSocket bridge to Python ML

    Streamed assembled chunks over a persistent internal WebSocket to a FastAPI inference worker, avoiding per-chunk HTTP handshake and serialization overhead.
  3. 03

    Dual-stream acoustic + Whisper analysis

    Combined RawNet2, WavLM, and SpeechBrain (ECAPA-TDNN) spoof scoring on raw waveforms with OpenAI Whisper transcripts fed into a deterministic scam-risk agent.
  4. 04

    Off-path PostgreSQL write queue

    Decoupled database persistence into an asynchronous write queue so disk I/O never stalls audio frame ingestion, while streaming live per-chunk risk scores to a Next.js dashboard via Socket.IO.

Architecture

From live 48 kHz PCM16 telephony frames to sub-second Socket.IO scam alerts.

48kHz PCM16Android · SIPWS GATEWAYNode.js + TS3s CHUNKERByte-InvariantACOUSTIC MLRawNet2 · ONNXWHISPER STTScam-Risk AgentASYNC DB QUEUEOff-Path PG WriteLIVE UISocket.IO Stream
  1. 01
    Live 48 kHz mono PCM16 stream

    Audio frames arrive over WebSockets from the Android CallVault client or SIP bridge.

  2. 02
    Non-blocking Node.js ingest

    The TypeScript gateway validates session state and buffers incoming binary frames.

  3. 03
    Exact 3-second window assembly

    Frames are sliced and concatenated into ordered 288,000-byte windows with zero dropped or duplicated bytes.

  4. 04
    Parallel acoustic + transcript inference

    FastAPI evaluates voice spoof probability alongside Whisper transcription and scam-intent rules.

  5. 05
    Off-path PostgreSQL persistence

    Call telemetry and chunk verdicts are queued asynchronously off the real-time audio thread.

  6. 06
    Real-time Socket.IO dashboard

    Per-window spoof scores and transcripts stream live to the operator dashboard.

Streaming benchmarks

median CPU inference per 3-second audio window
1.1 s
mono PCM16 sampling rate with 100% byte conservation
48 kHz
ordered sliding audio window (288 KB/chunk)
3 s
blocking database writes on the audio ingest path
0
Processing budget per 3.0s real-time windowMedian end-to-end CPU processing latency vs the 3.0-second real-time deadline (lower is better).
  • VoiceShield CPU Pipeline (Median)1.1s
  • Real-Time Window Budget3.0s
3.0s real-time budget

Technical deep dive

Complete system specification, schemas, and implementation notes.

Smart India Hackathon (SIH) College Finalist — VIT Vellore. VoiceShield AI intercepts live telephony streams and runs real-time acoustic deepfake detection, speech-to-text transcription, and scam-risk evaluation without interrupting the call.

The Problem

Voice cloning and social-engineering phone scams happen in real time during live calls. Traditional post-call forensic tools analyze recordings only after the victim has already transferred money or shared sensitive credentials. Worse, streaming raw telephony audio directly into heavy Python ML models causes frame drops, backpressure stalls, and database write bottlenecks.

The Outcome

A sub-second, fault-tolerant real-time audio analysis pipeline that streams 48 kHz mono PCM16 audio over WebSockets, isolates slow database writes off the hot audio path, runs multi-model acoustic + Whisper inference in 1.1s median per 3-second window, and anchors tamper-evident forensic commitments on-chain.

How It Works (The Mechanism)

  1. Deterministic 3-Second Audio Chunking: A custom Node.js/TypeScript Media Gateway ingests arbitrary WebSocket PCM16 frames from an on-device Android daemon (CallVault) or SIP/FreeSWITCH trunk, reassembling them into exact 3-second buffers (288,000 bytes) verified by a strict byte-conservation invariant.
  2. Hot-Path Isolation: Audio buffers are routed immediately over a persistent WebSocket to the Python/FastAPI inference worker while PostgreSQL persistence is decoupled onto asynchronous background queues—ensuring database latency never blocks live audio frames.
  3. Multi-Signal Scam Risk Agent: Combines acoustic synthetic-speech models (Indic RawNet2, Dhwani ONNX, WavLM, ECAPA-TDNN) with OpenAI Whisper transcriptions inside a quality-gated, rule-based scam-risk evaluator (1.1s median CPU latency per 3s window).
  4. Tamper-Evident Audit Trail: While playable WAV evidence remains off-chain, its SHA-256 fingerprint and canonical AI verdict commitment are anchored to an append-only EVM smart contract.

Architecture & System Design



Key Engineering Decisions


    

  • Byte-Conservation Invariant on Frame Reassembly: Network WebSocket frames arrive in arbitrary byte lengths. The gateway maintains an ordered ring buffer that slices exact 288,000-byte windows (3 seconds of 48 kHz 16-bit mono PCM) and asserts bytes_in == bytes_emitted + bytes_buffered across every session, backed by 50+ unit and protocol tests.
    • 

  • Off-Path PostgreSQL Queueing: Writing raw chunk metadata and per-window ML scores synchronously during audio ingestion introduces tail-latency spikes. Queuing Prisma/PostgreSQL writes asynchronously keeps the WebSocket audio loop non-blocking.
    • 

  • CPU-Optimized Inference (~0.37x Real-Time Factor): By combining ONNX Runtime quantization with quality-gated speech windows (VAD/RMS energy filtering), the pipeline achieves a 1.1-second median processing time per 3-second audio window on CPU, enabling real-time alerting without requiring dedicated cloud GPUs.
    • 

  • Zero-PC Android Call Interception (CallVault): Built a companion Android daemon that pairs via on-device Wireless Debugging to capture live call audio streams directly on the handset without requiring a tethered PC.
    • 



The Value Stack


    

  • Media Gateway: Node.js, TypeScript, WebSockets (ws), Socket.IO, Prisma ORM, Vitest (50+ tests).
    • 

  • ML Inference Backend: Python, FastAPI, PyTorch, OpenAI Whisper, ONNX Runtime, SpeechBrain (ECAPA-TDNN), RawNet2, WavLM.
    • 

  • Frontend & Telephony: Next.js, React, Tailwind CSS, FreeSWITCH (SIP), Kotlin (Android CallVault daemon).
    • 

  • Data & Integrity: PostgreSQL, Solidity / EVM Append-Only Audit Contract.
    • 





The Proof


    

  • Selected as a Smart India Hackathon (SIH) College Finalist at VIT Vellore.
    • 

  • Benchmarked at 1.1 s median inference latency per 3-second audio chunk on CPU (2.7x faster than real-time).
    • 

  • Validated by 50+ automated unit and integration tests covering frame reassembly, protocol invariants, and session lifecycle.
    • 

Next case studyMoss XProduction e-commerce platform serving 100+ users with sub-50ms homepage latency, Redis catalog caching, and idempotent Razorpay webhooks.