Available for new opportunities

Spoorthi
Basu

Software Engineer · Distributed Systems · Real-Time Data Infrastructure

I build distributed systems that move data at scale. Real-time pipelines, high-throughput APIs, and the infrastructure that keeps them reliable.

Speaker · Data Streaming Summit 2026 InfoQ Published Author Apache Flink CDC Contributor 100M+ Events / Day
Get in Touch Resume

Indianapolis, IN · Open to relocation

Spoorthi Basu
5+
Years of
experience
100M+
Events processed
per day
99.99%
Uptime
delivered
5K+ RPS
APIs
engineered

What I Build

Real-time Stream Processing

Apache Kafka + Flink pipelines ingesting hundreds of millions of events daily into schema-driven, analytics-ready Iceberg tables on S3.

KafkaFlinkIcebergAvro

Distributed Systems

Fault-tolerant, multi-region Java microservices with circuit breakers, event partitioning, and 99.99% uptime across sustained production workloads.

JavaResilience4jMulti-region

High-Scale API Engineering

Spring Boot REST APIs handling 5K+ RPS with Redis cache sharding that cuts database load by 50%, built for reliability under sustained traffic.

Spring BootRedisREST

Data Infrastructure

End-to-end data platforms: event modeling, schema consolidation, download APIs, and analytics infrastructure that scales to hundreds of millions of events per day.

AWS S3DynamoDBSchema Design

About Me

I'm a Software Engineer with 5+ years building the infrastructure that makes data move: reliably, at scale, and in real time. At Genesys, I've engineered systems processing hundreds of millions of events per day, APIs that handle 5K+ requests per second, and testing infrastructure that went from zero to production-grade in weeks.

My work spans the full data path, from Kafka topics and Flink jobs to Iceberg tables and the REST APIs that serve data to end users. I care deeply about correctness, fault tolerance, and the operational rigor that makes systems boring, which is exactly what production needs.

I recently built the company's first end-to-end data validation framework, now adopted across three teams. I've also published in InfoQ on schema proliferation in Kafka and Flink pipelines, and I'm an active contributor to Apache Flink CDC. This October I'll be speaking at Data Streaming Summit 2026 in San Francisco on why the pipelines behind AI agent memory fail silently.

Currently at
Genesys
Software Engineer
Mar 2021 · Present
Core Stack
JavaApache KafkaApache Flink Apache IcebergAWSRedis Spring BootDockerTerraform

Speaking & Credentials

Upcoming Talk Accepted

Your Agent’s Memory Is Only as Good as the Pipeline Behind It

Data Streaming Summit 2026 Oct 7–8, 2026 San Francisco, CA

I don’t build agents. I build the pipelines that feed them. Durable memory and continuous context are CDC problems wearing new clothes. Kafka into a vector store, Flink into a lakehouse, a change log materialized into something an agent reads. That path breaks without ever tripping an alarm. Exactly-once is a guarantee about processing, not about what ends up in your table. An agent doesn’t crash on wrong memory. It answers confidently, and you never trace it back.

  • Selected through the open Call for Proposals
  • Correctness checks across Kafka, Flink, and Iceberg
Data Streaming Summit

Education

M.S. in Computer Science
Cal Poly Pomona
2018 – 2020 · GPA 3.66
B.S. in Computer Science
Dr. Ambedkar Institute of Technology
2014 – 2018 · GPA 4.0

Publication

Schema Proliferation in Kafka and Flink Pipelines
InfoQ · 2026

Explores schema proliferation in event-driven systems and presents a discriminator-based consolidation approach for scalable Kafka and Flink pipelines.

  • #1 Top Article in InfoQ's weekly Round-Up
  • 550K+ monthly readers
  • Reviewed by a Java Champion
Read on InfoQ

Open Source

Apache Flink CDC Contributor
apache/flink-cdc · 2026

Fixed silent data duplication in the Iceberg sink during in-checkpoint schema changes. Also fixed a JobManager OOM on a 2.5B-row MySQL CDC table retaining ~300K snapshot splits.

  • Iceberg sink fix merged, shipping in 3.7
  • JobManager OOM fix open, in review
  • Reviewed by a committer
View Pull Requests
Under Review
  • Audit-Preserving Compaction for Merge-on-Read TablesPVLDB
  • The History Was Already ThereCACM Practice

Work Experience

Software Engineer

Genesys · Cloud Contact Center Platform
Mar 2021 · Present
  • Built Flink pipelines behind the company’s first data lakehouse, turning company-wide Kafka streams into schema-driven Iceberg/S3 datasets at hundreds of millions of events/day; recognized with the company’s Go Big Award.
  • Authored public Genesys Cloud Data Lakehouse dataset documentation for schemas I built, including agent presence definitions, conversation attribute updates, external contacts, and outbound campaign contacts.
  • Architected and built the company's first end-to-end data validation framework, asserting correctness, completeness, compliance, and query performance across Kafka, Flink, and Iceberg; adopted by 3 teams, run continuously.
  • Scaled fault-tolerant Java/Kafka microservices to 10M+ events/day, cutting latency 25%.
  • Engineered customer-facing REST APIs at 5K+ RPS with Redis cache sharding, halving DB load.
  • Stood up org-wide integration testing 0→1 (LocalStack/AWS), cutting test creation to <1 min for 5+ teams.
  • Delivered 99.99% uptime via multi-region deployment and circuit breakers (Resilience4j).
  • Drove 10+ customer-facing features with Product and QA, growing daily active users 25%.
  • Owned on-call triage for production data services with a sub-5-min P0/P1 response SLA, fixing incidents same day.
  • Mentored a junior engineer and ramped teammates onto the export project, halving onboarding time.

Software Engineer

Coding Minds, Inc · Ed-Tech Platform
Jul 2020 · Feb 2021
  • Developed academic system (React/Java/Node.js) deployed on Heroku, serving 500+ daily active users.
  • Built RESTful APIs (Java/Spring) with MySQL CRUD operations, achieving 95% test coverage.
  • Led full SDLC from requirements to deployment using Agile/Scrum, delivering 3 major releases.
  • Enhanced React performance via memoization and lazy loading, improving page load speed by 30%.
  • Implemented automated testing, catching 20+ critical bugs pre-production.

Selected Projects

Featured Project

Kafka Flink Schema Consolidation

Reference implementation for discriminator-based schema consolidation in Kafka and Flink pipelines. Collapses twelve schemas into one consolidated Avro record, enabling single filtered queries over Apache Iceberg on S3. Companion to the InfoQ article on schema proliferation.

Apache FlinkApache KafkaApache IcebergAvroJava
GitHub InfoQ Article
ConsolidatedRide.avsc 
// Discriminator-based union field
{
  "name": "standardRideAttributes",
  "type": ["null", {
    "type": "record",
    "name": "StandardRideAttributes",
    "fields": [
      { "name": "vehicleClass",
        "type": "string" },
      { "name": "surgeMultiplier",
        "type": "double" }
    ]
  }],
  "default": null
}
Research Artifact

MOR Faithfulness

When a CDC stream is materialized by a merge-on-read table, is the “current” row per key actually correct, and can you tell from the physical table alone? A Lean 4 and Mathlib development proves the ordering value must linearly extend logical version order, and that in general you cannot tell. Compaction then destroys the evidence: a read-only checker certified all 5,440 flagged violations as clean once it ran. An audit-preserving compaction mechanism recovers all 5,440 with no false positives, in 657 lines that are off by default and change nothing in the reader or the format spec. Ships with the checker and a FLINK-38450 reproduction. Companion artifact to a paper under review at PVLDB.

Lean 4Apache IcebergApache HudiDelta LakePython
GitHub
MorFaithful/*.lean 
/-- Def 6. Faithful: the materialized set is
    exactly the current version. -/
def Faithful : Prop :=
  distinct M.Zphys = Finsupp.single M.cur 1

/-- Def 7. LinearExtension: seq strictly
    increases along logical order. -/
def LinearExtension : Prop :=
  ∀ i j, i < j → M.s i < M.s j

/-- Corrected MAIN. -/
theorem prefixFaithful_iff_linear
    (inj : Function.Injective M.d) :
    M.PrefixFaithful ↔ M.LinearExtension :=
  ⟨M.linear_of_prefixFaithful inj,
   M.prefixFaithful_of_linear⟩

-- No `sorry`. 21 theorems audited
-- against the 3 standard axioms.

Kafka Flink Audit Trail

Flink pipeline writing profile change events to an append-only Iceberg table. Old values are rebuilt at query time with LAG and window functions instead of stored in previous-value columns. Removing 67 previous-value fields cut schema size 48% and Avro payloads 38%. Companion to an article under review at CACM Practice.

FlinkKafkaIcebergSQL

Ecommerce Order Processing

Order microservices using Spring Boot and JPA with PostgreSQL. Swagger docs, Docker containerized, Log4j logging.

Spring BootPostgreSQLDocker

Hair & Skin Segmentation

Deep autoencoder using U-NET for hair/skin segmentation with Keras, tested on Celeb-A dataset with data augmentation and transfer learning.

KerasU-NETNumPy

Skills & Technologies

Streaming & Data

Apache Kafka Apache Flink Apache Iceberg Avro Schema Registry CDC Exactly-Once Semantics

Languages

Java Python JavaScript SQL Lean 4

Cloud & Infrastructure

AWS Docker Terraform Kubernetes CI/CD

Frameworks

Spring Boot Spring Cloud React gRPC

Databases & Caching

DynamoDB Redis PostgreSQL MySQL S3

Testing & Observability

JUnit 5 Mockito Testcontainers LocalStack Sumo Logic New Relic Prometheus

Community & Mentorship

Keys to Success

Cal Poly Pomona · Incoming Graduate Student Welcome

  • Talked 50+ incoming students through my path into Computer Science
  • Focused on the transition from undergraduate to graduate school
Cal Poly Pomona

Judge, Game Gala 2021

K–12 Coding Competition

  • Evaluated 20+ K-12 developers on digital game projects
  • Provided feedback on code quality and game performance
  • Helped select the competition winner
Game Gala 2021