Sujay is a seasoned software engineer with expertise in building scalable data platforms and stream processing systems. Currently part of Netflix's Data Platform team, he leads efforts in Data Mesh, which handles trillions of daily events. Sujay spearheaded building real-time streaming transformations, enabling quick, data-driven decisions.
Previously, he contributed to Meta’s Data Infrastructure, working on Presto, and held roles at Microsoft on Azure Cloud and real-time messaging services.
As data volume and velocity continue to grow at unprecedented rates, organisations are increasingly turning to stream processing to unlock real-time insights and fuel data-driven decision-making. This presentation will explore how Netflix has evolved its Data Mesh platform, a distributed data architecture, to embrace the power of Streaming SQL. The session will showcase the journey from the initial implementation of Data Mesh as a data movement platform to its transformation into a comprehensive stream processing powerhouse with the integration of the Data Mesh SQL Processor.
Attendees will learn about the challenges Netflix faced with its earlier system, which relied on pre-built processors and the low-level Flink DataStream API, leading to limitations in expressiveness and a steep learning curve for developers. The presentation will then unveil the innovative solution: the Data Mesh SQL Processor, a platform-managed Flink job that harnesses the familiarity and versatility of SQL to simplify stream processing. Through practical examples, attendees will discover how the SQL Processor interacts with Flink's Table API, converting data streams into dynamic tables for seamless SQL processing. Moreover, the session will spotlight the platform's user-friendly SQL-centric features, including an interactive query mode for live data sampling, real-time query validation, and automated schema inference.
Searching for speaker images...