JustGiving – Serverless Data Pipelines, API, Messaging and Stream Processing
By Richard T. Freeman
Many organisations are moving away from monolithic and batch data processing, to microservices and real-time analytics architectures. To do this they need more efficient messaging, data pipelining and stream processing patterns.
Typically such microservice, service bus or stream processing engines required running servers or clusters that have to be maintained and are always running. Serverless computing has emerged as a challenger to existing always-on virtual machines and more recent containers.
JustGiving is the world’s largest social platform for online giving, and we will share our experience in using serverless computing. In this talk we discuss how AWS serverless technologies can be used to create a simple flexible messaging system, big data pipeline and streaming analytics solution as used in part of our in-house data science platform called RAVEN.
We will walk through how to leverage AWS Lambda, Amazon S3, Amazon Kinesis, Amazon SQS / SNS, and other services to build serverless, event-driven, data and stream processing pipelines in your organisation. We will discuss different serverless patterns, best practices and lessons learned.