When Generic Doesn't Cut It: Building a Purpose-Built Data Pipeline on AWS

In the early days of our startup, speed was everything, so we adopted an off-the-shelf, generic SaaS offering to integrate customer data quickly. It worked well at first, but as we onboarded more clients, expanded to more sources, and encountered more edge cases, the limitations of generic ETL started to show.

In this talk, I’ll share how we transitioned from a one-size-fits-all tool to a custom-built, AWS-native ingestion system using Step Functions, Fargate, S3, DynamoDB, and RDS. You’ll hear:

• Why/When “buy” stopped working, and how we made the case for “build”
• How designing a solution that overfits our needs, in our case, beat the generic approach
• Lessons learned scaling a data platform under startup constraints
• Practical insights and a bit of cloud economics