Snowplow Analytics is a highly-scalable system that powers structured data creation for millions of sites on the internet. Snowplow tracking is incorporated into dbt, dbt cloud, Trello, Gitlab, Citi bank, Backcountry.com, and the list goes on.
After setting up data infrastructure like Snowplow for years I've frequently found myself wishing for both less and more.
Fewer streams, fewer machines or containers to manage, fewer moving pieces to help prevent event duplication or loss, less configuration, and less in-house documentation to keep things running would be a dream.
Deployment flexibility, flexible schema storage, cost efficiencies, seamless migration between transport systems, improved utility from the data in transit, and increased visibility would also be very helpful.
Meanwhile, serverless technologies have come into their own and point the way toward a very bright data-processing future. Which is how buz.dev was born.
Between 6 River Systems and CarGurus, a very significant amount of my time over the past five years has been dedicated to data platform automation, reducing cross-team friction, and improving data quality.
Schemas have played a critical role in the process; this post outlines the why and the how. But before diving straight
Defining, implementing, and evolving events is hard.
Running event collection systems is also hard.
Buz aims to make it easy.