Workloads - Airbyte's Next Generation Job Framework.

[davinchia]

Summary

We are excited to announce a significant update to Airbyte’s infrastructure that will improve scalability and resilience - Workloads. This is automatically enabled with Airbyte Helm Chart 0.390.0.

Workloads is Airbyte's next generation Worker architecture. It is more scalable, reliable and maintainable than the previous Worker architecture. It performs particularly well in low-resource environments.

Workloads is an Airbyte-internal job abstraction decoupling the number of running jobs (including those in queue), from the number of jobs that can be started. Jobs stay queued until more resources are available or canceled. This allows for better back pressure and self-healing in resource constrained environments.

Careful users will notice two new Airbyte services 1) the Workload Api Server and the 2) Workload Launcher. Both services function together to enable Workloads. While we do not expect OSS users to interact with these services often, or at all, we still want to note the additional services to minimise surprises.

How This Impacts You

1. Operational Improvements: The Workload API allows for better tracking and management of workloads, reducing the need for manual interventions. This paves the way for future work to reduce synchronous polling.
2. Deadlock Mitigation: The introduction of back-pressure mechanisms minimises job spike deadlocks, where too many concurrently starting jobs overwhelms an environment.
3. Load Balancing: Our new load-balancing strategy allows different data-plane instances to compete for workloads, allowing for a horizontally-scaling self-healing set up. This is only relevant to Airbyte Cloud and Airbyte Enterprise users.

By separating scheduling and orchestration from data movement tasks, Workloads ensures a more scalable, resilient and flexible architecture and minimized maintenance of your Airbyte instance.

Next Steps

Nothing - this is automatically enabled for Community users as of Airbyte Helm Chart version 0.390.0.

We recommend users upgrading from existing deployments to pause existing connections before upgrading to minimise any errors.

Users interested in learning more can check out this blogpost and the public docs.

This set up has been extensively tested in Airbyte Cloud over the last 6 months and we are confident in its enhanced reliability and scalability. We expect users to not need to interact with Workloads on a day-to-day basis, or at all.

Nonetheless, we look forward to any feedback/improvement we can make!