A Tale of Two Flink Autoscalers
Netflix operates two Flink autoscalers, transitioning from a homegrown system to an open-source solution, learning valuable lessons about metrics, cost, and infrastructure maintenance, while adapting the new system to their large-scale needs.
MAIN POINTS
- Netflix initially built an in-house Flink autoscaler due to lack of mature options.
- Apache Flink community later developed an autoscaler that better handles complex workloads.
- The new autoscaler estimates true processing rates for more efficient scaling.
- Transitioning to the open-source autoscaler has reduced costs and improved resource management.
TAKEAWAYS
- Understanding metrics is crucial for effective autoscaling and debugging.
- Default configurations should be flexible to accommodate diverse job requirements.
- Adopting community solutions can be more beneficial than maintaining in-house systems.
- The new autoscaler has significantly reduced Netflix's Flink compute expenditures.