Open Source universal Big Data and Machine Learning resource scheduler in use at Alibaba, Apple, Cloudera, Lyft, Visa, and Zillow, among others.
Wilmington, DE —16 May 2022— The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of more than 350 Open Source projects and initiatives, announced today Apache® YuniKorn™ as a Top-Level Project (TLP).
Apache YuniKorn is a cloud-native, standalone Big Data and Machine Learning resource scheduler for batch jobs and long-running services on large scale distributed systems. The project was originally developed at Cloudera in March 2019, entered the Apache Incubator in January 2020, and graduated as a Top-Level Project in March 2022.
"The Apache YuniKorn community is striving together to solve the resource scheduling problems on the cloud," said Weiwei Yang, Vice President of Apache YuniKorn. "It's really great to see the Apache Way shine in the incubating process of YuniKorn. We are lucky to have such an open, collaborative, and diverse community, which is sympathetic and cares about everyone's success. This motivates us to keep evolving and gets better every day."
Apache YuniKorn natively supports Big Data application workloads and mixed workloads, and provides a unified, cross-platform scheduling experience. Features include:
- Cloud native —runs on-premise and in a variety of public cloud environments; maximizes resource elasticity with better throughput.
- Hierarchical resource queues —efficiently manages cluster resources; provides the ability to control the resource consumption for each tenant.
- Application-aware scheduling —recognizes users, applications, and queues; schedules according to submission order, priority, resource usage, and more.
- Job ordering —built-in robust scheduling capabilities; supports fairness-based cross-queue preemption, hierarchies, pluggable node sorting policies, preemption, and more.
- Central management console —monitors performance across different tenants; one-stop-dashboard tracks resource utilization for managed nodes, clusters, applications and queues.
- Efficiency —reduces resource fragmentation and proactively triggers up-scaling; cloud elasticity lowers overall operational costs.
In addition, the Project has announced the release of Apache YuniKorn v1.0, the fifth update since entering the Apache Incubator. Improvements include:
- Decreased memory and cpu usage
- Extended metrics and diagnostics information
- New deployment model supporting future upgrades
- Technical preview of the plugin deployment mode
Optimized to run Apache Spark on Kubernetes (open source software container orchestration system), Apache YuniKorn’s performance makes it an optional replacement to the Kubernetes default scheduler. Apache YuniKorn excelled in benchmark tests with other schedulers in resource sharing, resource fairness, preemption, gang scheduling, and bin packing categories, with throughput exceeding 610 allocations per second across 2,000 nodes.
YuniKorn is in use at Alibaba, Apple, Cloudera, Lyft, Visa, and Zillow, among others.
"We're thrilled to see this offering come to fruition. Apache YuniKorn powers Apache Spark workloads for Cloudera Data Engineering (CDE), a key Kubernetes-based service supporting the Cloudera Data Platform," said Vinod Kumar Vavilapalli, Senior Director, Engineering at Cloudera and former PMC chair of Apache Hadoop. "As part of Cloudera’s Public and Private Cloud offerings, Apache YuniKorn adds tremendous flexibility and control when running large-scale analytics, enabling customers to better optimize the performance and value of their deployments."
"Apache YuniKorn is an essential infra service for bringing Big Data/ML workloads onto the cloud," said Chunde Ren, Engineering Manager at Alibaba Big Data Open-source team. "YuniKorn brings better scheduling capabilities, performance, elasticity, and usability for running workloads on Kubernetes, especially for Big Data and Machine Learning workloads, which benefits many users on the cloud. It's a great pleasure for us to have participated in the YuniKorn community since its inception and to see it grow up to be a Top-Level Project."
"Apache YuniKorn is becoming a popular choice for those who want to run Big Data workloads on Kubernetes, with more use cases developing," added Yang. "We welcome all who are interested to join the YuniKorn community and work with us on solving these challenging problems."
Availability and Oversight
Apache YuniKorn software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project. A Project Management Committee (PMC) guides the Project's day-to-day operations, including community development and product releases. For downloads, documentation, and ways to become involved with Apache YuniKorn, visit
https://yunikorn.apache.org/ and
https://twitter.com/YuniKorn_Sched .
About the Apache Incubator
The Apache Incubator is the primary entry path for projects and codebases wishing to become part of the efforts at The Apache Software Foundation. All code donations from external organizations and existing external projects enter the ASF through the Incubator to: 1) ensure all donations are in accordance with the ASF legal standards; and 2) develop new communities that adhere to our guiding principles. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. For more information, visit
http://incubator.apache.org/
About The Apache Software Foundation (ASF)
Established in 1999, The Apache Software Foundation is the world’s largest Open Source foundation, stewarding 227M+ lines of code and providing more than $22B+ worth of software to the public at 100% no cost. The ASF’s all-volunteer community grew from 21 original founders overseeing the Apache HTTP Server to 820+ individual Members and 200 Project Management Committees who successfully lead 350+ Apache projects and initiatives in collaboration with 8,400+ Committers through the ASF’s meritocratic process known as "The Apache Way". Apache software is integral to nearly every end user computing device, from laptops to tablets to mobile devices across enterprises and mission-critical applications. Apache projects power most of the Internet, manage exabytes of data, execute teraflops of operations, and store billions of objects in virtually every industry. The commercially-friendly and permissive Apache License v2 is an Open Source industry standard, helping launch billion dollar corporations and benefiting countless users worldwide. The ASF is a US 501(c)(3) not-for-profit charitable organization funded by individual donations and corporate sponsors that include Aetna, Alibaba Cloud Computing, Amazon Web Services, Anonymous, Baidu, Bloomberg, Capital One, Cloudera, Comcast, Confluent, Didi Chuxing, Facebook, Google, Huawei, IBM, Indeed, LINE Corporation, Microsoft, Namebase, Pineapple Fund, Red Hat, Replicated, Talend, Target, Tencent, Union Investment, VMware, Workday, and Yahoo!. For more information, visit
http://apache.org/ and
https://twitter.com/TheASF
© The Apache Software Foundation. "Apache", "YuniKorn", "Apache YuniKorn", and "ApacheCon" are registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other brands and trademarks are the property of their respective owners.
# # #