Storage PIONEERING
OPEN-SOURCE DATA STORAGE
Johann Lombardi, TSC chair of the DAOS Foundation, discusses the evolution and current status of the DAOS project.
T
he DAOS Foundation was established to maintain DAOS as an independent open- source project, encourage community
engagement, and guide its technical direction. Its primary objective is to support the continued development and adoption of DAOS across various sectors, particularly in high-performance computing. DAOS originated in 2012 as part of Intel’s Fast
Forward Storage IO program, addressing the need for significantly higher IOPS and broader vendor support. Te initial goal was clear: “We need way more IOPS. We need more vendors. Can we leverage NVMe technology?” Intel collaborated with the SGA group to integrate multi-tenant file systems with NVMe technology. By 2014, performance benchmarks revealed inefficiencies in traditional storage architectures. “Te legacy storage stack was built for disk drives. Tere was no way to optimise for NVMe because many algorithms in the kernel were designed for traditional storage mechanisms,” says TSC chair of the DAOS Foundation, Johann Lombardi. Tis prompted DAOS’s transition into an independent system optimised for NVMe. DAOS is designed as a high-performance, low-latency file system
capable of scaling to petabyte levels. Key deployments include Google’s Pilot Store, which supports petabyte-scale storage, and Intel’s Aurora system, which has 30 petabytes of capacity. Te Aurora DAOS system operates across 1,024 storage nodes, providing between 220 and 249 petabytes of usable storage. SuperMUC-NG Phase 2 supports 63,000 MPI tasks across 42 storage nodes, demonstrating DAOS’s ability to handle extensive workloads. Aurora alone hosts 177 billion files, including one reaching 8.5 petabytes in size. In late 2023, the foundation transitioned DAOS from an Apache
license to BSD+Patent to mitigate GPL-related conflicts. DAOS integrates multi-version concurrency control, optimistic locking, and flexible storage pools, ensuring adaptability in diverse computing environments. Te foundation includes members such as Google, Intel, and HPE, each contributing to technical advancements and community engagement.
40 | May/June 2025 Structured to accommodate different levels
of participation, the foundation offers Premier, General, and Associate memberships. Premier members contribute $25,000 annually and appoint voting representatives to committees. General members pay $15,000 annually and elect representatives to the Governing Board, while Associate membership is available at no cost to LF Associate Members. “General members elect three voting members that are part of the foundation,” Lombardi says, while premier members secure a board seat, influencing decisions.
Governance is overseen by a Technical Steering Committee
(TSC), which meets weekly. Te foundation maintains a total budget of $108,750, with expenditures reaching $37,300. While board operations have been fully funded, community engagement and legal expenses remain unspent, reflecting ongoing strategic allocation. Te DAOS 2.8 release marks a significant milestone, representing
the first version formally attributed to the foundation. “Te goal is to maintain DAOS as an open-source project, independent of any organisation,” says Lombardi. Modelled aſter the Linux community structure, the foundation ensures that “all community members can collaborate, contribute… the project can continue” without corporate constraints.
Advancing scalability, performance, and adaptability Architecturally, DAOS eliminates the need for a central metadata server, supporting multiple data models – files, blocks, and objects – while achieving peak speeds of hundreds of gigabytes per second and over a million IOPS per server. Built-in multi-tenancy and snapshot capabilities enhance scalability and resource management. Te Foundation prioritises rigorous soſtware testing and
integration across multiple hardware architectures, ensuring adaptability across Intel, AMD, ARM CPUs, and cloud environments. To facilitate this, the foundation established a master CI framework capable of executing integration tests across diverse platforms. “Creating a main CI, producing integration that can run tests on
www.pcr-online.biz
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52
