The open source, parallel file system Lustre is moving fast – features and functions are being added which make it more consumable across a wider variety of HPC sites and use cases. But, crucially, due to its scalability, Lustre is also becoming popular choice for businesses with high performance needs – manufacturing, oil and gas, personalized medicine, finance and anyone with a large machine learning or business analytics program.
As its popularity grows (it is the most widely used data access approach for the world’s TOP500 fastest computers, and has been since 2005), so does the tension between users who want it to be easier to consume and those who rely on it to get bigger and faster. As a community-developed project, Lustre’s strength comes from community participation. As that community grows, a larger emphasis is placed on usability and stability features.
A common theme
Lustre is now getting a lot more attention because parallel file systems deliver the performance and capacity at large scale. Unlike regular file systems, a parallel file system allows very large clusters, or multiple systems to compute against the same data sets without storage silo sprawl or time and resource consuming data movement. As data volumes continue to grow at all levels more workflows will require a parallel file system solution to keep up. This ability to increase performance and capacity more simply using a single, centralized storage resource is very attractive, as it the fact that Lustre is an open source product with the added benefit that a fully supported version is also available from Intel who – with DDN and others – are top Lustre contributors.
New systems – such as DDN’s EXAScaler 3.0 - brings all of the new features and capabilities coordinated and managed by Intel on its Enterprise Edition for Lustre but also add unique and exclusive features such as high availability improvements, metadata scaling, project quotas back-end, and support for differentiated storage services framework, to name a few.
Keeping a security focus
Two major trends are pushing the development of multi-level security for Lustre. On the traditional HPC side, increased security requirements are emerging for workflows that historically were purely research and had no strong security requirements. For example, increased stewardship concerns about depersonalized human genetic data is driving life sciences researchers to look for stronger security, and research grant terms across US, Europe and parts of Asia now stipulate security and retention requirements for data related to the research funded. On the Enterprise side, environments that already have a need or expectation for strong security, are reaching for traditionally HPC tools to help them meet their ever-increasing performance and scale needs.
To address these requirements, DDN Developed a Multi-Level Security solution for Lustre which combines community-developed capabilities like Lustre interoperability with key technologies like Docker, Kerberos and SE Linux, and added DDN specific monitoring and performance enhancements as well as a DDN developed feature called subdirectory mounts which is what allows for a portion of the file system to be isolated – the final piece in the security puzzle.
This solution requires major efforts in development, quality assurance, compliance testing and certification, from the OS kernel level on the servers up to the user’s tools on the Lustre clients. Due the extensiveness of these requirements, we’re working together with Intel and the rest of the Lustre community to deliver functionality in a phased approach, where several software modules and components are expected to be released at a time, including in the next versions of Lustre.
Keeping up the communication
DDN contributes to Lustre by developing new features and making them publicly available and by helping to test features contributed by other members. To broaden participation in development of major initiative and tool sets, we’ve started an open source project where we share some of the projects we are currently working on to encourage collaboration. That initiative includes tools for Lustre developers, such as the “Lustre Automatic Test System” that will help developers to run regression tests in parallel on hundreds of test nodes. We are also working with a major OS distribution to provide in long term an alternative OSD backend based on BTRFS.
What lies ahead?
Knowing what the community wants and expects helps better define more use case scenarios which, in turn, improves and accelerates the development and applicability of Lustre across more research and commercial workflows. We certainly try to pay our way by contributing solid and useable tools, features and frameworks.
Laura Shepard is senior director of Marketing at DDN.