The Accelerated Box of Flash: Accelerating Intensive Data Operations with Computational Storage

Radically new approach to storage acceleration aids data manipulation for research and discovery

San Diego, March 18, 2022

News Facts

• Los Alamos National Laboratory, nVidia (Mellanox), Eideticom Aeon Computing and SK Hynix co-developed the “Accelerated Box of Flash” or ABOF platform
• The ABOF platform incorporates accelerator technology to offload performance critical storage functions from host systems.

Data is a vital part of solving complicated scientific questions, in endeavors ranging from genomics, to climatology, to the analysis of nuclear reactions. However, an abundance of data is often only as good as the ability to efficiently store, access and manipulate that data. To facilitate discovery with big data problems, researchers at Los Alamos National Laboratory, in collaboration with industry partners, have developed an open storage system acceleration architecture for scientific data analysis, which can deliver 10 to 30 times the performance of current systems. The architecture enables offloading of intensive functions to an accelerator-enabled, programmable and network-attached storage appliance called an Accelerated Box of Flash or simply ABOF. ABOF systems are destined to be a key component of the Laboratory’s future HPC platforms.

“Scientific data and the data-driven scientific discovery techniques used to analyze that data are both growing rapidly,” said Dominic Manno, researcher with Los Alamos National Laboratory’s High Performance Computing division. “Performing the complex analysis to enable scientific discovery requires huge advances in the performance and efficiency of scientific data storage systems. The ABOF programmable appliance enables high-performance storage solutions to more easily leverage the rapid performance improvements of networks and storage devices, ultimately making more scientific discovery possible. Placing computation near storage minimizes data movement and improves the efficiency of both simulation and data-analysis pipelines.”

Scalable computing systems are adopting Data Processing Units (DPUs) placed directly on the data path to accelerate intensive functions between CPUs and storage devices; however, the ability to leverage DPUs within production-quality storage systems for use in complex HPC simulation and data-analysis systems has proven difficult. While DPUs have specialized computing capabilities that are tailored to data processing tasks, their integration into HPC systems has not fully realized available efficiencies.

The ABOF appliance is the product of hardware and storage system software co-design. It enables simpler use of NVIDIA BlueField-2 DPUs and other accelerators for offloading intensive operations from host CPUs without major storage system software modifications and allows users to leverage these offloads and the resulting speedups with no application changes. The current ABOF implementation accelerates three critical functional areas necessary to storage system function – compression, erasure coding and checksums – by applying specialized accelerators. Each of these functions represents time, expense and energy-use in storage systems. It utilizes BlueField-2 DPUs with 200Gb/s InfiniBand networking. The performance-critical functions of the popular Linux Zettabyte File System (ZFS) are offloaded to the accelerators in the ABOF. This ZFS offload is accomplished by using a new ZFS Interface for Accelerators (available at the GitHub software platform). The Linux DPU Services Module, also on GitHub, is a Linux kernel module that enables the use of DPUs from directly within the kernel, irrespective of where they exist along the data path.

The project underwent a successful internal demonstration following the January release of the ABOF appliance hardware and its supporting software. Collaborators included NVIDIA, which built the data processing units and provided a scalable storage fabric; Eideticom, which created the NoLoad computational storage stack used to accelerate data-intensive operations and minimize data movement; Aeon Computing, which designed and integrated each component into a storage enclosure; and SKHynix, which partnered on providing fast storage hardware. “HPC is solving the world’s most complex problems, as we enter the era of exascale AI,” said Gilad Shainer, senior vice president of networking at NVIDIA. “NVIDIA’s accelerated computing platform dramatically boosts performance for innovative exploration by pioneers such as Los Alamos National Laboratory, allowing researchers to drastically speed up breakthroughs in scientific discoveries.”

“The Next Generation Open Storage Architecture enables a new level of performance and efficiency thanks to its hardware-software co-design, open standards and innovative use of technologies such as DPUs, NVMe and Computational Storage.” said Stephen Bates, Chief Technology Officer at Eideticom. “Eideticom is proud to work with Los Alamos National Laboratory and the other partners to develop the computational storage stack used to showcase how this architecture can achieve these new levels of performance and efficiency. The efficient use of accelerators, coupled with innovative software and open standards are key to the next generation of data-centers.”

“Developing a cutting-edge storage product with an end-user has been a very positive experience,” said Doug Johnson, cofounder of Aeon Computing. “Working together with the technology vendors and end- user in collaboration allowed for rapid iteration and enhancement of a new type of storage product that will serve the most important goal a product can have, acceleration of the end-user’s workflow.”

“SK hynix joined this collaboration building ABOF because we understand the need for a new flash memory-based system that can accelerate data analysis,” said Jin Lim, vice president of the Solution Lab at SK Hynix. “Building on this showcase technology, we are committed to work with the collaboration partners in further defining the new architecture of the computational storage device and requirements that are critical to its best use cases.” Building on the file system acceleration project, researchers plan to next pursue integrating a set of common analysis functions in the system. That functionality would allow scientists to analyze the data using the existing programming, potentially warding off the need for additional data movement and supercomputing resources. This functionality would be specialized and tailored to the scientific community – another robust tool for tackling the complicated, data-intensive questions that underlie the challenges in our world.

About Aeon Computing

Aeon Computing is based in San Diego, California and has over 55 years of staff experience in high performance computing, enterprise computing architectures, and data storage, with a focus on architecting perfectly suited customer solutions. Their customers include academic, government, and commercial institutions that prefer high performance design over stock solutions.

About Los Alamos

Los Alamos National Laboratory, a multidisciplinary research institution engaged in strategic science on behalf of national security, is operated by Los Alamos National Security, LLC, a team composed of Bechtel National, the University of California, BWX Technologies, Inc. and URS for the Department of Energy’s National Nuclear Security Administration. Los Alamos enhances national security by ensuring the safety and reliability of the U.S. nuclear stockpile, developing technologies to reduce threats from weapons of mass destruction, and solving problems related to energy, environment, infrastructure, health, and global security concerns.

Contact

Doug Johnson
Co-founder, Aeon Computing
858.412.3810
doug.johnson@aeoncomputing.com
www.aeoncomputing.com

Follow Aeon at @AeonComputing

The post The Accelerated Box of Flash: Accelerating Intensive Data Operations with Computational Storage first appeared on Aeon Computing.

Protection of data at rest is possible in a few different ways. LUKS encryption, hardware RAID controllers with encryption and SED disks (self encrypting drives) to name a few. Here at Aeon we do *a lot* of ZFS and ZFS is happiest when it is configured to interface directly with disk drives and without a hardware RAID controller in the mix. Doing encryption of data at rest in a ZFS configuration that directly interfaces disk drives requires implementing LUKS or SED drives. This benchmark compares two identical ZFS file systems with identical underlying zpool vdevs, one utilizing standard SAS disk drives and the other utilizing SED SAS disk drives running in encrypted mode.

Drive Encryption

The SED SAS disk drives are TCG-FIPS certified drives. Specifically the drives meet the Opal-Enterprise specification from Trusted Computing Group and also comply with the US Government’s FIPS specification for encryption. For enterprise-grade drive encryption TCG-FIPS is currently the gold standard for encryption of data at rest on a disk drive.

ZFS Configuration

There are two identical ZFS file systems in this comparison; Dark Helmet and Lone Starr. If you’re catching a Spaceballs vibe here you’re not mistaken.

Dark Helmet: Four 8TB HGST He10 Helium 7200 RPM 12Gb SAS disk drives, 2+2 raidz2 zpool, compression=off, atime=off, checksum=on

Lone Starr: Four 8TB HGST He10 Helium 7200 RPM TCG-FIPS 12Gb SAS disk drives, 2+2 raidz2 zpool, compression=off, atime=off, checksum=on

1-2-3-4-5?

Like the air shield in Spaceballs, SED drives have a combination lock of sorts. SED drives require configuration with a unique private password that gets merged with the drive’s internal salt and a new unique key is created and it resides in volatile memory in the drive’s protected area. This is the encryption key for the data when written to media. When the drive is powered off or removed from its chassis the key is gone and the data on disk is unreadable. Power on or reinsert the drive into the system, write the same unique password to the drive and your data is unencrypted on the fly. The process does not decrypt the data on the entire drive, or re-encrypt it. It stays encrypted on disk and when given the proper key is decrypts on the fly when read by the host and new data written is written to media encrypted on the fly. A block range must also be defined on the drive that will fall under the encrypted data model. In this case I used an encryption password of ‘usetheschwartz’ and defined the entire block range of the drive for encryption mode. This process was duplicated on the other three drives comprising the Lone Starr zpool and file system. The following is a response from the drive when queried as to its secure mode status. The drive has been unlocked (Locked = N) and locking mode and media encryption are enabled (LockingSupported = Y) and (MediaEncrypt = Y).

Locking function (0x0002)
 Locked = N, LockingEnabled = Y, LockingSupported = Y, MBRDone = N, MBREnabled = N, MediaEncrypt = Y

We now have two operational ZFS file systems to benchmark, the unencrypted Dark Helmet and TCG-FIPS encrypted Lone Starr.

The Benchmark

The benchmark system is an Aeon Computing Eclipse HA-324 with two Intel Xeon E5-2637 v4 processors, 128GB of DDR4-2400 RAM and Intel/Avago 12Gb SAS host bus adapters. The operating system is CentOS 7.2 and ZFS v0.6.5.8-1 is installed.

I used fio for the benchmark process. The fio benchmark runtime configuration was:

size=256g, bs=1m, ioengine=libaio, iodepth=1, ramp_time=15

The Results

Write Performance

In the graph below you will see the two benchmarked ZFS file systems tracked very closely on write performance, averaging 641.8MB/second. Pretty good for a four drive raidz2 pool considering each drive has a media transfer rate of 225MB/second. I re-ran the benchmark over a dozen times and the two pools took turns being the slightly faster of the two. As you can see in the benchmark graph the TCG-FIPS encrypted drives suffer no performance degradation compared to the non-encrypted drives. TCG-FIPS drives can be used in place of standard SAS enterprise disk drives with no performance hit. But what about the reads?

Read Performance

In the graph below you will see the two benchmarked ZFS file systems tracked very closely on read performance as well. An average of 741.9MB/second. As you can see in the benchmark graph the TCG-FIPS encrypted drives suffer no read performance degradation compared to the non-encrypted drives.

Conclusion

In this case real world testing matches the product vendor sales literature. The use of TCG-FIPS hard disk drives imposes no identifiable performance penalties. Many secure facilities with important data protect their data with high walls and men with guns. If you have a need for protection of data at rest and cannot afford a high wall and men with guns then using SED (self-encrypting drive) disk drives like the HGST Ultrastar He10 TCG-FIPS in storage solutions from Aeon Computing can bring data-at-rest piece of mind to your data resource.

The post Benchmark – ZFS with TCG-FIPS encrypted disk drives first appeared on Aeon Computing.

Today Aeon Computing announced that the company will provide two Lustre file systems to enhance LANL’s technical and supercomputing capabilities. Each of the two Lustre file systems provide 14 Petabytes of data storage capacity and are capable of up to 160 Gigabytes per second of parallel access performance. According Aeon, this next generation system pushes the limits of Lustre storage performance.

Read the entire story on InsideHPC

Read More

The post InsideHPC story on Aeon Computing’s 28PB Lustre/OpenZFS deployment at LANL first appeared on Aeon Computing.

Los Alamos National Labs Selects Aeon Computing’s Next Generation of Supercomputing Infrastructure

Aeon Computing delivers two site-wide Lustre File Systems to meet existing and future demands for parallel access data storage performance for laboratory’s technical computing program.

San Diego, November 16, 2015

News Facts

• Los Alamos National Security, LLC (LANS) has selected high-performance storage from Aeon Computing to support its Advanced Simulation and Institutional Computing programs, which encompass a broad range of secure and collaborative scientific efforts that involve national security, physical and environmental sciences, cosmology, and other scientific research at Los Alamos National Laboratory (LANL).
• LANL is one of the premier supercomputing and scientific research institutions in the world. Its mission is to solve national security challenges through scientific excellence. To support and enhance its constantly evolving environment for scientific simulations and technical computing architectures, LANL sought a high-performance, open, scalable, and reliable site-wide Lustre file system that represented the best overall value.
• LANL selected Aeon Computing’s high-performance open Lustre Scalable Unit to meet the compute-intensive demands of several computing clusters by delivering two separate file systems. Each system featured 14 Petabytes of storage capacity and up to 160 GB/second I/O performance using Lustre on OpenZFS file system.
• Aeon Computing’s deployment represents the largest known ZFS-based Lustre file system that does not rely on hardware-based or proprietary RAID storage technology.

Lustre on OpenZFS: Performance and Reliability Based on Open Standards

Using Aeon Computing’s Lustre storage, LANL brings a large, reliable, and open standards-based performance tier data storage resource to its different HPC platforms, with shared access across its wide-ranging supercomputing environment.

Aeon Computing’s Lustre file system, based on its Lustre Scalable Unit, delivers 14 Petabytes at up to 160 Gigabytes per second performance over single-rail FDR14 Infiniband. Each Lustre Scalable Unit is comprised of two Lustre OSS nodes and 120 6 Terabyte Enterprise 12G SAS disk drives employing OpenZFS with raidz2 data parity protection. Additional resiliency is provided by multipath and high-availability failover connectivity, eliminating single points of failure. The two 14 Petabyte file systems deployed by LANL use 5,020 6 Terabyte disk drives combined.

Aeon Computing’s Lustre File System has the ability to handle a wide range of compute-driven storage and data I/O workloads, ranging from small jobs to jobs spanning many thousands of processor cores in parallel.

Aeon Computing, a leading HPC and Lustre file system storage vendor, has been awarded a contract by Los Alamos National Security, LLC (LANL) to provide two Lustre file systems to enhance LANL’s technical supercomputing capabilities in support of its national security mission. Each of the two Lustre file systems provides 14 Petabytes of data storage capacity and is capable of up to 160 Gigabytes per second of parallel access performance. These next-generation systems push the limits of Lustre storage performance.

The two 14 Petabyte Lustre file systems will serve the intense data IO workloads of both the facility-wide open research computing and the security-focused computing missions. Each file system is connected to the high-speed computing fabric, with 2.35 Terabits per second of fabric bandwidth using FDR14 Infiniband. The two Lustre file systems employ OpenZFS and high-availability for data integrity and redundancy. Each Lustre file system contains 40 Lustre OSS nodes, each capable of 4 Gigabytes per second of sustained data performance. The two Lustre file systems are powered by end-to-end enterprise-grade technology, including LSI/Avago 12G SAS (serial attached SCSI), Mellanox FDR14 Infiniband, HGST 12G Enterprise SAS disk drives, SanDisk 12G SAS SSDs, and Intel server technologies.

The file systems are integrated into site-wide monitoring infrastructure without the need for cumbersome or closed vendor APIs. “We were targeting an open solution that would utilize our Tri-Lab Operating System TOSS with Lustre, and provide a great performance to cost ratio,” says Kyle Lamb, Infrastructure Team Lead in the High Performance Computing Division at Los Alamos National Laboratory. “Utilizing commodity hardware and OpenZFS for RAID provides a cost-effective high performance solution with the added benefit of compression to increase available usable capacity. This allows us to provide the high density performance required for our existing clusters as well as our future Commodity Technology Systems.”

According to Jeff Johnson, co-founder of Aeon Computing, “We were able to architect a Lustre file system to meet LANL’s needs that was affordable and employed open standards in hardware and software. We were able to deliver a solution that met and exceeded LANL’s rigorous demands of multi-system HPC data IO and provide a system that was truly open.”

The Aeon Lustre Scalable Unit is a 12U system containing 120 enterprise SAS disk drives and two Lustre OSS system nodes that are fully redundant and hot-swappable, including hot-swap OSS nodes. The Lustre Scalable Unit features 12G SAS storage technology and supports single and dual raid QDR, FDR and EDR Infiniband as well as Intel’s Omnipath fabric and 10/40/100 Gigabit Ethernet. The Aeon Computing Lustre Scalable Unit is available for sale and can be used in a wide range of Lustre file system designs.

Visit Aeon Computing in booth 1746 at SC15 to see their Lustre Scalable Unit and Eclipse-NV NVMe storage system.

About Aeon Computing

Aeon Computing is based in San Diego, California and has over 55 years of staff experience in high performance computing, enterprise computing architectures, and data storage, with a focus on architecting perfectly suited customer solutions. Their customers include academic, government, and commercial institutions that prefer high performance design over stock solutions.

About Los Alamos

Los Alamos National Laboratory, a multidisciplinary research institution engaged in strategic science on behalf of national security, is operated by Los Alamos National Security, LLC, a team composed of Bechtel National, the University of California, BWX Technologies, Inc. and URS for the Department of Energy’s National Nuclear Security Administration. Los Alamos enhances national security by ensuring the safety and reliability of the U.S. nuclear stockpile, developing technologies to reduce threats from weapons of mass destruction, and solving problems related to energy, environment, infrastructure, health, and global security concerns.

Contact

Doug Johnson
Co-founder, Aeon Computing
858.412.3810
doug.johnson@aeoncomputing.com
www.aeoncomputing.com

Follow Aeon at @AeonComputing

The post Los Alamos National Labs Selects Aeon Computing’s Lustre/OpenZFS for 28PB Lustre file system first appeared on Aeon Computing.

Early in 2015, a partnership between SDSC, Aeon Computing, and Intel began working together on Data Oasis to include OpenZFS.

Now with the Lustre file system designed by Aeon, Comet users will have access to 7.6 petabytes of Lustre-based high-performance storage, with 200 gigabytes-per-second bandwidth to the cluster. It is split between a scratch file system and an allocated file system for persistent storage.

Read the entire story on ThePlatform

Read More

The post Aeon Computing – Intel – SDSC collaboration on Lustre / OpenZFS featured in a story on ThePlatform.Net first appeared on Aeon Computing.