The Fast Forward Storage and IO (FFSIO) Stack project completed phase one and was preparing for the working prototype phase two. However, many of the assumptions and design descisions made are likely to lead to problems leaving an unworkable system. We focused on doing a top-to-bottom evaluation of the design and proposed many solutions or pointed out unsolved problems that were potentially overlooked. The FFSIO team welcomed the effort as it helps them be more successful. Should phase two be funded, we will help incorporate the suggestions delivering an exascale parallel file system that meets our needs. This project has its own page at ffsio.
This was started Summer 2007 during an internship at Oak Ridge National Laboratory with Scott Klasky. His existing effort trying to develop a portable, high performance, next generation IO library needed an injection of fresh perspectives. Working together, we abandonded the existing work replacing it with the award winning library availble today.
The key observations driving ADIOS were the frequent changes required for optimal performance as science simulations move from platform to platform and that the dominant two libraries, HDF5 and PnetCDF/NetCDF, were desgined at least a decade earlier in a very different hardware environment.
We focused on fully exploiting the parallel file system with an easily replaceable transport layer to change how the data is managed, be it to disk or to some other purpose, through a simple API nearly 100% POSIX.
My particular project is investigating the possibilities afforded through dynamic code generation and placement based on declarative descriptions, particularly in the High Performance Computing (HPC) domain. These operations are intended to simplify offline data processing operations and to reduce the need for non-computation code within the host application.
The current phase of this work employs an XML format listing source format, destination format, and transformations. The associated code parses these files from multiple clients recognizing and merging identical transforms and reordering generating a fan-out tree of operations to incrementally create each destination type from a common source type. Ultimately, the XML used to describe the source and destination formats and the transformation operations will be easily generated from a tool freeing the programmer from the burden of writing the conversion code themselves while still obtaining near optimal to optimal performance of the transformation with least impact on overall system performance.
This work ties neatly with my funding source by providing semantically rich data storage and retrieval operations through the storage system interface of HPC systems. Building interesting and useful examples on top of the LWFS tools affords exploration of realistic HPC-related application operations.
Other applications being considered and investigated are the airline ticket pricing domain and healthcare IT.