Center for Computing Research (CCR)

Center for Computing Research

Project

Decaf - High-Performance Decoupling of Tightly Coupled Data Flows

Exponential increases in data size and the need to distill enormous amounts of information into usable knowledge are pushing the limits of data processing in science applications. Connecting simulations with various data analyses either through storage (loose coupling) or directly using local or remote memory (tight coupling) is the way that scientists process data, but neither approach is optimal for extreme-scale science.

We are loosening the grip of tight coupling, in essence decoupling tightly coupled data flows while keeping their favorable high performance and low power characteristics. Our use of optional short- and long- term storage in the dataflow gives the best of both worlds: tight coupling whenever possible but loose coupling when usage patterns require persistent data. Our research, called Decaf, targets in situ methods and workflows, and we are evaluating our research in full and proxy applications in order to improve performance, reduce power, add fault tolerance, and enhance usability.

We are generating (1) a library of dataflow primitives, (2) a method for automatically constructing broadly applicable dataflows from the same set of primitives, designed as (3) a generic and reusable solution that other workflow and coupling tools can use.

 

The SmartBlock library is the outcome from this project.

 

Associated Software: SmartBlock reusable workflow components  

Contact: Lofstead, Gerald Fredrick (Jay), gflofst@sandia.gov
2015-8888