Accelerating the Lagrangian particle tracking of residence time distributions and source water mixing towards large scales

Travel/residence time distributions (TTDs/RTDs) are important tools to evaluate the vulnerability of catchments to contamination and understand many aspects of catchment function and behavior. In recent years, the calculation of TTDs/RTDs based on the Lagrangian particle tracking approach together with the integrated hydrologic modeling has become a popular counterpart to analytical approaches and lumped numerical models. As global water availability becomes more stressed due to anthropogenic disturbance and climate change, the requirement of large-scale and long-term simulations for TTDs/RTDs further pushes the high computational costs of Lagrangian particle tracking. Hence, speeding up the Lagrangian particle tracking approach becomes an important barrier to advancement. In this study, we accelerate the Lagrangian particle tracking program EcoSLIM, using a combination of distributed (e.g. MPI) and manycore accelerator (CUDA) approaches for large-scale and long-term simulations. EcoSLIM was developed to be seamlessly paired with the integrated ParFlow.CLM model for calculations of transient RTDs and source water mixing and was originally developed using threaded OpenMP. This work extends this implementation to compare combinations of MPI, CUDA and OpenMP. Of these combinations, the OpenMP-CUDA parallelism performed the best moving from single-GPU to multi-GPU. The multi-GPU shows strong scalability which becomes increasingly efficient with more particles, demonstrating a potential feasibility for regional-scale, transient residence time simulations. This work largely improves the computational capability of EcoSLIM, and results also show the advantages of using GPU to traditional parallel-APIs (application programming interfaces) and its potential to widely accelerate the next generation programs in subsurface environment modeling.