Flexible import of hydrometeo data

Hydrometeo data (water levels, currents, waves, etc.) is offered by many sources, and in as many formats. One approach to using such a wide range of formats in an operational application is to parse the externally available data and process it into a shared internal format for further use by the application. It is a challenge to provide an approach to configuring the parsing and processing stages that is both user-friendly and flexible.

MATROOS, operated by Rijkswaterstaat (Dutch Ministry of Infrastructure and Water Management), provides NetCDF files with model runs for expected water levels, currents, waves, etc. for up to hundreds of locations. We built a parser to select time series for the required locations from these NetCDF files. The time series are stored by a generic processor that writes them to an internal storage format.

We have made the parser flexible by allowing the configuration of locations, precision of location matching, NetCDF variable names, and the option to convert between reference planes for water level time series. The parser’s configuration is user-friendly, providing a form tailored to the properties of the parser and allowing a parser configuration to be imported/exported as JSON between different environments. The latter allows a parser to be prepared on a test/acceptance environment and easily deployed to a production environment.