Scott Stevens: Highlights 2015

Data Stewardship / Access and Services Development, Information Technology

Support for NOAA Big Data Project

CICS-NC is providing technical and scientific support to help make data from NCEI available via several cloud-based services as part of NOAA’s Big Data Project. This project established a partnership between NOAA, Amazon Web Services, Google Cloud Platform, IBM, Microsoft, and the not-for-profit Open Cloud Consortium. The goals include providing the public and private sectors with unprecedented access to environmental data and positioning data in close proximity to cloud-based high-performance computing resources, which will expand opportunities for both research and economic development.

As part of this initiative, the entire Level 2 Next Generation Weather Radar (NEXRAD) archive from NOAA’s NCEI is being transferred to both the Amazon Web Services and Microsoft Azure platforms.

On October 27, 2015, Amazon Web Services became the first of the Big Data partners to make NEXRAD data available publicly via their website. The AWS platform provides users with seamless, single-API access to both historical and real-time NEXRAD data.

NEXRAD data is now available via Amazon Web Services.

NEXRAD data is now available via Amazon Web Services.

Reference Environmental Data Records

  • Initiated work on a project to move the quality control of Hydrometerological Automated Data System (HADS) data from research to operations. CICS-NC work focuses on streamlining and automating existing algorithms so that they can be run autonomously and in an archive mode on archived data. Accomplishments include development of a decoder to read incoming HADS data in Standard Hydrometeorological Exchange Format (SHEF) and replicating existing metadata checks. The metadata checks have been streamlined and rewritten to run on a variety of machines.
  • Work continues on a joint project with the NOAA National Severe Storms Laboratory (NSSL)/Cooperative Institute for Mesoscale Meteorological Studies (CIMMS) in Norman, OK, to apply the National Mosaic and Multisensor Quantitative Precipitation Estimate (NMQ/Q2) algorithms to the entire archive of NEXRAD data (1997–2011). This reanalysis produced a suite of gridded precipitation products at a far finer temporal frequency (five-minute) and spatial scale (0.01°/1 km) than was previously available.In addition to providing the data via cloud providers as part of the “Big Data” effort described earlier in this report, efforts are underway to introduce the completed dataset into the NCEI archive. A process is being finalized which will attempt to fill in the few remaining gaps in the record. The dataset is already being used in some applied science, including the use of hourly rain rates to study fatal traffic accidents, in the SERDP heavy precipitation project.

Comparison of spatial resolution in the operational Stage IV gridded precipitation product (left), and the NMQ/Q2 reanalysis product (right).