Highlights

Data Stewardship / Access and Services Development, Information Technology

Support for NOAA Big Data Project

CICS-NC is providing technical and scientific support to help make data from NCEI available via several cloud-based services as part of NOAA’s Big Data Project. This project established a partnership between NOAA, Amazon Web Services, Google Cloud Platform, IBM, Microsoft, and the not-for-profit Open Cloud Consortium. The goals include providing the public and private sectors with unprecedented access to environmental data and positioning data in close proximity to cloud-based high-performance computing resources, which will expand opportunities for both research and economic development.

As part of this initiative, the entire Level 2 Next Generation Weather Radar (NEXRAD) archive from NOAA’s NCEI is being transferred to both the Amazon Web Services and Microsoft Azure platforms.

On October 27, 2015, Amazon Web Services became the first of the Big Data partners to make NEXRAD data available publicly via their website. The AWS platform provides users with seamless, single-API access to both historical and real-time NEXRAD data.

NEXRAD data is now available via Amazon Web Services.

Information Technology

Current Infrastructure

• High speed networking
• 802.11 AC WIFI
• Video Communications
• 10 Gigabit Ethernet
• High-performance computing
• 45 Nodes
• 528 Cores / 3.2TB RAM
• Virtual Machine Platform
• 6 Hosts, 72 Cores, 640GB RAM
• Over 30 virtualized systems
• Data services
• Over 1.2 PB on disk
• Over 2.25 PB on Tape
• 8 Gigabit Fiber Channel

Other Recent Accomplishments

• Continuing management of system security, OS and application updates, and infrastructure upgrades
• Provided enhanced videoconferencing capabilities in support of the transition to the new, multi-location NCEI organization
• Rolled out a new scheduling tool (OpenLava) for managing high-performance computing cluster jobs
• IT staff continue to refine and improve the Institute’s high-performance computing infrastructure. Virtualization capabilities have been transitioned to a new implementation (oVirt 3.6), which allows for better stability and reliability while reducing the management demands.
• The Institute moved to using Google apps for email, which also provides access to Hangouts, Calendar, and other Google services while improving integration with NOAA and NCSU, who also use Google apps.
• Improvements to tape vaulting allows our 500-slot tape library to manage 1200 tapes.