Integrating the Pieces: NOAA's Data Management Challenges
NOAA manages environmental data and services for a wide range of activities that touch the lives of Americans every day. NOAA's vision for the future is to integrate the many pieces of Earth's environmental puzzle in an informative way. NOAA expects that the amount of environmental data collected each year will double because of improved instruments and other technologies.
As a technologically savvy society, we hardly pay attention anymore when a new satellite is launched. However, scientists use the data provided by environmental satellites and other observing systems to create products such as satellite images seen on televised weather forecasts, sea surface temperature maps, or harmful algal bloom forecasts. These tools are vital for us to carry on in our daily lives.
The newest Geostationary Operational Environmental Satellite, GOES-13, was launched on May 25, 2006. Click image for larger view.
On May 25, 2006, the newest satellite in the Geostationary Operational Environmental Satellite (GOES) system was launched. This satellite, designated GOES-13, is aimed at providing information that scientists need. Like other satellites in the GOES system, GOES-13 orbits about 22,370 miles above the Earth while "hovering" over one location on the Earth's surface.
The GOES system provides data and information that help scientists forecast severe weather conditions such as tornadoes, flash floods, hail storms, and hurricanes. Additionally, the GOES detect solar storm activity, relay distress signals from emergency beacons, monitor the oceans, and scan the landscape for the latest drought and flood conditions.
As GOES and other data collection technologies improve, the volume of information that NOAA will need to manage will dramatically increase. This article looks at how NOAA will meet the challenge of processing, delivering, and storing this information in the future.
It's All About Data
We are accustomed to seeing the satellite images of Earth from space on television, with swirling white clouds and beautiful blue seas. But satellites are more than hunks of metal with a camera drifting in space. These spacecraft carry sophisticated instruments and sensors that are constantly streaming critical data to collection stations on Earth. Properly storing and maintaining the data are key to preserving it for future use.
NOAA receives data not only from its satellites, but also from other observing systems on ships, planes, buoys, subsurface platforms, and land-based observation stations. All of these data provide the information scientists need to forecast environmental conditions in the future. When these types of data are collected over many years, they have the potential to help us understand Earth systems in new and useful ways.
NOAA currently has three world-class data centers: the National Oceanographic Data Center, the National Climatic Data Center, and the National Geophysical Data Center. These centers collect, archive, and manage data gathered from all types of observation platforms. Researchers, scientists, and the general public can access these data to help monitor, access, and predict climate, solar, terrestrial, and oceanographic conditions.
NOAA expects that the amount of environmental data collected each year will double because of improved instruments and other technologies. How will NOAA manage this data challenge of the future?
A Mission-Oriented Assessment
NOAA has adopted five primary goals to guide its future. Each of these five goals represents one of the environmental areas for which NOAA has a public responsibility and one piece of NOAA's data management puzzle. These goals motivate NOAA data management activities, and, at the same time, data management is an essential part of working toward these goals.
Together, all of the information collected under each one of these goals helps NOAA advance our knowledge of the Earth and deliver products and services that save lives and property.
Ecosystems
Click image for larger view.
NOAA is the lead federal agency in the study and preservation of coastal and ocean ecosystems. NOAA's goal is to achieve a balance between the use and protection of coastal and ocean resources. NOAA provides data such as sea surface temperature and harmful algal bloom information to try to achieve this balance. This will ensure the health and vitality of coastal and ocean resources for the present and the future.
Climate
Click image for larger view.
NOAA conducts visionary research that addresses climate variability and long-term climate change by using data such as global and ocean surface temperatures. NOAA is committed to delivering climate information, thus enabling society to better understand and respond to changing climate conditions. Understanding climate requires long-term temperature records, rain fall data, carbon dioxide levels, and other observations. NOAA is the lead agency for preserving long-term climate data.
Weather and Water
Click image for larger view.
NOAA provides observations, analyses, predictions, and warnings for a range of weather and water conditions, such as floods, droughts, hurricanes, tornadoes, and tsunamis. In addition, NOAA has activities relating to water supply, hazards, air quality, and space weather.
Commerce and Transportation
Click image for larger view.
NOAA's products and services are essential to the safe and efficient transport of people and goods on the water, in the air, and on the land. This includes using data such as satellite images and wind speed and direction to create accurate marine forecasts, provide advanced navigational charts, respond to hazardous material spills, and assist in search and rescue efforts.
Mission Support
Click image for larger view.
NOAA's efficient data collection support structure is necessary to achieve the other mission goals. This structure includes NOAA's data-collecting platforms (ships, aircraft, satellites, etc.), information technology systems, and management and administration.
The Data Challenges
Recently, NOAA assessed its data management successes and challenges. Using the mission goals as a guide, the assessment team discovered that the increasing data volume was the most pressing concern. Data volume in turn drives other data challenges such as data accessibility, data integration, metadata, and data rescue.
Click image for larger view.
Data accessibility is the ability to access information in real time and from long-term archives at various levels of technical need and ability.
Data integration involves enabling the merging of data from different observational domains, environments, formats, and time scales in new and innovative ways.
Metadata is data about data. It is the "where, when, how, and why" of data and information. Knowing the background information about data collection is essential to using that data effectively over time.
Data rescue means collecting data at risk to fill in gaps in the overall data record. As an example, data collected prior to the computer age are very useful when combined with new data to show long-term changes to Earth's environment.
Each one of these challenges must be addressed in order to manage the increasing volumes of data that await NOAA in the near future.
It's Also About People
Aerial phtograph of Grand Isle, Louisiana, on August 31, 2005, shortly after Hurricane Katrina made landfall in the area. Note the help signs in the mud. Click image for larger view.
While working on future organizational goals, NOAA learned that achieving these goals is tied not only to NOAA data, but also to the people who use that data.
NOAA's data products and services touch the lives of Americans on a daily basis, in ways that most people don't realize. Societal benefits that come from these calibrated and quality-controlled data include safer navigation, better forecasting, cleaner coasts, disaster mitigation, and many others.
Current events, both natural and political, create their own directions in data collection, data maintenance, and data integration. The needs of the people set the direction, and NOAA responds to the people's needs.
The Superdome in New Orleans, Louisiana, sustained extensive damage August 31, 2005, when Hurricane Katrina ravaged the area. Click image for larger view.
For example, the day after Hurricane Katrina made landfall on the Gulf Coast, NOAA began aerial photography flights over the affected areas. For nine days, the NOAA Cessna Citation aircraft flew two to three missions each day, only stopping to refuel. Nearly 7,000 aerial images were produced from these missions. From September 1 – 7, 2005, about five million photos were downloaded from the Web site of NOAA's National Geodetic Survey each day, resulting in more than 27 terabytes of data being delivered to meet the intense public need.
Long-Term Data Management and Delivery
NOAA’s long-term (10-year) objective is to integrate data management and delivery to create a more cohesive system. This is a lofty goal given NOAA’s extraordinarily diverse data programs and observing systems that monitor everything from the ocean floor to the surface of the sun. It follows that the data types and formats are just as diverse.
NOAA’s long-term goal to create an integrated, cohesive data management system is a response to customer needs for data that bridge scientific disciplines; timescales; and federal, state, and local agencies. NOAA's recent assessment of data challenges reminds us that we have been working toward integrating services for years now, yet still have much work to do.
There is no quick way to leap from “here” to “there.” However, NOAA has made great progress in coordinating activities at many levels. For example, NOAA has a robust, secure information technology infrastructure and is developing exciting new uses for Web-based technology for distributing and archiving its data. Furthermore, NOAA-wide data standards will ensure that data collected by different components of NOAA for various purposes can be integrated and used together.
NOAA—and our public and private sector partners—continue to move toward the vision of an integrated global observation and data management system. This is an end-to-end data management process within each NOAA mission goal, connecting to an all-NOAA, inter-functional system. This is a strategic and progressive approach.
In the world of data management, a 10-year objective is difficult to envision because technology changes quickly. There will be advantages in the future that we do not have today, and the nation’s needs for data and information will be moving in new directions that are hard to predict.
What we do in the next 20 years may determine Earth’s environmental destiny. What we do in the next 10 years may determine what is possible in the next 20 years. What we do today will provide a clearer focus for tomorrow.
Contributed by Joy Ikelman and Susan McLean, NOAA’s National Environmental Satellite, Data, and Information Service
Puzzle pieces designed by Mike Walker, NOAA’s Office of the Under Secretary