Climate Data Management Changes During the Last Decade

NOAA's National Climatic Data Center is the world's largest active archive of weather and climate data. The Center develops national and global data sets that have been used to maximize the resource provided by our climate and minimize the risks of climate variability and weather extremes. The Center also produces publications that describe the climate of the United States and acts as the nation's “Scorekeeper” regarding the trends and anomalies of weather and climate.

Since the 1800s, NOAA and its predecessor agencies have been managing our nation's climate data—from weather observations such as temperature, precipitation, and wind speed to satellite imagery to ice core records that reveal the climate of Earth's distant past. These data have been used in a variety of applications including agriculture, construction, energy, forestry, health, recreation and tourism, transportation, and water resources management, among other areas. The nation's climate data are critical to our modern lifestyles.

During the winter of 1995, the nation's climate archive located in Asheville, North Carolina, moved down the street—from the historic Grove Arcade Building to the new Federal Climate Complex. More than half a million magnetic tapes, 200 million manuscript records, and millions of microfilm and microfiche were relocated, along with the people, computers, and equipment that provided support needed to manage all of these data.

The new building had state-of-the-art environmental controls designed for the archival of climate data. The old building was designed in the 1920s for an entirely different purpose: to be one of the nation's first indoor shopping malls.

Federal Climate Complex in Asheville

The Federal Climate Complex in Asheville, North Carolina, home of the National Climatic Data Center and the nation's Climate Archive from 1995 to present. Click image for larger view.


More than a decade has passed since this historic relocation, and many of the same dedicated and talented employees are still managing NOAA's climate data legacy today at the National Climatic Data Center. But how the work was done in 1995, and how it is done today, are vastly different. From the time the data are received, processed, and archived, to when the data are sent out the door for access and analysis, technological advances have paved the way for more efficient and effective ways of managing the nation's climate data archive.

(top)

Receipt and Processing of Climate Data

From Paper Forms and Magnetic Tape...

In 1995, almost all climate data were received either on magnetic tape or on paper forms. Most weather observation networks recorded their reports manually on paper forms. The forms were mailed to Asheville at the end of the month and then keypunched and verified with a quality-control process. It took a minimum of two months after the observations were taken to produce the final climate products. The delays were months longer for some networks.

NOAA's remotely sensed data from satellites and radars were downloaded onto magnetic tapes and sent by U.S. mail to Asheville. The satellite data were mailed each day and usually made it to the archive within a week. Data from NOAA's NEXRAD weather radars, however, had more variable delivery. The data were not mailed by the station until the storage tape was full, which could sometimes take a month or more. For a few legacy stations which had not been modernized to NEXRAD instrumentation, the radar scopes were still being recorded as they were in the 1950s—as photographic images on 35mm film and mailed months after the data were observed. Needless to say, these various delays in receiving NOAA's climate data resulted in similar holdups in getting the data into a form where it could be accessed and used.

...to Electronic Ingest Via the Internet

Today, almost all of NOAA's climate data are received online via the Internet. Data from a few of NOAA's observing networks still arrive on paper forms and still take months to get in the archive. Progress is being made in automating these networks to provide real-time data transmission. However, for most networks, data that once took months to place in the archive are now being received and processed within minutes of the actual observation time. This allows data users to make more timely decisions that can have very far-reaching economic or social impacts.

(top)

Access to Climate Data

From Telephones and Letters...

In 1995, Web access to climate data was in its infancy. Almost all climate data requests were received by either telephone calls or letters. With about 160,000 of these manual requests in 1995—and the demand forecasted to increase—a new NOAA initiative was introduced called the National Virtual Data System. Under this initiative, new terms such as "e-Government" and "24/7" would soon be heard; these concepts involve accessing and retrieving climate information via the Web without human interface, 24 hours a day and seven days a week.

Graphs  Illustrate the Transition of Climate Data Access Toward e-Government

These graphs illustrate the transition of climate data access toward e-Government. User requests via the Internet continue to increase as manual requests decrease. Click image for larger view.


In the summer of 2001, President George W. Bush announced his Management Agenda; one of the five components of this agenda was expanding e-Government. NOAA was already in step with this vision. NOAA's climate data were accessible in a totally functioning e-Government system with an On-Line Store in 1998, a full three years ahead of this announcement. NOAA's foresight in the area of e-Government placed it in a commanding position as the Internet grew exponentially.

...to e-Government

Today, there are four million users accessing an unprecedented amount of climate data via the Internet. Almost 20 terabytes of data are downloaded every month, yet staffing needed to service user requests has dropped by more than 50 percent.

Even though the large majority of those accessing data use e-Government, some still prefer telephone and letter requests for data. This was made very clear at the 2003 and 2005 NOAA Data Users Workshops: One of the top recommendations made to NOAA was to allow customers the option to speak or write to a person for climate data access.

(top)

Archiving Climate Data

From Megabytes and Magnetic Tapes...

Rows and rows of magnetic tape cartridges in the mid-1990s

Rows and rows of magnetic tape cartridges in the mid-1990s. About half a million cartridges contained in-situ weather observations, satellite, and radar data.


NOAA's climate data were archived on magnetic tape, paper, and film in 1995. There were more than half a million magnetic tapes, and most of these tape units held only 25-150 megabytes. There were about 228 million megabytes (228 terabytes) of data at that time.

Almost all of NOAA's environmental satellite data, which represented about 75 percent of the digital data, was not backed up. This was a major issue since it placed these data at great risk in the event of a disaster.

NOAA's non-digital climate data contained 200 million paper records and were stored in a room that was bigger than a football field. There were also 2 million microfiche and 100,000 reels of 16mm and 35mm film that contained photographic images of weather records. The supporting metadata that described NOAA's climate data were largely produced and distributed to customers in paper-based catalogs.

...to Petabytes and Robots

One of several tape robotic systems used to manage NOAA's climate data  in 2006

One of several tape robotic systems used to manage NOAA's climate data in 2006. Tape cartridges are on the left and right walls with the robot in the center. Click image for larger view.


Numerous advances in technology during the last decade have resulted in more efficient management of NOAA's climate data archives, most notably storage media. By the summer of 2006, NOAA's climate archive had increased by a factor of 10 and total storage was 2,296 terabytes (~2.3 petabytes). The capacity of the media used to store these data increased by a factor of 3,000 compared to 1995. As a result of this progress, the rows and rows of magnetic tapes that were prevalent in the 1990s slowly disappeared; they were replaced by tape robotic systems.

Graph Showing the Increase in NOAA's Climate Data Archive Since  1995

This graph shows the increase in NOAA's climate data archive since 1995. The archive contains in-situ, satellite, radar, and modeled data. Click image for larger view.


For the climate data stored on paper, NOAA's public-private sector partnerships via the Climate Data Modernization Program preserved millions of manuscript records through scanning and placing the images online to allow easy access.  Many were keypunched and added to the digital archives. NOAA’s climate data Web systems also were enhanced with additional metadata, allowing users to more easily find the climate data they need.

(top)

Analysis of Climate Data

From a Single-discipline, National Focus...

In 1995, a typical climate data analysis focused on a single data set usually addressing an important national issue. Those analyses that were global in scope concentrated on a limited number of parameters that were observed by surface-based instruments measuring quantities such as atmospheric temperature, precipitation, and sea surface temperature. It took weeks, months, or sometimes even years to get the final, high-quality data that were needed to produce these analyses. Studies that involved climate models or remotely sensed data from satellites and radars were not common, and most were limited to regional analyses since they required the most powerful computers to process these massive amounts of data.

...to Comprehensive Global Investigations

Today, computer processing power has increased dramatically, providing the capability to process climate data at speeds that were unheard of just a decade ago. The Earth is an integrated system requiring integration of data from many disciplines to answer scientific questions. However, only recently has processing power allowed for such an integrated analysis of different data types. As an example, new products for parameters such as precipitation and sea surface temperature are now developed using observations from rain gauges or thermometers blended with data from satellites, weather radar, and climate models.

An example of a blended analysis of rainfall

An example of a "blended" analysis of rainfall. Measurements at weather stations with rain gauges have been integrated with NEXRAD radar data to produce a multi-sensor mosaic product. Click image for larger view.


Also, new climate networks have been introduced, such as the U.S. Climate Reference Network, which will provide a first-class observing network for the next 50 to 100 years. This network will serve as a benchmark for climate monitoring. Coupled with the long-term historical records, this will improve the ability of decision makers to develop policies about programs impacted by climate variability and change.

(top)

Conclusion

NOAA's climate data legacy is considerable and is vital to our nation's economic and societal well-being. There is growing concern that the climate is changing, causing an increased awareness of climate impacts on humans and the impacts of human activity on climate. It has long been known that climate does not begin and end at political boundaries. Progress is being made on major international programs such as the Global Earth Observation System of Systems that will help us span political boundaries while balancing social, economic, and environmental health.

Dr. Thomas Karl, current Director of the National Climatic Data Center, surmises

All Earth processes are linked and a small change in one process may have a dynamic effect on another. Having integrated access to global data that spans these Earth process disciplines will enable scientists to better answer climate change questions and address such critical societal issues as drought, rising sea level, natural disasters, human health, and countless others.

NOAA's climate data are providing vital information to these contemporary societal issues and unquestionably many more issues not even envisioned today that will arise in the future.

Contributed by Pete Steurer, NOAA's Environmental Satellite, Data, and Information Service

(top)

Works Consulted

National Climatic Data Center. (2001). Celebrating 50 Years of Excellence. [Asheville, NC] 28 p.

Steurer, P.M. (1995). Data rescue at the National Climatic Data Center, progress and challenges. [Asheville, N.C.: National Climatic Data Center], 1995, 25 p.

(top)