GUVI New Calibration and Data Products

GUVI New Calibration and Data Products

Detailed description of changes to GUVI files in version 13 is available here



 

Introduction

The GUVI products went through many changes internal to the GUVI team before the current release of the new Level 1B product, including an extensive evaluation of the version 12 products. The radiances and pointing angles are now more accurate, and there are new variables to allow easy access to information not previously contained in the L1B (counts, backgrounds, and geolocations at other altitudes). In order to reprocess the archive, we set up multiple (10) processing streams on a cluster in order get through the 10 years of data in less than one year. Please, click here for additional info about new GUVI products

GUVI L1B Versions Old Archival Version New Version
Calibration 9 13
Software 3 6
Product 8 13
 

 

Calibration changes

The list of changes made to the calibration are listed below. The GUVI team went through three intermediate calibration versions (v010, v011, v012) internally before we were satisfied with the state of the improved calibration (v013). Note that we have verified that the radiances in the UV colors found in the spectrograph mode products are identical to those found in imaging mode products.

 

   Calibration file names:
   Archival:  GUVI1_TIMED_v009.ncdf
   New:       GUVI1_TIMED_v013.ncdf

Changes to the Calibration since version 9

Change Description
121.6 nm lineshape On orbit GUVI 1216 lineshape was measured and is now used for 1216 background subtraction
130.4 nm lineshape On orbit GUVI 1304 lineshape was measured and is now used for 1304 background subtraction
Effective Slit Width The effective width of the optical slit was determined on orbit by observations of daylit earth. The slit width determination was used to calculate radiances (leading to a slight increase in radiances for a given count rate). The slight unevenness in the slit size results in a gradual change of photon counts along the spatial pixels, which then had to corrected by flat fielding (see entry below). .
LBH long radiance correction LBHL definition was incorrect - there was a mismatch between the definition on the satellite and the ground software that caused the LBHL radiances to be too high. The combination of this correction and the updated effective slit width leads overall to slightly smaller LBHL radiances
Yaw pointing correction changed Comparisons with stellar positions in limb observations showed that the yaw pointing correction needed to be changed 0.3 degrees from its value -0.8 to -1.1 degrees. The yaw direction is perpendicular to the scan direction.
Limb pixel pointing offsets Optical vignetting affects limb pixels away from the boresight pointing. All pixels pointing angles have now been corrected to give accurate tangent point altitudes.
Spectrographic Flat Fielding Although slit averaged spectrographic pixel radiances were accurate, individual pixels were not uniformly calibrated to give the right radiances, until version 13.
Long Wavelength Background Subtraction A small amount of light at wavelengths longer than 180 nm gets into the GUVI instrument and must be subtracted. Version 13 uses a more accurate algorithm developed for the SSUSI instrument that subtracts a small fraction of the total light input to the instrument. This new technique gets superior results to the original GUVI algorithm.
 

Software changes

The software has gone through many changes as well. Some of these changes were to create a new near realtime product to assist the Air Force Weather Agency (AFWA) with monitoring of Space Weather. Major changes to the software are listed in the sections below.

Adoption of NetCDF 4 files with internal compression

GUVI generated large data files to cover the information in all of the GUVI pixels. For this reason older archival Level 1B files were compressed with gzip, forcing investigators to uncompress each file before reading the variables . NetCDF 4 is internally compressed, saving storage space, while eliminating the need to decompress the file before reading. .

New product filenaming scheme

.

The new GUVI product filenames have changed to be more compatible with NASA's SPDF data service.The basic format is SPACECRAFT_INSTRUMENT-PRODUCT-MODE_starttime-endtime_orbit_versionnumbers.nc. For example, a Level 1 C imaging mode file from day 273 of 2007 starting at 1:41:46 UT and lasting until 3:18:51 UT from orbit 31465 is named TIMED_GUVI_L1C-disk-IMG_2007273014146-2007273031851_REV31465_Av13-01r001.nc

Replacement of Level 1C with Sensor Data Record (SDR) similar to DMSP/SSUSI SDRs

The Air Force 557th Weather Wing uses data made from observations of GUVI's sibling instrument, the Special Sensor Ultraviolet Spectrographic Imager (SSUSI) on the Defense Meteorological Satellite Program (DMSP). The Air Force uses the Sensor Data Record (SDR) product instead of the Level 1C previously used by GUVI. The SDR product development has now superseded the GUVI Level 1C. The SDR contains gridded, binned radiances and there is now a limb product. An advantage to doing this is that eventually analysis software can be used on either GUVI or SSUSI data products.

Spectrographic Level 1B and Level 1C

On Dec. 7, 2007, the GUVI mirror scanning motor failed, putting the GUVI instrument in a fixed disk observation position. To get the most science out of the instrument, GUVI was put into spectrographic stare mode. A new Spectrographic Level 1B product was developed so that that data could be made available for detailed studies of spectral features in the data. In addition, a level 1C product that bins 5 color spectrographic data (synthesized from color tables applied to spectrographic data during ground processing) is available, making for radiances with very low levels of uncertainty.

Changes by software version

The changes by software version are listed below. We are now in version 5.0, with version 6.0 in development

Software Version 4.0

 

  • Software was ported from Solaris to Linux.
  • Changes made to enable realtime processing capability for Space Weather nowcasting.
  • C++ overall control structure was removed and replaced with a more flexible perl script.
  • Repository moved to Subversion CM.
  • Many addtions to imaging L1B:
    • Night and auroral reference altitude geolocations.
    • Subtracted backgrounds written to L1B files.
    • Count decompression errors written in L1A.
  • Spectrographic L1B product.

 

Software Version 5.0

 

  • New Calibration file and file format.
  • Software adapted from DMSP/SSUSI to make SDR product to replace L1C products.
  • Software was written to make Spectrographic L1B products.

 

Software Version 6.0

 

  • New Calibration file.
  • Implemented limb optical vignetting correction .
  • Implemented improved long wavelength background subtraction
  • Explicit naming of Calibration file used
  • Implemented NetCDF 4 libraries with internal compression

 

 

Product Format changes to L1B

New variables have been added. We have not changed the naming of any variables in the v008 product version so that any analysis tools written with the old data can still be used. However we feel the new variables will enable more complete analyses to be done.

Level 1B Disk and Limb.

Added additional leap second to config file for leap seconds added on 1/1/2006 and 1/1/2009 - v008 data in 2006 and 2007 one second off

New Variables:

Count Information:

  • Counts - photon counts
  • CountDecompError - estimated error in compressing high count rate range

 

Backgrounds:

  • Background1216 - background counts estimated due to 1216 scattering
  • Background1304 - background counts estimated due to 1216 scattering
  • BackgroundLong - background counts estimated due to long wavelength light
  • BackgroundDark - background counts estimated due to detector noise

 

New Disk Variables New Piercepoint Geolocations at 350 km (night) and 110 km (auroral)

  • PiercePointLatitude_Night
  • PiercePointLongitude_Night
  • SolarZenithAngle_Night
  • PiercePointLatitude_Auroral
  • PiercePointLongitude_Auroral
  • SolarZenithAngle_Auroral

 

Altitudes specified as global attributes in files

  • Geolocation_Altitude_m = 150000 meters
  • Geolocation_Altitude_Night_m = 350000 meters
  • Geolocation_Altitude_Auroral_m = 110000 meters

 

 

Radiance Changes to GUVI Colors

Responsivity Changes between V013 and V009 calibrations

The responsivities used in GUVI Calibration v013 and v009 are somewhat different. Definitions of the line shapes on the detector have changed (especially 1216 and 1304), so the additional counts added up in the expanded line shapes and the different amount of backgrounds subtracted partially account for the new v13 definition of responsivities.   We present the ratio of responsivities in the table below, but show comparison of typical radiances in the two versions in the plots below. Taking the ratio of slit-averaged responsivities from v009 and v013, we can determine the ratio of the responsivities as shown in the table below.

Color Responsivity differences (cal v13/ cal v09)
1216 +8%
1304 +6%
1356 -6%
LBHS -5%
LBHL -35%

The LBHL ratio is different from the other colors, because the color table was redefined on the spacecraft in 2004, but was not changed in the V009 calibration file, so a much larger change was made to this color's responsivity.

Radiances Changes For a Typical GUVI Orbit

Radiances are plotted for a day's worth of data, orbit 22095 on Jan 6, 2006. Vertical axis in the plots corresponds to old v009 radiances, while the horizontal axis corresponds to new v012 radiances.

The following plots show radiances compared pixel by pixel for imaging data taken on Jan 6, 2006, orbit 22095. Since each pixel is calibrated separately 14 lines appear on graph, one for each spatial pixel, the slope of which is the ratio of the old responsivity to the new responsivity for that particular pixel.

The thick white diamonds are the ratios of the slit mean values, and thus the overall slit-averaged change in responsivity from v009 to v012 for a given color.

Due to the effective slit width correction, the individual pixels have a wider spread with one (pixel 0, in dark blue) having slightly larger responsivity in v012. The remaining pixels all had some level of decrease in responsivity, most pronounced for the pixels near slit center, which are colored various shades of yellow and green.

For colors longward of 1216, the revised 1216 line shape enters only in the sense that the fraction of the line contained within the color table boundaries is computed from that shape. For 1356 and the LBH bands the plots get a bit more complicated since the subtracted backgrounds create a lot of "fuzz" in these radiances. Any values that are less than zero are a consequence of that fuzz and the generally low counting statistics for those colors, which produces the clumping of points corresponding to discrete integer (raw) count values (most evident in the 1356 plot). Changes made to the 1216, 1304, and long background masks also play an increasingly large role in the radiance distributions as we go through colors 1356, LBHS, and LBHL. The "fan effect" that is most obvious in the color 1216 plot is mainly a consequence of the per-pixel responsivity ratio between the two cal files.

 

New Product: Fig1
Fig. 1
121.6 nm disk radiance data. The Old (v009) calibrated GUVI data is displayed on the vertical axis while the new (v012) data is on the horizontal axis. The slit width correction provides an overall increase of about 10% while the new 121.6 nm lineshape causes a range of different estimates of the data. At higher radiances we can see that the different spatial pixels follow different tracks. The solid green line is y=x (v012 on x axis, v009 on y axis) and the thick white diamonds are the ratios of the slit mean values, and thus the overall slit-averaged change in responsivity from v008 to v012 for a given color. The average 1216 responsivity decreased from v009 to v012, which means that the average radiance increased (by 10%), and the white diamonds lie below the y=x line.


New Product: Fig1
Fig. 2
1304 Radiance changes. Old (v009) calibrated data on the vertical axis and new (v012) data is on the horizontal axis. Note in this instance, the green line represents a 22 % increase in new (v012) over the old (v009) radiances.


New Product: Fig1
Fig. 3
135.6 nm Radiance changes. Old (v009) calibrated data on the vertical axis and new (v012) data is on the horizontal axis. Radiances have increased by about 10%, while changes to the 121.6 nm background subtraction cause some scatter in the amount of change.


New Product: Fig1
Fig. 4
LBH short Radiance changes. Old (v009) calibrated data on the vertical axis and new (v012) data is on the horizontal axis. Radiances have increased by about 20%.


New Product: Fig1
Fig. 5
LBH long Radiance changes. Old (v009) calibrated data on the vertical axis and new (v012) data is on the horizontal axis. Radiances change from a new effective slit width combined with a corrected definition of LBH long in the ground software. Dayside Radiances have decreased by about 20%.
 

Spectrographic L1B

 

The current archival spectrographic L1B is product version 10 and created with Calibration version number 11. The new product will be version 13 created with Calibration version number 13.

 

 

GUVI Spectrographic L1B Versions Old Archival New (2017) archival
Calibration 11 13
Software 4 6
Product 10 13