Creating Master Reference Files

Master calibration reference (MasterCal) files are derived from calibration or science observations, and are used to remove the various components of the instrument signature from the data. Calibration exposures may combined or characterized to create a calibration, so that it may be applied when science data are processed. Some other instrument calibrations (e.g., slit mask definition files) have already been created for you by Gemini scientists, and are distributed with the gemini.gmos package software.

Caution

The code examples in this chapter for creating MasterCals are for IRAF users. Code examples for PyRAF users are integrated into the various PyRAF tutorials (see in addition SQL-Based File Selection). However, the conceptual descriptions in this chapter are relevant for all users.

Note

In the code blocks below the commands are intended to be executed within the IRAF command language (cl), unless otherwise noted. The traditional prompt (e.g., “cl> ”) has been omitted to make it easier to cut-and-paste commands into your own IRAF session. Terse comments are prepended with a pound sign (“#”, or hashtag).

This chapter summarizes how to create the reference files that will be needed for your data reduction.

Create File Lists

The first step is to create lists of input files for MasterCal creation.

Caution

The PHDU must be explicitly inherited to access keyword values (this is because Gemini sets INHERIT = 'F' in the raw extension HDUs). Thus, an extra step must be performed, using gemextn, to generate the file list. See the IRAF FITS Kernel User’s Guide for details.

# Bias exposures must match the RoI and CCD binning:
cd /path/to/science_files/raw
string biasSelect = "obstype=='BIAS' && obsclass=='dayCal' && detrO1ys>1024 && ccdsum=='2 2'"
string omit = "exten,index,kernel"

# Select using information from both PHDU and HDU-1
hselect ("*.fits[1,inherit=yes]", "$I", biasSelect, > "bias_tmp.txt")

# Now remove the IRAF index and kernel parameters from the file names
gemextn ("@bias_tmp.txt", omit=omit, outfile="biasfiles.txt")
rename ("biasfiles.txt", "../")

Similar steps apply for imaging flat-fields, one per filter:

# Identify flat exposures:
# Note: IRAF refers to the OBJECT keyword with the special name: "i_title"
string flatS1 = "i_title=='Twilight' && obsclass=='dayCal' "

# Flat-fields must match the RoI, CCD binning, grating, and aperture.
s2 = "&& detro1ys>1024 && ccdsum=='2 2' && grating='MIRROR' && maskname='None' "
s1 = flatS1 // s2

# Generate lists of filenames using above criteria and the filters of interest.
# For the r' filter;
hselect ("*.fits[1,inherit=yes]", "$I", (s1 // "&& filter2=='r_G0326'"), > "flt_tmp.txt")
gemextn ("@flt_tmp.txt", omit=omit, outfile="flt_r.txt")
rename ("flt_r.txt", "../")

The selection criteria for the arcs is similar to the bias exposures, except for MOS and IFU modes where the slitlets or fibers are split into separate spectra.

# Identify Arc exposures:
string arcS1 = "obstype=='ARC' && obsclass=='dayCal' && detro1ys>1024 && ccdsum=='2 2' "
#s1 = arcS1

# For MOS and IFU modes:
# Arcs for MOS or IFU must match the grating, central wavelength, and aperture.
s2 = "masktyp==1 && grating=='R831+_G5302' && centwave==853. "
s1 = arcS1 // s2

# Generate lists of filenames using above criteria.
hselect ("*.fits[1,inherit=yes]", "$I", s1, > "arc_tmp.txt")
gemextn ("@arc_tmp.txt", omit=omit, outfile="arc_R831_860.txt")
rename ("arc_R831_860.txt", "../")
dele ("*_tmp.txt")

Each of the configuration-specific tutorials will provide specific selection examples for MasterCals as well as science exposures.

Bias Residual

The bias residual is a persistent, spatially variable component of the read-out process that is not removed during overscan subtraction. The structure in this residual gives an indication of the stability of the read-out electronics, which for GMOS is usually quite stable over periods of months. Generally, this structure will be different for full-frame read-outs than for subsets of the array, such as the central spectrum region.

Note

The bias correction involves the following process:
  1. Perform the overscan correction on each Amp section
  1. Determine the mean for each row of the serial overscan region, exclusive of any columns that are excluded to avoid contaimination from strong signal in the science array.
  2. Subtract from the Amp array a low-order fit to this column of mean values. The best value for the order depends upon the detectors in use.
  3. Trim the overscan region(s), and concatenate Amp sections to form each CCD image
  1. Subtract the matching Bias Residual MasterCal from the image
The Bias Residual MasterCal files are created in the following way:
  1. Perform overscan correction on a large number of bias exposures, with the following properties in common:
  1. RoI
  2. CCD binning
  3. CCD gain and read-out speed
  1. Combine the corrected bias exposures, using by default:
  1. 3-sigma clipped mean
  2. Optionally compute and append variance [VAR] and data quality [DQ] extensions. The output DQ array is most meaningful if the Static BPM MasterCal is provided as input. See Bad Pixel Masks.

It is best to co-add more than 30 bias exposures, obtained within a couple of months of the exposures to be calibrated, to create a good Bias Residual MasterCal.

Basic Processing

Use gbias to: update the headers and perform the overscan correction using the default low-order fit to the mean along the slit (axis-1) of the overscan region. The resulting files are then combined to form the Bias Residual MasterCal.

cd /path/to/work_directory

# Use primarily the default task parameters.
unlearn gbias
gbias.bpm="bpm_gmos-s_EEV_v1_2x2_img_MEF.fits"
gbias.rawpath="./raw"
gbias.logfile="gbiasLog.txt"

# Create the VAR and DQ arrays as well.
gbias ("@biasfiles.txt", "MCbias.fits", fl_vardq+)

# List the file contents, display the image, etc., for QA
fxheader mcBias.fits

The Bias Residual MasterCal should have as many extensions as the input images, or 3 times that number if you elected to create the VAR and DQ arrays.

Darks

Because of the low dark-current of the GMOS sensors, dark exposures (finite-length exposures with the shutter closed) are usually only used for Nod-and-Shuffle data acquisition.

[N&S Additions TBD]

Flat-Fields

Constructing a Flat-field MasterCal is largely a matter of combining bias-corrected flat-field exposures, with appropriate scaling, outlier rejection, normalization, and conditioning. Flat-field exposures of either the twilight sky or the GCAL flatfield lamp can correct for pixel-to-pixel variations in sensitivity, though twilight flats may be used to correct the illumination pattern of the GCAL lamp. They are combined and normalized to create a Flat-field MasterCal. Separate flats must be created for each choice of:

  • CCD:
    • read-out Region of Interest (RoI)
    • binning factor
  • choice of illumination: twilight sky or GCAL
  • for imaging mode: filter
  • for spectroscopic modes
    • aperture (slit or mask)
    • disperser
    • central wavelength

It is best to combine a few to several well exposed flat-field exposures (if available) to keep noise in the flat-field from dominating the uncertainties in well exposed portions of the science data. Corrections to pixel-level response of more than a few tens of percent are not likely to result in high photometric accuracy. The process for creating Flat-field MasterCals is somewhat different for each observing mode: each will be described in turn. The output Flat-field MasterCal should have as many extensions as the input images, or 3 times that number if you elected to create the VAR and DQ arrays.

Imaging Flats

Creating Master Flat-field reference files for imaging mode is straightforward. However, it is very important to preview twilight flats to exclude any exposures that are not well exposed or contain bright stars.

Note

Once exposures with common attributes have been identified, the process to create a MasterCal Flat-field is:
  1. Perform Overscan and Bias Residual corrections.
  2. Combine the flats from step 1, with outlier rejection, scaling by the clipped mean of the images to account for variations in the flat-field illumination.
  3. Normalize the combined image:
    1. Divide by the clipped mean of the combined image excluding bad pixels and outliers.
    2. Condition the flat by setting a floor and ceiling on the pixel values.

Basic Processing

Use giflat to perform the overscan and bias corrections. The resulting files are then combined per filter to form a Flat-field MasterCal.

cd /path/to/work_directory

# Use primarily the default task parameters.
unlearn giflat
unlearn gemextn
giflat.bpm="bpm_gmos-s_EEV_v1_2x2_img_MEF.fits"
giflat.rawpath="./raw"
giflat.logfile = "giflatLog.txt"

# Process the raw flat-field exposures, e.g. for the r-filter
# Create the VAR and DQ arrays as well.
giflat ("@flats_r.txt", "MCflat_r.fits", bias="MCbias.fits", fl_vardq+)

Review the flat-field for quality. The normalized flats will show a variety of features, including charge traps, dust on the dewar window, etc.

Long-Slit and MOS Flats

Spectral flats require an additional step in the normalization to remove the response to the illumination source (i.e., the “color term”). MOS flats must in addition be extracted for each slitlet.

Note

Processing long-slit spectroscopic flat-field exposures is straightforward:
  1. Perform Overscan and Bias Residual corrections on each exposure.
  2. Combine the exposures, with outlier rejection, scaling by the clipped mean of the images to account for variations in the flat-field illumination.
  3. Characterize the shape in the dispersion direction (i.e., the color term) by fitting a curve of relatively high order to an average over the slit direction. It is safest to perform the fit interactively. Narrow spectral features must be rejected from the fit, and fringe features in the red must be excluded (or averaged over; this is particularly true for the E2V detectors).
  4. Divide each column of the combined flat by the polynomial.
  5. Normalize the color-corrected flat-field image.
  6. Condition the flat by setting a floor and ceiling on the pixel values.

The normalized spectral flats will show a variety of features, including charge traps, dust on the dewar window, and narrow features of the slit throughput. Be aware that exposures obtained with GCAL show large fluctuations with wavelength, which will require a fitting function with a large order.

Basic Processing

Use gsflat to: perform the overscan and bias corrections, combine the exposures, and normalize to form the Flat-field MasterCal. Note that the CCDs are not mosaiced, which is appropriate for longslit spectra, but not for MOS spectra. When the plotting window appears for the interactive fit, you must type a row number to perform the initial fit to the response function.

cd /path/to/work_directory

# Use primarily the default task parameters.
unlearn gsflat
unlearn gemextn
gsflat.rawpath = "./raw"
gsflat.logfile = "gsflatLog.txt"

# Process the raw flat-fields for grating B600, including VAR & DQ arrays.
# Specify orders for normalization function per CCD, and fit interactively.
gsflat ("@flats_b600.txt", "MCflat_b600.fits", bias="MCbias.fits", \
   fl_vardq+, fl_detec+, fl_oversize-, fl_inter+, order="13,11,28")

The following plot shows example fits to the flat-field spectral response for each of 3 CCDs.

../_images/RespFit_B600_485.png

A Flatfield MasterCal constructed in this way should have 3 times the number of image extensions as sensors in the FPA (i.e., it includes the DQ and VAR extensions), plus one extension for the MDF table.

Custom Processing for MOS Flats

Flat-fields for Multi-object spectral (MOS) mode are somewhat more complicated than longslit flats. Each slitlet produces its own wavelength dependent flat-field because of the offsets in the wavelength origin. The position of the slitlets in the FPA are defined by the MDF table, and the MOS flat-fields are used to refine the cross-dispersion position of the slitlets. When the normalized flat-fields are created, it is essential to save the combined (unnormalized) flat-field exposure in order to serve as a template for the slit positions in the other exposures, as shown below.

gsflat ("@flatFull_M01_B6-520.txt", "MCflatFull_M01_B6-520.fits", \
   bias="MCbiasFull.fits", combflat="MCflatComb_M01_B6-520.fits", \
   fl_keep+, fl_oversize-, fl_usegrad+, fl_detec-, fl_seprows-, \
   fl_inter+, order="53")

Note that a large order is needed for the cubic spline fit to the response function.

Caution

If you are using v1.3 or earlier of the gemini package, you must replace the gmos$gscut.cl task with the patched verion from the Gemini website. See Known Problems for details.

IFU Flats

IFU flat-fields are summed over the spatial extent of each fiber and are used to remove few-pixel variations in sensitivity in the dispersion direction. They are also used to define the trace parameters for contemporaneous on-sky exposures, as instrument flexure can displace the spectra on the FPA; for this reason they are rarely combined with other flat-field exposures. Finally, flat-field exposures are used to locate the order gaps on the FPA in order to model the scattered light on science and calibration exposures.

Note

Processing IFU spectroscopic flat-field exposures is somewhat more involved than for IFU:

  1. Perform Overscan and Bias Residual corrections on each exposure.
  2. Determine the location of the fiber bundle gaps in the spectrogram.
  3. Characterize and remove the scattered light background from the spectrogram as derived from the gap regions.
3a. (Optional) remove cosmic rays from the flat-field exposure if necessary.
  1. Perform the QE correction for affected CCDs.
  2. Re-extract the fiber flats from the background-subtracted spectrograms.
  3. Characterize the shape in the dispersion direction (i.e., the color term) by fitting a curve of relatively high order to each fiber spectrum, and divide the spectrum by the polynomial.
  4. Normalize the color-corrected flat-field spectra.

Basic Processing

Use gfreduce to bias-correct and trim the images. You will need to add (or append) the static BPM MasterCal to these images.

cd /path/to/science_files/

# Use primarily the default task parameters.
unlearn gemextn
unlearn gfreduce
gfreduce.logfile = "gfreduceLog.txt"
gfreduce.rawpath = "./raw/"

# Process the raw flat-fields for grating B600/625, including VAR & DQ arrays.
gfreduce ("@flatGcal_B6-625.txt", outimages="@trace_B6-625.txt", \
   bias="MCbiasS", fl_inter-, fl_vardq+, fl_extract+, fl_gscrrej-,
   fl_gsappwave-, fl_wavtran-, fl_skysub-, fl_fluxcal-)

Use gffindblocks and gfsubscat to characterize and remove the scattered light background.

# Loop over individual flats to correct for scattered light.
unlearn gfscatsub
sections ("rg//@flatGcal_B6-625.txt", option="root", >"procFlats_B6-625.txt")

list = "procFlats_B6-625.txt"
while (fscan (list, s1) != EOF) {
    # Find the gaps of unilluminated regions
    gffindblocks (s1, "ex"//s1//"_tr", s1//"_gaps.txt")

    # Use the gaps to model & remove the scattered light from the flats
    gfscatsub (s1, "ex"//s1//"_tr.fits", xorder="5,9,5", yorder="5,7,5", \
        cross+)
}

Now re-extract the fiber spectra, and normalize.

gfreduce.rawpath="./"
gfreduce ("b@procFlats_B6-625.txt", fl_addmdf-, fl_over-, fl_trim-, fl_bias-, \
   fl_bias-, fl_inter-, fl_vardq+, fl_extract+, fl_gscrrej-, fl_gsappwave+, \
   fl_wavtran-, fl_skysub-, fl_fluxcal-)

# Normalize flats to create the response functions.
unlearn gfresponse
gfresponse.logfile="gfresponseLog.txt"

sections ("ebrg//@flatGcal_B6-625.txt", option="root", >"bFlats_B6-625.txt")
list = "bFlats_B6-625.txt"
while (fscan (list, s1) != EOF) {
    gfresponse (s1, s1//'_flat', skyimage='', function='spline3',
        order=47, sample="1:1,30:2855,2879:2879", fl_inter+)
}

Wavelength Calibration

Overscan-corrected exposures of the Cu-Ar lamp are used to determine the dispersion solution for a given disperser and slit position. An atlas of such a comparison arc is shown below.

../_images/CuAr_HiRes.png

Cu-Ar spectrum at full scale (blue) with portions magnified (purple) and offset vertically for clarity. More than 400 identifiable lines are marked (red ticks) along the wavelength axis. Some of the brighter or more isolated lines are labelled, which should suffice to bootstrap a wavelength solution. Second-order lines in the near IR (lower right, labelled in green) may be useful for constraining the dispersion solution in this sparse part of the arc lamp spectrum. Click image to enlarge.

Additional line identifications may be found on the Gemini/GMOS CuAr Spectra page.

Note

You can use the high-dispersion line list (download: cuarHiRes.txt) to determine the dispersion solution for the highest resolution configurations (B1200 and R831 with a slitwidth <1.0arcsec), provided the Arc exposure is sufficiently deep. In these cases it should be possible to include more than 100 lines in the dispersion solution (with an RMS less than a few tenths of a pixel), except in the near IR. The fit will require a polynomial of order 6 or 7 to yield an acceptable solution.

Basic Processing

Arc lamp exposures need only be processed through Overscan and Bias corrections, and initializing the dispersion solution. However, MOS and IFU modes require that the arcs be split: one for each slitlet or fiber.

# Use primarily the default task parameters.
unlearn gsreduce
gsreduce.logfile="gsreduceLog.txt"
gsreduce.rawpath="./raw"

# Perform basic reductions on the list of arc exposures.
gsreduce ("@arcFull_B600.txt", bias="MCbiasFull", \
   fl_flat-, fl_fixpix-, fl_oversize-)

Now perform determine the dispersion solution with gswavelength, using a line list and linewidth appropriate to the slit width and CCD binning in use. The dispersion solution will be stored in a subdirectory of the working directory called ./database. The fit to the dispersion solution should be done interactively (which is the default) to ensure the quality of the solution.

# Use primarily the default task parameters.
# In this case, the default medium-resolution line list will work well.
unlearn gswavelength
gswavelength.logfile="gswaveLog.txt"
gswavelength.coordlist="gmos$data/CuAr_GMOS.dat"

gswavelength ("gsS20070623S0071.fits", fwidth=6, order=5, nsum=50)

It is best to experiment a bit with the fit order, though usually an order of 5 to 7 will remove any obvious power in the residuals plot and minimize the RMS (see below). If you are unfamiliar with the IRAF identify family of tasks, see the summary of Wavelength Calibration cursor commands.

../_images/autoid.jpg

Screen shots of interactive wavelength solution for B600 centered at 495.0 nm (left). The fit to 67 identified lines with a Chebyschev polynomial of order 5 resulted in an RMS of 0.13 (right).

The wavelength solution in this example is stored in the directory ./database in the file idgsS20070623S0071_001 and can be applied to science spectra with the gstransform task.

Custom Processing for MOS

The wavelength calibration will proceed for each slitlet. The default Chebyshev fitting function with order=7 should suffice.

Warning

Most slitlets have a small spatial extent, so it is critical that the stepsize in the spatial direction for mapping the geometric distortion. Use step=2 to avoid an incorrect distortion solution.

Custom Processing for IFU

The wavelength calibration will proceed for each extracted fiber. The default Chebyshev fitting function with order=7 should suffice, although it is a good idea to set fwidth=8.

Warning

A Trace MasterCal (derived from a flat-field exposure) must be available to extract the fiber spectra. The large number of fibers makes interactive wavelength solutions impractical. It is best to use the sparse but default gmos$data/CuAr_GMOS.dat line list.

Spectrophotometric Calibration

The flux calibration for spectra is derived from exposures of one or more spectrophotometric standard stars. These exposures should be processed through flat-fielding, wavelength calibration, sky subtraction and 1-D extraction (sky subtraction may be performed during extraction). Be sure to use the improved atmospheric extinction curve for Mauna Kea (download: mk_extinct.txt), or the curve for CTIO in the IRAF directory onedstds$ctioextinct.dat, as appropriate. Note that the name of the observed standard must match the IRAF name: see the table in Standards List.

# Derive the sensitivity function.
unlearn gsstandard
gsstandard.logfile="gsstdLog.txt"

# Use the calibration directory appropriate for your standard star.
gsstandard.caldir="onedstds$ctionewcal/"
gsstandard.extinction="./mk_extinct.txt"

gsstandard ("estLTT9239.fits", sfile="std", sfunction="sens", order=7, \
   starname="l9239", fl_inter-)

The fit to the sensitivity function is shown below.

../_images/LTT9239_sens.png

Screen shot of the fit to the sensitivity function using a spectrum of LTT9239, with a spline3 interpolant of order 7. Note the deletion of points near the chip gaps, and at the blue end of the spectrum where the sensitivity is lowest.