4.1. Modules¶

The PAPI pipeline consists of a set of processing modules which implement from basic calibration to generating final co-added registered mosaics (See table below). These modules can be run as a stand alone routines depending of your needs, e.g. to create a master dark or flat-field for calibration, or you can use them as a pipeline calling the main script papi, which will use each of the modules as they are needed in order to accomplish a complete data reduction of a set of raw images.

Main Modules	Description
`papi`	Main pipeline script to start the entire data reduction process
`applyDarkFlat`	Applies basic calibration (dark and flat-field) to the given list of files.
`astrowarp`	Creates final aligned and coadded frame using SEx, SCAMP and SWARP
`calBPM`	Creates a master Bad Pixel Mask from a set of darks and flats calibration files
`calCombineFF`	Combine a dome Flat-field and a sky Flat-field into a new Flat-field
`calDark`	Creates a master dark by combination of a dark sequence
`calDarkModel`	Creates a master dark model from a dark series
`calDomeFlat`	Creates a master Dome Flat
`calSuperFlat`	Creates a master Super Flat from a set of object or sky frames
`calTwFlat`	Creates a master Twilight Flat
`calGainMap`	Creates a Gain Map from any master flat
`correctNonLinearity`	Corrects the images pixel values for non-linearity
`dxtalk`	Removes cross-talk spots from input images
`makeobjmask`	Creates a objects mask (SExtractor OBJECTS images) for a list of FITS images.
`photometry`	Performs a photometric calibration comparison with 2MASS
`solveAstrometry`	Performs a astrometric calibration using Astrometry.net and 42xx index files
`remove_cosmics`	Detects and clean cosmic ray hits on images based on Pieter van Dokkum’s L.A.Cosmic algorithm.
`eval_focus_serie`	Estimates the best focus value of a focus exposures
`cleanBadPix`	Cleans masked (bad) pixels from an input image.

Utilities	Description
`createDataSeq`	Modifies headers of a set of FITS files to create a Data Sequece compliant with PAPI
`getBPM`	Creates the BPM file from the NonLinearity correction MEF file. The bad pixels will be saved as 1’s
`mef`	Tool to convert from SEF to MEF and viceversa; also allows to give splits of the extensions or join SEFs.
`collapse`	Collapses (add them up arithmetically) each cube of a list files into a single 2D image.
`genLogsheet`	Generates a text file as a log sheet from a set of images.
`imtrim`	Crops/cuts the input image edges
`modFITS`	Allows to perfom the modification of any FITS keyword
`runStarfocus`	Run IRAF.starfocus for a focus sequece and return the best focus value and a plot of the fit.
`runPsfmeasure`	Run IRAF.psfmeasure for a focus sequece and get field FWHM of given stars
`getDarks`	Gives the unique values of [read_mode, itime, ncoadd, save_mode] of a set of files of a given directory. Used to know the DARKS required from them.
`getImageOffsets`	Gives the image offsets (arcsecs) based on the WCS of the image headers

4.1.1. `papi`¶

Description:

The papi module (see PAPI) is the main PAPI module to run the data reduction. It starts by creating a subdirectory in the output_dir directory using the name give on the command line or in the $PAPI_CONFIG file. Within the run directory a [Q1-Q4] subdirectories, one for each detector, will be created. The temporal files will be saved (and deleted at the end) in the temp_dir directory.

Syntax:

Usage: papi.py [OPTION]... DIRECTORY...

This is PAPI, the PANIC PIpeline data reduction system - IAA-CSIC - Version 1.2.20150508064845

Options:
--version             show program's version number and exit
-h, --help            show this help message and exit
-c CONFIG_FILE, --config=CONFIG_FILE
                        Config file for the PANIC Pipeline application.If not
                        specified, './config_files/papi.cfg' is used.
-s SOURCE, --source=SOURCE
                        Source file list of data frames. It can be a fileor
                        directory name.
-d OUTPUT_DIR, --out_dir=OUTPUT_DIR
                        Output dir for product files
-o OUTPUT_FILE, --output_file=OUTPUT_FILE
                        Final reduced output image
-t TEMP_DIR, --temp_dir=TEMP_DIR
                        Directory for temporal files
-r ROWS, --rows=ROWS  Use _only_ files of the source file-list in the
                        rangeof rows specified (0 to N, both included)
-R, --recursive       Does recursive search for files in source directory
-l, --list            Generate a list with all the source files read fromthe
                        source and sorted by MJD
-M REDUCTION_MODE, --red_mode=REDUCTION_MODE
                        Mode of data reduction to do (quick|science|lab|lemon
                        |quick-lemon).
-m OBS_MODE, --obs_mode=OBS_MODE
                        Observing mode (dither|ext_dither|other)
-S SEQ_TO_REDUCE, --seq_to_reduce=SEQ_TO_REDUCE
                        Sequence number to reduce. By default, all sequences
                        found will be reduced.
-W DETECTOR, --window_detector=DETECTOR
                        Specify which detector to process:Q1(SG1), Q2(SG2),
                        Q3(SG3), Q4(SG4), Q123(all except SG4), all [default:
                        all]
-p, --print           Print all detected sequences in the Data Set
-T SEQ_TYPE, --sequences_type=SEQ_TYPE
                        Specify the type of sequences to show: DARK,
                        FLAT(all), DOME_FLAT, SKY_FLAT, FOCUS, SCIENCE, CAL,
                        all [default: all]
-b, --build_calibrations
                        Build all the master calibrations files
-C EXT_CALIBRATION_DB, --ext_calibration_db=EXT_CALIBRATION_DB
                        External calibration directory (library of Dark & Flat
                        calibrations)
-D MASTER_DARK, --master_dark=MASTER_DARK
                        Master dark to subtract
-F MASTER_FLAT, --master_flat=MASTER_FLAT
                        Master flat to divide by
-B BPM_FILE, --bpm_file=BPM_FILE
                        Bad pixel mask file
-g GROUP_BY, --group_by=GROUP_BY
                        kind of data grouping (based on) to do with thedataset
                        files (ot |filter)
-k, --check_data      if true, check data properties matching (type, expt,
                        filter, ncoadd, mjd)
-e, --Check           Check if versions of PAPI modules are right.

PAPI creates a in-memory SQLite database to store the uncalibrated input data fits headers and pipeline metadata.

Results:

FITS file/s with coadd as result of the reduction and calibration of the specified sequences; otherwise, the error will be shown in the console and log file.

Examples:

The following example reduce, in quick mode, all the sequences of the given directory:

$papi.py -s /my/raw_data/directory -d /my/output/directory -M quick

4.1.2. `applyDarkFlat`¶

This module receives a series of FITS images and applies basic calibration: subtract and divide by the given calibration files (master dark and master flat-field).

Options:

-h, --help            show this help message and exit
-s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                      Source file listing the filenames of raw frames
-d DARK_FILE, --dark=DARK_FILE
                      Master dark to be subtracted
-f FLAT_FILE, --flat-field=FLAT_FILE
                      Master flat-field to be divided by
-o OUT_DIR, --out_dir=OUT_DIR
                      Directory where output files will be saved

4.1.3. `astrowarp`¶

The astrowarp module performs the alignment and warping of a set of images, in principle previously reduced, but not mandatory. The module uses the Astromatic packages sextractor , scamp and swarp to accomplish this task.

Usage:

Options:
  -h, --help            show this help message and exit
  -c CONFIG_FILE, --config_file=CONFIG_FILE
                        config file
  -s SOURCE_FILE, --source=SOURCE_FILE
                        Source file list of data frames. It can be a file or directory name.
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        final coadded output image
  -v, --verbose         verbose mode [default]

Example:

$ astrowarp.py -c papi.cfg -s /tmp/test_files.txt -o /tmp/astrowarp.fits

4.1.4. `calBPM`¶

This module creates a master Bad Pixel Map (.pl iraf file) from a set of dome (on and off) flats.

The algorithm followed to create the BPM is the next:

Classify/split the frames in 3 sets (DOME_FLAT_LAMP_ON, DOME_FLAT_LAMP_OFF, DARKS) and and check whether there are enough calib frames

Check the master dark (Texp)

Subtract the master dark to each dome flat

Combine dome dark subtracted flats (on/off)

Compute flat_low/flat_high

Create BPM (iraf.ccdmask)

Usage:

Options:
  -h, --help            show this help message and exit
  -s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                        list of input (optionally  corrected) dome ON and OFF flat images..
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        The output bad pixel mask.
  -L LTHR, --lthr=LTHR  The low rejection threshold in units of sigma [default 20]
  -H HTHR, --hthr=HTHR  The high rejection threshold in units of sigma [default 20]
  -D MASTER_DARK, --master_dark=MASTER_DARK
                        [Optional] Master dark frame to subtract
  -S, --show_stats      Show statistics [default False]
  -v, --verbose         verbose mode [default]

Example:

$ calBPM.py -s /tmp/domesF.txt -D /tmp/masterDark.fits -o /tmp/masterBPM.pl

4.1.5. `calCombineFF`¶

Combine a master dome Flat-field and a master sky Flat-field into a combined master Flat-field. The procedure followed is :

The procedure for taking advantage of the facts that the large-scale flat-field variation of the dark-sky flat match that of the program frames and the dome flats have very high S/N in each pixel goes as follows:

(a) Median smooth the combined, dark-sky flat -this improves the S/N and preserves the large-scale features of the flat.

(b) Median smooth the combined dome flats using the same filter size as was used for the dark-sky flat.

(c) Divide the combined dome flat by it’s median smoothed-version. The result is a frame that is flat on large scales but contains all the high spatial frequency flat-field information.

(d) Now multiply the smoothed dark-sky frame and the result of the division in the previous step.

As result a flat-field with the low spatial frequency properties of the dark-sky flat combined with the high S/N, high spatial frequency properties of the dome flat is obtained.

Usage:

$ calCombineFF.py [options] arg1 arg2 ...

Module to combine a dome Flat-field and a sky Flat-field.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -d DOMEFF, --domeFF=DOMEFF
                        input dome Flat-Field
  -s SKYFF, --skyFF=SKYFF
                        input sky Flat-Field
  -o OUTPUT_IMAGE, --output=OUTPUT_IMAGE
                        output filename of combined Flat-Field (default = combinedFF.fits)

Example:

$ calCombineFF.py -d /data/masterDF.fits -s /data/masterSF.fits -o /data/masterFF.fits

4.1.6. `calDark`¶

The calDark module receives a series of FITS images (master darks) and create the master dark and computer several statistics.

Usage:

Usage: calDark.py [options] arg1 arg2 ...

Options:
  -h, --help            show this help message and exit
  -s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                        Source file listing the filenames of dark frames.
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        final coadded output image
  -n, --normalize       normalize master dark to 1 sec [default False]
  -e, --scale           scale raw frames by TEXP [default False]
  -S, --show_stats      Show frame stats [default False]
  -t, --no_type_checking
                        Do not make frame type checking [default False]
  -v, --verbose         verbose mode [default]

   Usage: calDark.py [options] arg1 arg2 ...

Example:

$ calDark.py -s /data/PANIC_V0/dark_seq.txt -o /data/out/masterDark.fits

4.1.7. `calDarkModel`¶

The calDarkModel module performs a dark model. To do that, a input dark series exposures with a range of exposure times is given. Then a linear fit is done at each pixel position of data number versus exposure time. A each pixel position in the output map represents the slope of the fit done at that position and is thus the dark current expressed in units of data numbers per second. The dark model obtained will be a FITS files with two planes (extensions):

plane 0 = dark current in DN/sec

plane 1 = bias

DARKCURRENT value: The median dark current in data numbers per second found from the median value of the output dark current map.

Usage:

Usage: calDarkModel.py [options] arg1 arg2 ...

Options:
  -h, --help            show this help message and exit
  -s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                        Source file listing the filenames of dark frames.
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        final coadded output image
  -S, --show_stats      Show frame stats [default False]

Example:

$ calDarkModel.py -s /tmp/darkModel.txt -o /tmp/darkModel.fits

4.1.8. `calDomeFlat`¶

The calDomeFlat module creates a master flat field from dome flat observations, a bad pixel map an various statistics.

Usage:

Options:
  -h, --help            show this help message and exit
  -s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                        Source file list of data frames. It can be a file or directory name.
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        final coadded output image
  -n, --normalize       normalize master flat by median. If image is multi-detector,                  then normalization wrt chip 1 is done) [default False]
  -m, --median_smooth   Median smooth the combined flat-field [default False]
  -v, --verbose         verbose mode [default]

Example:

$ calDomeFlat -s /tmp/domeFlats.txt -o /tmp/masterDF.fts -n

4.1.9. `calSuperFlat`¶

The calSuperFlat module creates a master super flat field from science observations, a bad pixel map an various statistics.

Usage:

Options:
  -h, --help            show this help message and exit
  -s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                        Source file list of data frames. It has to be a fullpath file name
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        output file to write SuperFlat
  -b BPM, --bpm=BPM     bad pixel map file (default=none)
  -N, --norm            normalize output SuperFlat. If image is multi-chip, normalization wrt chip 1 is done (default=True)
  -m, --median_smooth   Median smooth the combined flat-field (default=False)

Example:

$ calSuperFlat.py -s /tmp/test_files.txt  -o /tmp/superFlat.fits -N

4.1.10. `calTwFlat`¶

This module receives a series of FITS images (twilight flats) and a master dark model and creates the master twilight flat-field.

Usage:

Options:
  -h, --help            show this help message and exit
  -s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                        Source file list of data frames. It can be a file or directory name.
  -d MASTER_DARK, --master_dark_model=MASTER_DARK
                        Master dark model to subtract each raw flat (it will be scaled by TEXP)
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        final coadded output image
  -b MASTER_BPM, --master_bpm=MASTER_BPM
                        Bad pixel mask to be used (optional)
  -n, --normalize       normalize master flat by median. If image is multi-detector,then normalization wrt chip 1 is done)[default False]
  -m, --median_smooth   Median smooth the combined flat-field [default False]
  -L MINLEVEL, --low=MINLEVEL
                        flats with median level bellow (default=1000) are rejected
  -H MAXLEVEL, --high=MAXLEVEL
                        flats with median level above (default=100000) are rejected
  -v, --verbose         verbose mode [default]

Example:

$ calTwFlat.py -s /tmp/twflats.txt -d /tmp/darkModel.fits  -o /tmp/masterTF.fits -n

4.1.11. `calGainMap`¶

The calGainMap module creates a master gain map from a master flat field (dome, twilight or superflat) NOT normalized and previously created. The flatfield will be normalized to make a gainmap and set bad pixels to 0.

Usage:

Options:
-h, --help            show this help message and exit
-s SOURCE_FILE, --source=SOURCE_FILE
                        Flat Field image NOT normalized. It has to be a fullpath file name (required)
-o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        output file to write the Gain Map
-L MINGAIN, --low=MINGAIN
                        pixel below this gain value  are considered bad (default=0.5)
-H MAXGAIN, --high=MAXGAIN
                        pixel above this gain value are considered bad (default=1.5)
-x NXBLOCK, --nx=NXBLOCK
                        X dimen. (pixels) to compute local bkg (even) (default=16)
-y NYBLOCK, --ny=NYBLOCK
                        Y dimen. (pixels) to compute local bkg (even) (default=16)
-n NSIGMA, --nsigma=NSIGMA
                        number of (+|-)stddev from local bkg to be bad pixel (default=5)
-N, --normal          if true, the input flat-field will be normalized before build the gainmap (default=True)

Example:

$ calGainMap.py -s /tmp/masterTF.fits -o /tmp/masterGain.fits
$ calGainMap.py -s /tmp/masterTF.fits -o /tmp/masterGain.fits -L 0.7 -H 1.2

4.1.12. `dxtalk`¶

PANIC HAWAII-2RG sensors with multiple parallel readout sections show crosstalk in form of compact positive and negative ghost images whose amplitude varies between readout sections. PAPI has a optional de-crosstalk module that assumes that the amplitude is the same, therefore the correction will only partially remove the effect (if at all). If you know in advance that this will be a problem for your science case, then consider choosing different camera rotator angles for your observations.

It can be activated or deactivated in the Main config file (remove_crosstalk=True|False).

Usage:

Options:
  -h, --help            show this help message and exit
  -i INPUT_IMAGE, --input_image=INPUT_IMAGE
                        input image to remove crosstalk
  -o OUTPUT_IMAGE, --output=OUTPUT_IMAGE
                        output filename (default = dxtalk.fits)
  -O, --overwrite       overwrite the original image with the corrected one

Example:

$ ./dxtalk.py -i /tmp/pruebaDC.fits -O
$ ./dxtalk.py -i /tmp/pruebaDC.fits -o /tmp/pruebaDC_dx.fits

4.1.13. `makeobjmask`¶

Creates object masks (SExtractor OBJECTS images) for a list of FITS images or a single FITS image. Expects the command “sex” (SExtractor Version 2+) in path. If weight maps exist they will be used (assume weight map filename given by replacing .fits with .weight.fits).

The module can produce single poing masks,i.e, a single pixel set to 1 per each detected object if single_poing option is true.

Usage:

Options:
  -h, --help            show this help message and exit
  -s INPUTFILE, --file=INPUTFILE
                        It can be a source file listing data frames or a single FITS file to process.
  -o OUTPUTFILE, --output=OUTPUTFILE
                        Output text file including the list of objects mask files created by SExtractor ending with '.objs' suffix
  -m MINAREA, --minarea=MINAREA
                        SExtractor DETECT_MINAREA (default=5)
  -t THRESHOLD, --threshold=THRESHOLD
                        SExtractor DETECT_THRESH (default=2.0)
  -l SATURLEVEL, --saturlevel=SATURLEVEL
                        SExtractor SATUR_LEVEL (default=300000)
  -1, --single_point    Create a single point object mask (default=False)

Example::: $ ./makeobjmask.py -s /tmp/reduced_SEQ.fits -o /tmp/obj_mask.txt $ ./makeobjmask.py -s /tmp/reduced_SEQ.fits -o /tmp/obj_mask.txt -1 -l 100000 -m 10

4.1.14. `photometry`¶

This module receives a reduced image of any known NIR filter and match to 2MASS catalog performing a fit in order to get a estimation of the Zero Point. It is based on the method followed here

http://www.ast.cam.ac.uk/ioa/research/vdfs/docs/reports/2masscal.pdf

Usage:

Options:
  -h, --help            show this help message and exit
  -i INPUT_IMAGE, --input_image=INPUT_IMAGE
                        Input image to calibrate to do photometric comparison with
  -c BASE_CATALOG, --base_catalog (2MASS, USNO-B)=BASE_CATALOG
                        Name of base catalog to compare with (2MASS, USNO-B) -- not used !!! (default = 2MASS)
  -S SNR, --snr=SNR     Min SNR of stars used for linear fit (default = 10.0)
  -z ZERO_POINT, --zero_point=ZERO_POINT
                        Initial Magnitude Zero Point estimation [25.0]; used for SExtractor
  -o OUTPUT_FILENAME, --output=OUTPUT_FILENAME
                        Output plot filename (default = photometry.pdf)

Example:

$ photometry.py -i /data/reduced.fits -o /tmp/calibration.pdf

4.1.15. `correctNonLinearity`¶

HAWAII-2RG near-IR detectors exhibit an inherent non-linear response. It is caused by the change of the applied reverse bias voltage due to the accumulation of generated charge. The effect increases with signal levels, so that the measured signal deviates stronger from the incident photon number at higher levels, and eventually levels out when the pixel well reaches saturation.

The correctNonLinearity module corrects PANIC images for their count-rate dependent non-linearity. It used the header keywords READMODE and DET_ID to determine the correction. It corrects the first image, and in the case of a multi-extension image, the second image as well, with the appropriate power law. For details see PANIC detector non-linearity correction data.

Usage:

Options:
  -h, --help            show this help message and exit
  -m MODEL, --model=MODEL
                        FITS MEF-cube file of polinomial coeffs (c4, c3, c2, c1).
  -s SOURCE_FILE_LIST, --source=SOURCE_FILE_LIST
                        Source file list of FITS files to be corrected.
  -o OUT_DIR, --out_dir=OUT_DIR
                        filename of out data file (default=/tmp)
  -S SUFFIX, --suffix=SUFFIX
                        Suffix to use for new corrected files.
  -f, --force           Force Non-linearity correction with no check of headervalues (NCOADD, DATE-OBS, DETROT90, ...

4.1.16. `solveAstrometry`¶

Performs the astrometric calibration of a set of images, in principle previously reduced, but not mandatory; this routine is built on top of Astromety.net tool.

Usage:

Options:

-h, --help            show this help message and exit
-s SOURCE_FILE, --source=SOURCE_FILE
                        Source file list of data frames. It can be a file or directory name.
-o OUTPUT_DIR, --output_dir=OUTPUT_DIR
                        Place all output files in the specified directory [default=/tmp]
-p PIXEL_SCALE, --pixel_scale=PIXEL_SCALE
                        Pixel scale of the images
-r, --recursive       Recursive subdirectories (only first level)

4.1.17. `remove_cosmics`¶

Remove the cosmic ray hits in the input image; it is built on top of Pieter van Dokkum’s L.A.Cosmic algorithm.

Usage:

Options:

-h, --help            show this help message and exit
-i INPUT_IMAGE, --input_image=INPUT_IMAGE
                        input image to remove cosmics
-o OUTPUT_IMAGE, --output=OUTPUT_IMAGE
                        output filename (default = without_cosmics.fits)
-O, --overwrite       overwrite the original image with the corrected one
-m, --mask            If true, the mask with cosmics detected and removed is written into a FITS file.

4.1. Modules¶

4.1.1. papi¶

4.1.2. applyDarkFlat¶

4.1.3. astrowarp¶

4.1.4. calBPM¶

4.1.5. calCombineFF¶

4.1.6. calDark¶

4.1.7. calDarkModel¶

4.1.8. calDomeFlat¶

4.1.9. calSuperFlat¶

4.1.10. calTwFlat¶

4.1.11. calGainMap¶

4.1.12. dxtalk¶

4.1.13. makeobjmask¶

4.1.14. photometry¶

4.1.15. correctNonLinearity¶

4.1.16. solveAstrometry¶

4.1.17. remove_cosmics¶

4.1.1. `papi`¶

4.1.2. `applyDarkFlat`¶

4.1.3. `astrowarp`¶

4.1.4. `calBPM`¶

4.1.5. `calCombineFF`¶

4.1.6. `calDark`¶

4.1.7. `calDarkModel`¶

4.1.8. `calDomeFlat`¶

4.1.9. `calSuperFlat`¶

4.1.10. `calTwFlat`¶

4.1.11. `calGainMap`¶

4.1.12. `dxtalk`¶

4.1.13. `makeobjmask`¶

4.1.14. `photometry`¶

4.1.15. `correctNonLinearity`¶

4.1.16. `solveAstrometry`¶

4.1.17. `remove_cosmics`¶