PANIC Pipeline Documentation

Release

3.0

Date

Oct 04, 2022

Warning

1. This “Documentation” is still a work in progress; some of the material is not organized, and several aspects of PAPI are not yet covered with sufficient detail.

2. This manual support HAWAII-2RG and HAWAII-4RG detectors.

Welcome! This is the Documentation for PAPI, the Data Reduction Pipeline of PANIC instrument, and for PANIC Quick-Look Tool (PQL), the GUI of PAPI used during the observation; all processing routines used by PQL are implented in PAPI.

PAPI is the automatic image processing pipeline for data taken with the PAnoramic Near Infrared Camera (PANIC) for the 2.2m and 3.5m Telescopes at Calar Alto Observatory (CAHA). The pipeline is written in Python and developed at the Institute of Astrophysics of Andalusia (CSIC). The automated processing steps include basic calibration (removeing instrumental signature), cosmic-ray removal, treatment for electronic ghosts (cross-talk), sky subtraction, non-linear count-rate correction, robust alignment and registration.

This manual is a complete description of the data reduction recipes implemented by the PAPI pipeline, showing the status of the current pipeline version and describing data reduction process of the PANIC data using PAPI.

Although PAPI was developed for the PANIC camera with H2RG detector, it was updated in 2022 to support the new H4RG detector integrated into PANIC.

Moreover, its development was initiated using data from Omega2000 camera at the 3.5m CAHA telescope and HAWK-I camera at the VLT; thus, in principle it should work with data these instruments, although it is not optimized for them.

In addition to this html version of the manual, there is also a pdf version to download.

Development: José-Miguel Ibáñez-Mengual (IAA-CSIC)

Contribution: PANIC Team

Caveat

Currently PAPI it is able to reduce data taken with the Observing Tool (OT) defining the required observing blocks (OB), or manually through GEIRS scripts. PAPI was primarily developed and optimized for reducing broad-band imaging data of extragalactic sources (such as imaging data taken for field galaxy surveys and galaxy cluster surveys). Other types imaging data have been reduced with PAPI but results can not be as good as desired. (See Troubleshooting for tips). PAPI is not designed to reduce any kind of field taken with PANIC.

Contents

• 1. Installation & Configuration
  • 1.1. Requirements and Supported Platforms
    • 1.1.1. System Requirements
    • 1.1.2. Software Requirements
  • 1.2. Download
  • 1.3. Environment Installation
  • 1.4. PAPI Installation
  • 1.5. Installing for developers
  • 1.6. Building the documentation
  • 1.7. Bug reports
  • 1.8. Release Notes
• 2. PANIC Quick-Look Tool (PQL)
  • 2.1. Purpose
  • 2.2. FITS files and headers
  • 2.3. Starting PQL
  • 2.4. Configuration files
  • 2.5. PQL’s main window
    • 2.5.1. Menu bar
    • 2.5.2. Tool bar
    • 2.5.3. Main panel
      • 2.5.3.1. Data directories
      • 2.5.3.2. Input directory
      • 2.5.3.3. Output directory
      • 2.5.3.4. Temporal directory
    • 2.5.4. Current night checkbox
    • 2.5.5. Use GEIRS file
    • 2.5.6. Filename filter
    • 2.5.7. Data list view
    • 2.5.8. List view filter
    • 2.5.9. QuickLook mode
    • 2.5.10. Last file received
    • 2.5.11. Buttons
      • 2.5.11.1. Subract-last2 button
      • 2.5.11.2. Create calibrations button
      • 2.5.11.3. START button
      • 2.5.11.4. Add button
      • 2.5.11.5. Remove button
      • 2.5.11.6. Clear All button
    • 2.5.12. Setup Panel
      • 2.5.12.1. Common Settings
      • 2.5.12.2. Data grouping
      • 2.5.12.3. Astrometry
      • 2.5.12.4. Display
      • 2.5.12.5. Lazy mode
      • 2.5.12.6. Pre-reduction
    • 2.5.13. Calibrations panel
      • 2.5.13.1. Set Calibs Dir
      • 2.5.13.2. Load last
    • 2.5.14. Log panel
  • 2.6. Event log window
  • 2.7. Pop-up menu
    • 2.7.1. Display image
    • 2.7.2. Image info
    • 2.7.3. Copy files to clipboard
    • 2.7.4. Copy files to text file
    • 2.7.5. Show Dither pattern
    • 2.7.6. Calibrations
      • 2.7.6.1. Build Master Dark
      • 2.7.6.2. Build Master Dome Flat
      • 2.7.6.3. Build Master Twlight (sky) Flat
      • 2.7.6.4. Build GainMap
      • 2.7.6.5. Build BPM
      • 2.7.6.6. Apply Dark & FlatField & BPM
      • 2.7.6.7. Apply Non-Linearity Correction
      • 2.7.6.8. Apply and show BPM
    • 2.7.7. Focus evaluation
    • 2.7.8. Subtract own-sky
    • 2.7.9. Subtract near-sky
    • 2.7.10. Quick reduction
    • 2.7.11. Astrometric calibration
    • 2.7.12. Photometric calibration
    • 2.7.13. Statistics
    • 2.7.14. FWHM mean estimation
    • 2.7.15. Background estimation
    • 2.7.16. Math operations
    • 2.7.17. FITS operations
  • 2.8. How do I …?
    • 2.8.1. How to determine the telescope focus ?
    • 2.8.2. How to determine the field rotation ?
    • 2.8.3. How to inspect the profile of the stars in an image ?
    • 2.8.4. How do I quick-reduce an observed sequence ?
    • 2.8.5. How do I quick-reduce an observed sequence using dark and flat master calibration files ?
    • 2.8.6. How do I make mosaics with PQL?
    • 2.8.7. How do I make use of parallelisation ?
    • 2.8.8. How do I report a issue ?
• 3. PAPI
  • 3.1. Purpose
  • 3.2. Quickstart
    • 3.2.1. Optional Arguments
    • 3.2.2. Input FITS data files
    • 3.2.3. Show grouped files in a raw directory
    • 3.2.4. Show grouped files per filter and coordinates of a raw directory
    • 3.2.5. Reduce (quick) a specificied number of sequences of the group list
    • 3.2.6. Reduce all the sequences of a given directory
    • 3.2.7. Reduce all the sequences of a given set of directories
    • 3.2.8. Calibrations
      • 3.2.8.1. Location of calibrations
      • 3.2.8.2. Example: use a specific calibration directory for data reduction
      • 3.2.8.3. Enable the Non-Linearity correction for the data processing
    • 3.2.9. Reduce a single detector
  • 3.3. Reduction modes
  • 3.4. How NOT to use PAPI
  • 3.5. Configuration files
  • 3.6. Examples
  • 3.7. Main config file
    • 3.7.1. Principal parameters to set
    • 3.7.2. Bad Pixel treatment
  • 3.8. Getting PAPI Data
  • 3.9. Caveats
• 4. PAPI Reference
  • 4.1. Modules
    • 4.1.1. papi
    • 4.1.2. applyDarkFlat
    • 4.1.3. astrowarp
    • 4.1.4. calBPM
    • 4.1.5. calCombineFF
    • 4.1.6. calDark
    • 4.1.7. calDarkModel
    • 4.1.8. calDomeFlat
    • 4.1.9. calSuperFlat
    • 4.1.10. calTwFlat
    • 4.1.11. calGainMap
    • 4.1.12. dxtalk
    • 4.1.13. makeobjmask
    • 4.1.14. photometry
    • 4.1.15. correctNonLinearity
    • 4.1.16. solveAstrometry
    • 4.1.17. remove_cosmics
  • 4.2. Utilities
    • 4.2.1. check_papi_modules
    • 4.2.2. collapse
    • 4.2.3. mef
    • 4.2.4. checkQuality
    • 4.2.5. eval_focus_series
    • 4.2.6. genLogsheet
    • 4.2.7. health
    • 4.2.8. imtrim
    • 4.2.9. modFits
    • 4.2.10. skyfilter
    • 4.2.11. spatial_noise
• 5. Data formats
  • 5.1. Detector
  • 5.2. FITS
    • 5.2.1. HAWAII-4RG
      • 5.2.1.1. H4RG Header
    • 5.2.2. HAWAII-2RG
      • 5.2.2.1. H2RG Header
  • 5.3. Observation Tool keywords
  • 5.4. Data
  • 5.5. Classification
  • 5.6. Data grouping
• 6. Processing description
  • 6.1. Outline
    • 6.1.1. Main configuration file
    • 6.1.2. Data-set classification
    • 6.1.3. Data Preparation
    • 6.1.4. Calibrations
  • 6.2. Computing the master dark
  • 6.3. Computing the master flat-field
  • 6.4. Computing the Bad Pixel Mask
    • 6.4.1. First pass sky subtraction
  • 6.5. Sky model
    • 6.5.1. Object detection
    • 6.5.2. Offset computation
    • 6.5.3. First pass coaddition
    • 6.5.4. Master object mask
    • 6.5.5. Non-Linearity
    • 6.5.6. Crosstalk
    • 6.5.7. Extended Objects
• 7. References
  • 7.1. Publications
  • 7.2. Webpages
• 8. PAPI FAQ
  • 8.1. Installation
    • 8.1.1. Does PAPI work with python 3?
    • 8.1.2. What GUI toolkit does PAPI-QL use?
    • 8.1.3. Can PAPI-QL work with PyQt5?
  • 8.2. Data reduction
    • 8.2.1. What is the best way to reduce PAPI data?
    • 8.2.2. How can process a broken, interrupted or non OT sequence ?
    • 8.2.3. How does PAPI treat bad pixels ?
    • 8.2.4. How can we reach hundredth of magnitude accuracy in photometry ?
    • 8.2.5. How good is PAPI astrometry and how are PSF variations corrected ?
• 9. Troubleshooting
  • 9.1. Why do I get a “command not found” error when trying to run one of the PAPI modules?
  • 9.2. PAPI does not recognize a group of files as a well formed sequence.
  • 9.3. PAPI can not run the astrometric calibration
  • 9.4. PAPI can not run the photometric calibration
  • 9.5. PAPI says some files in the input file list does not look a FITS file.
  • 9.6. What is the best way to reduce PAPI data?
  • 9.7. How can we reach hundredth of magnitude accuracy in photometry ?
  • 9.8. How good is PAPI astrometry and how are PSF variations corrected ?
  • 9.9. Is there a way to look at or edit FITS headers?
  • 9.10. Does PAPI generate a log file of the processing ?
• 10. Acknowledgments
• 11. License
• 12. Glossary

Citation

If your research uses PAPI, we’d appreciate it if you could acknowledge the fact by including the following citation:

“This research made use of PAPI, the pipeline of PANIC instrument. It is funded by the Spanish Ministry of Economy and Competitiveness with funds from the European Union (FEDER) and the Spanish national budget, through the grants ICTS-2006-15, ICTS-2007-10, ICTS-2008-24, ICTS-2009-32 and the project Intramural 200450E458 of the Spanish National Research Council.”

• Naranjo, V. et al. “PANIC-4K: upgrade with a HAWAII-4RG array.” Astronomical Telescopes + Instrumentation (2020), Proc. SPIE 2020

• Cárdenas Vázquez, María Concepción et al. “PANIC: A General-purpose Panoramic Near-infrared Camera for the Calar Alto Observatory.” Publications of the Astronomical Society of the Pacific 130 (2017), Paper_PASP_2017

• Ibáñez Mengual, J.M., Fernández, M., Rodríguez Gómez, J. R., García Segura, A. J., Storz, C., “The PANIC software system”, Proc. SPIE 7740, 77402E (2010)

• Ibáñez Mengual,J.M, García A.J, Storz C., Fried J. W., Fernández M., Rodríguez J. F., “Advanced PANIC quick-look tool using Python”, Proc. SPIE 8451, (2012)

Indices and tables

• Index

• Module Index

• Search Page

Documentation last updated on Oct 04, 2022