Newer
Older
.. _dl3:
DL3 files generation
====================
There are two different scripts to generate the DL3 file:
* `DL2_to_DL3.py`, which allows to analyze DL2 data generated by the user beforehand from DL1 data (:ref:`dl2`)
* `create_DL3_files_from_database.py`, which uses the DL2 files created auyomatically by LST OSA.
.. note::
At the moment is recommended the use of the first script `DL2_to_DL3.py` and generate your own DL2 data, since the automatized generation of DL2 data is not optimized.
.. note::
In the case the run of the job fails due to an astropy error about too old values of the IERS data:
.. code-block::
ValueError: interpolating from IERS_Auto using predictive values that are more than 30.0 days old.
run before the script `IERS_download.py` to download offline the data and update the astropy tables.
DL2_to_DL3.py
--------------
The script `DL2_to_DL3.py` uses DL2 data generated by the user beforehand (:ref:`dl2`) and produces DL3 files.
Usage:
.. code-block::
usage: DL2_to_DL3.py [-h] [--verbose] [--outdir OUTDIR] --config CONFIG --source_name SOURCE_NAME [--ra RA] [--dec DEC] [--cut_file CUT_FILE] [--gh_cut GH_CUT] [--theta_cut THETA_CUT] [--submit] [--dry] [--globber]
DL2 to DL3 converter
optional arguments:
--config CONFIG, -c CONFIG
Specify a personal config file for the analysis
Cut file
--dec DEC Dec coordinate of the target. To add if you want to use custom position
--dry Make a dry run, no true submission
--gh_cut GH_CUT Fixed selection cut for gh_score (gammaness)
--globber, -g If True, overwrites existing output file without asking (default: False).
--outdir OUTDIR, -o OUTDIR
Directory to store the output
--ra RA RA coordinate of the target. To add if you want to use custom position
--source_name SOURCE_NAME, -n SOURCE_NAME
Name of the source
--submit Submit the cmd to slurm on site
--theta_cut THETA_CUT
Fixed selection cut for theta
--verbose, -v Increase output verbosity
-h, --help show this help message and exit
It makes use of a configuration file `config_dl2_to_dl3.yaml` (option ``--config``).
Edit the configuration file changing the paths based on the analysis you want to perform.
For the moment the IRF file to use has to be specified in the configuration file.
IRF files with different settings can be found in ``/fefs/aswg/data/mc/IRF/``, otherwise create your own (:ref:`irf_generation`).
The coordinates of the source will be searched with astropy from the name specified with the option ``-n``. If no source can be found or you want custom Ra,Dec coordinates use the options ``--ra`` and ``--dec``.
Configuration file:
.. code-block::
# Directory where job files are written
jobmanager: ../jobmanager
# Database file name
db: database.csv
# Path to main data folder of the user
# Change it accordingly to your working env
base_dir: /fefs/aswg/alice.donini/Analysis/data
# Path to personal directory where output data will be saved.
# Uncomment and modify in case you want to use a non standard path
#output_folder: ../DL3/Crab
# Path to the folder where cut files are stored
cut_folder: ../cuts
# Path to IRF file, change it based on your analysis
irf_file: "/fefs/aswg/data/mc/IRF/20200629_prod5_trans_80/zenith_20deg/south_pointing/20210416_v0.7.3_prod5_trans_80_local_taicut_8_4/off0.4deg/irf_20210416_v073_prod5_trans_80_local_taicut_8_4_gamma_point-like_off04deg.fits.gz"
# Values for automatic selection of DL2 data from personal directories
dl2_data:
night: [20210911, 20210912] # day(s) of observation (more than one is possible)
version: v0.9.1 # v0.7.3, v0.8.4, v0.9, v0.9.1
cleaning: tailcut84
The generated DL3 data will be saved in a personal folder following the tree structure visible in :ref:`install`.
It is possible to define a different ouput folder adding the option ``-o``.
Examples of usage:
.. code-block:: bash
python DL2_to_DL3.py -c config_dl2_to_dl3_GRB220521.yaml -n GRB220521 --runlist $CONFIG_FOLDER/GRB220521.txt --ra 275.19660 --dec 10.38430 --submit -v -g
python DL2_to_DL3.py -c config_dl2_to_dl3_CrabNov.yaml -n Crab --runlist $CONFIG_FOLDER/CrabNov.txt -v -g --submit
.. note::
Same cuts as the one used for the generation of IRF should be used.
create_DL3_files_from_database.py
----------------------------------
For a fast analysis with the use of DL2 files automatically mergerd by LST OSA, run for example:
.. code-block:: bash
python DL2_to_DL3.py --tcuname "grb210807A" --ra 76.677 --dec 58.245 --outdir ../data/ --submit -v
python DL2_to_DL3.py --tcuname "Crab" --ra 83.633080 --dec 22.014500 --outdir ../data/ --submit -v
python DL2_to_DL3.py --distance 2 --ra 83.633080 --dec 22.014500 --outdir ../data/ --submit -v
`--tcuname` is the name of the TCU configuration and is used to select the runs to analyze from the Database file, unless specific nights or list of runs are specified using option `--night` and `--run`. Also Ra, Dec position can be specified if already known, and are mandatory in case the name of the target can not be resolved. If the option distance is specified, runlist is created using all runs with poiting position within the angular distance specified.
If no cut configuration file is given, standard configuration is applied.