dl3_production.rst 6.05 KB
Newer Older
Alice Donini's avatar
Alice Donini committed
1
2
3
4
5
6
7
8
9
10
11
12
.. _dl3:

DL3 files generation
====================

There are two different scripts to generate the DL3 file:

* `DL2_to_DL3.py`, which allows to analyze DL2 data generated by the user beforehand from DL1 data (:ref:`dl2`)
* `create_DL3_files_from_database.py`, which uses the DL2 files created auyomatically by LST OSA.

.. note::

Alice Donini's avatar
Alice Donini committed
13
  At the moment is recommended the use of the first script `DL2_to_DL3.py` and generate your own DL2 data, since the automatized generation of DL2 data is not optimized.
Alice Donini's avatar
Alice Donini committed
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64


DL2_to_DL3.py
--------------

The script `DL2_to_DL3.py` uses DL2 data generated by the user beforehand (:ref:`dl2`) and produces DL3 files.
Usage:

.. code-block::

    usage: DL2_to_DL3.py [-h] [--prod PROD] [--outdir OUTDIR] [--config CONFIG] [--config-analysis CONFIG_ANALYSIS]
                     [--verbose] [--source_name SOURCE_NAME] [--tcuname TCUNAME] [--runlist RUNLIST]
                     [--distance DISTANCE] [--ra RA] [--dec DEC] [--submit] [--dry] [--globber] [--cut_file CUT]
                     [--gh_cut GH_CUT] [--theta_cut THETA_CUT]

    DL2 to DL3 converter

    optional arguments:
      --config CONFIG, -c CONFIG
                            Specify a personal config file for the analysis
      --config-analysis CONFIG_ANALYSIS
                            Specify a config file which describes analysis profile to use
      --cut_file CUT, -cut CUT
                            Cut file
      --dec DEC             Dec coordinate of the target. To add if you want to use custom position
      --distance DISTANCE, -dis DISTANCE
                            Max distance in degrees between the target position and the run pointing position for the run
                            selection, negative value means no selection using this parameter (default: -1).
      --dry                 Make a dry run, no true submission
      --gh_cut GH_CUT       Fixed selection cut for gh_score (gammaness)
      --globber, -g         If True, overwrites existing output file without asking
      --outdir OUTDIR, -o OUTDIR
                            Directory to store the output
      --prod PROD, -p PROD  Prod to use (default: v0.9.4)
      --ra RA               RA coordinate of the target. To add if you want to use custom position
      --runlist RUNLIST, -rl RUNLIST
                            File with a list of run and the associated night to be analysed
      --source_name SOURCE_NAME, -n SOURCE_NAME
                            Name of the source
      --submit              Submit the cmd to slurm on site
      --tcuname TCUNAME     Apply run selection based on TCU source name
      --theta_cut THETA_CUT
                            Fixed selection cut for theta
      --verbose, -v         Increase output verbosity
      -h, --help            show this help message and exit

It makes use of a configuration file `config_dl2_to_dl3.yaml` (option ``--config``).
Edit the configuration file changing the paths based on the analysis you want to perform.
For the moment the IRF file to use has to be specified in the configuration file.
IRF files with different settings can be found in ``/fefs/aswg/data/mc/IRF/``, otherwise create your own (:ref:`irf_generation`).

Alice Donini's avatar
Alice Donini committed
65
The coordinates of the source will be searched with astropy from the name specified with the option ``-n``. If no source can be found or you want custom Ra,Dec coordinates use the options ``--ra`` and ``--dec``.
Alice Donini's avatar
Alice Donini committed
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
Configuration file:

.. code-block::
    
    # Directory where job files are written
    jobmanager: ../jobmanager

    # Database file name
    db: database.csv

    # Path to main data folder of the user
    # Change it accordingly to your working env
    base_dir: /fefs/aswg/alice.donini/Analysis/data

    # Path to personal directory where output data will be saved.
    # Uncomment and modify in case you want to use a non standard path
    #output_folder: ../DL3/Crab

    # Path to the folder where cut files are stored
    cut_folder: ../cuts

    # Path to IRF file, change it based on your analysis
    irf_file: "/fefs/aswg/data/mc/IRF/20200629_prod5_trans_80/zenith_20deg/south_pointing/20210416_v0.7.3_prod5_trans_80_local_taicut_8_4/off0.4deg/irf_20210416_v073_prod5_trans_80_local_taicut_8_4_gamma_point-like_off04deg.fits.gz"

    # Values for automatic selection of DL2 data from personal directories
    dl2_data:
      night: [20210911, 20210912]           # day(s) of observation (more than one is possible)
      version: v0.9.1                       # v0.7.3, v0.8.4, v0.9, v0.9.1
      cleaning: tailcut84

The generated DL3 data will be saved in a personal folder following the tree structure visible in :ref:`install`.
It is possible to define a different ouput folder adding the option ``-o``.

Alice Donini's avatar
Alice Donini committed
99
100
101
102
103
104
105
Examples of usage:

.. code-block:: bash
    
    python DL2_to_DL3.py -c config_dl2_to_dl3_GRB220521.yaml -n GRB220521 --runlist $CONFIG_FOLDER/GRB220521.txt --ra 275.19660 --dec 10.38430 --submit -v -g
    python DL2_to_DL3.py -c config_dl2_to_dl3_CrabNov.yaml -n Crab --runlist $CONFIG_FOLDER/CrabNov.txt -v -g --submit

Alice Donini's avatar
Alice Donini committed
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
.. note::

    Same cuts as the one used for the generation of IRF should be used.


create_DL3_files_from_database.py
----------------------------------

For a fast analysis with the use of DL2 files automatically mergerd by LST OSA, run for example:

.. code-block:: bash

    python DL2_to_DL3.py --tcuname "grb210807A"  --ra 76.677 --dec 58.245 --outdir ../data/ --submit -v
    python DL2_to_DL3.py --tcuname "Crab" --ra 83.633080 --dec 22.014500 --outdir ../data/  --submit -v
    python DL2_to_DL3.py --distance 2 --ra 83.633080 --dec 22.014500 --outdir ../data/  --submit -v

`--tcuname` is the name of the TCU configuration and is used to select the runs to analyze from the Database file, unless specific nights or list of runs are specified using option `--night` and `--run`. Also Ra, Dec position can be specified if already known, and are mandatory in case the name of the target can not be resolved. If the option distance is specified, runlist is created using all runs with poiting position within the angular distance specified.
If no cut configuration file is given, standard configuration is applied.