Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
.. _dl2:
DL2 files generation
====================
The script `DL1_to_DL2.py` allows to analyze DL1 data and produce DL2 files.
Usage:
.. code-block::
usage: DL1_to_DL2.py [-h] [--prod PROD] [--outdir OUTDIR] [--config CONFIG] [--config-analysis CONFIG_ANALYSIS]
[--verbose] [--source_name SOURCE_NAME] [--tcuname TCUNAME] [--runlist RUNLIST]
[--distance DISTANCE] [--ra RA] [--dec DEC] [--submit] [--dry] [--globber]
DL1 to DL2 converter
optional arguments:
--config CONFIG, -c CONFIG
Specify a personal config file for the analysis
--config-analysis CONFIG_ANALYSIS
Specify a config file which described analysis profile to use
--dec DEC Dec coordinate of the target. To add if you want to use custom position
--distance DISTANCE, -dis DISTANCE
Max distance in degrees between the target position and the run pointing position for the
run selection, negative value means no selection using this parameter (default: -1).
--dry Make a dry run, no true submission
--globber, -g If True, overwrites existing output file without asking
--outdir OUTDIR, -o OUTDIR
Directory to store the output
--prod PROD, -p PROD Prod to use (default: v0.8.4)
--ra RA RA coordinate of the target. To add if you want to use custom position
--runlist RUNLIST, -rl RUNLIST
File with a list of run and the associated night to be analysed
--source_name SOURCE_NAME, -n SOURCE_NAME
Name of the source
--submit Submit the cmd to slurm on site
--tcuname TCUNAME Apply run selection based on TCU source name
--verbose, -v Increase output verbosity
-h, --help show this help message and exit
It makes use of a configuration file `config_dl1_to_dl2.yaml` (option ``--config``):
.. code-block:: yaml
# Directory where job file are written
jobmanager: ../jobmanager
# Database file name
db: database.csv
# LST real data path (don't modify it)
data_folder: /fefs/aswg/data/real
# path to main data folder of the user
# change it accordingly to your working env
base_dir: /fefs/aswg/alice.donini/Analysis/data
# Path to personal directory where output data will be saved.
# Uncomment and modify in case you want to use a non standard path
#output_folder: ../DL2/Crab
# Directory where config files are stored
#config_folder: ./
# Path to trained RF files
path_models: ../models
# Values for automatic selection of DL1 data
dl1_data:
DL1_dir: /fefs/aswg/data/real/DL1 # path to DL1 directory
night: [20210911, 20210912] # day(s) of observation (more than one is possible)
version: v0.9.1 # v0.7.3, v0.8.4, v0.9, v0.9.1
cleaning: tailcut84
Edit the configuration file: change the paths based on your working directory and modify the DL1 data information used to search for the files at the IT.
The script uses the created database (:ref:`db_generation`) to find the runs to analyze, based on the nights specified in the configuration file `config_dl1_to_dl2.yaml`.
An extra selection can be done in coordinates (``--distance``, ``--ra`` and ``--dec`` are mandatory arguments) or by the name of the source as saved in TCU (argument ``--tcuname``).
If none of these selection methods is given, then all the runs available in the dates specified in the configuration file are considered for the analysis.
The search for DL1 files can be also done by giving a file list of runs and nights to be analyzed (option ``--runlist``).
No database file is needed in this case.
.. warning::
The search of runs through the database has an issue on the dates at the moment. The database is generated from the drive log, so all the runs taken after the midnight are saved under the following day. This doesn't happen at the IT, where the runs are stored under the day of the starting night. So for some runs the search could fail, even if they are there.
Thus if you use the database search always add in the config file also the date of the night after, so that you are sure all the runs take after the midnight are considered too.
Example of a file run list:
.. code-block::
2909 20201117
2911 20201117
3089 20201206
The argument ``--dry`` allows to perform a dry run.
No jobs are submitted and only the verbose output is printed.
Usefull to have a check of which runs are selected and the goodness of the sent command.
Some examples of how to run the script:
.. code-block::
python DL1_to_DL2.py -c config_dl1_to_dl2.yaml -n Crab --tcuname Crab -v --submit
python DL1_to_DL2.py -c config_dl1_to_dl2.yaml -n Crab --distance 2 --ra 83.633080 --dec 22.014500 -v --submit
python DL1_to_DL2.py -c config_dl1_to_dl2.yaml -n Crab --runlist $CONFIG_FOLDER/Crab_Nov2020.txt -v --submit