.. _dl3:

DL3 files generation
====================

There are two different scripts to generate the DL3 file:

* `DL2_to_DL3.py`, which allows to analyze DL2 data generated by the user beforehand from DL1 data (:ref:`dl2`)
* `create_DL3_files_from_database.py`, which uses the DL2 files created auyomatically by LST OSA.

.. note::

  At the moment is recommended the use of the first script `DL2_to_DL3.py` and generate you own DL2 data, since the automatized generation of DL2 data is not optimized.


DL2_to_DL3.py
--------------

The script `DL2_to_DL3.py` uses DL2 data generated by the user beforehand (:ref:`dl2`) and produces DL3 files.
Usage:

.. code-block::

    usage: DL2_to_DL3.py [-h] [--prod PROD] [--outdir OUTDIR] [--config CONFIG] [--config-analysis CONFIG_ANALYSIS]
                     [--verbose] [--source_name SOURCE_NAME] [--tcuname TCUNAME] [--runlist RUNLIST]
                     [--distance DISTANCE] [--ra RA] [--dec DEC] [--submit] [--dry] [--globber] [--cut_file CUT]
                     [--gh_cut GH_CUT] [--theta_cut THETA_CUT]

    DL2 to DL3 converter

    optional arguments:
      --config CONFIG, -c CONFIG
                            Specify a personal config file for the analysis
      --config-analysis CONFIG_ANALYSIS
                            Specify a config file which describes analysis profile to use
      --cut_file CUT, -cut CUT
                            Cut file
      --dec DEC             Dec coordinate of the target. To add if you want to use custom position
      --distance DISTANCE, -dis DISTANCE
                            Max distance in degrees between the target position and the run pointing position for the run
                            selection, negative value means no selection using this parameter (default: -1).
      --dry                 Make a dry run, no true submission
      --gh_cut GH_CUT       Fixed selection cut for gh_score (gammaness)
      --globber, -g         If True, overwrites existing output file without asking
      --outdir OUTDIR, -o OUTDIR
                            Directory to store the output
      --prod PROD, -p PROD  Prod to use (default: v0.9.4)
      --ra RA               RA coordinate of the target. To add if you want to use custom position
      --runlist RUNLIST, -rl RUNLIST
                            File with a list of run and the associated night to be analysed
      --source_name SOURCE_NAME, -n SOURCE_NAME
                            Name of the source
      --submit              Submit the cmd to slurm on site
      --tcuname TCUNAME     Apply run selection based on TCU source name
      --theta_cut THETA_CUT
                            Fixed selection cut for theta
      --verbose, -v         Increase output verbosity
      -h, --help            show this help message and exit

It makes use of a configuration file `config_dl2_to_dl3.yaml` (option ``--config``).
Edit the configuration file changing the paths based on the analysis you want to perform.
For the moment the IRF file to use has to be specified in the configuration file.
IRF files with different settings can be found in ``/fefs/aswg/data/mc/IRF/``, otherwise create your own (:ref:`irf_generation`).

Configuration file:

.. code-block::
    
    # Directory where job files are written
    jobmanager: ../jobmanager

    # Database file name
    db: database.csv

    # Path to main data folder of the user
    # Change it accordingly to your working env
    base_dir: /fefs/aswg/alice.donini/Analysis/data

    # Path to personal directory where output data will be saved.
    # Uncomment and modify in case you want to use a non standard path
    #output_folder: ../DL3/Crab

    # Path to the folder where cut files are stored
    cut_folder: ../cuts

    # Path to IRF file, change it based on your analysis
    irf_file: "/fefs/aswg/data/mc/IRF/20200629_prod5_trans_80/zenith_20deg/south_pointing/20210416_v0.7.3_prod5_trans_80_local_taicut_8_4/off0.4deg/irf_20210416_v073_prod5_trans_80_local_taicut_8_4_gamma_point-like_off04deg.fits.gz"

    # Values for automatic selection of DL2 data from personal directories
    dl2_data:
      night: [20210911, 20210912]           # day(s) of observation (more than one is possible)
      version: v0.9.1                       # v0.7.3, v0.8.4, v0.9, v0.9.1
      cleaning: tailcut84

The generated DL3 data will be saved in a personal folder following the tree structure visible in :ref:`install`.
It is possible to define a different ouput folder adding the option ``-o``.

.. note::

    Same cuts as the one used for the generation of IRF should be used.


create_DL3_files_from_database.py
----------------------------------

For a fast analysis with the use of DL2 files automatically mergerd by LST OSA, run for example:

.. code-block:: bash

    python DL2_to_DL3.py --tcuname "grb210807A"  --ra 76.677 --dec 58.245 --outdir ../data/ --submit -v
    python DL2_to_DL3.py --tcuname "Crab" --ra 83.633080 --dec 22.014500 --outdir ../data/  --submit -v
    python DL2_to_DL3.py --distance 2 --ra 83.633080 --dec 22.014500 --outdir ../data/  --submit -v

`--tcuname` is the name of the TCU configuration and is used to select the runs to analyze from the Database file, unless specific nights or list of runs are specified using option `--night` and `--run`. Also Ra, Dec position can be specified if already known, and are mandatory in case the name of the target can not be resolved. If the option distance is specified, runlist is created using all runs with poiting position within the angular distance specified.
If no cut configuration file is given, standard configuration is applied.