Updates notebooks for AGN SC2025

font.size: 12.0

font.family: serif

axes.linewidth: 1.0

axes.unicode_minus: False

axes.prop_cycle: cycler('color', ['1f77b4', 'ff7f0e', '2ca02c', 'd62728', '9467bd', '8c564b', 'e377c2', '7f7f7f', 'bcbd22', '17becf'])

axes.autolimit_mode: data

xtick.top: True

xtick.bottom: True

xtick.major.size: 7

xtick.minor.size: 3.5

xtick.major.width: 1.0

xtick.minor.width: 1.0

xtick.direction: in

xtick.minor.visible: True

xtick.major.top: True

xtick.major.bottom: True

xtick.minor.top: True

xtick.minor.bottom: True

ytick.left: True

ytick.right: True

ytick.major.size: 7

ytick.minor.size: 3.5

ytick.major.width: 1.0

ytick.minor.width: 1.0

ytick.direction: in

ytick.minor.visible: True

ytick.major.left: True

ytick.major.right: True

ytick.minor.left: True

ytick.minor.right: True

legend.loc: best

legend.frameon: False

figure.titleweight: normal

figure.figsize: 6.4, 4.8

figure.dpi: 100

image.origin: lower

savefig.format: pdf

savefig.bbox: tight

%% Cell type:markdown id:a737c50d-9836-409d-94a9-79fc98641a02 tags:

# AGILE DR1 (Viitanen et al., in prep.)

AGILE: AGN in the LSST era (Viitanen et al., in prep.) is an end-to-end

simulation pipeline and part of the official LSST INAF in-kind contribution.

AGILE is designed to allow for the characterization of the AGN population as

seen by Rubin-LSST. The pipeline starts with the creation of an empirical truth

catalog of AGN, galaxies, and stars. Using variability recipes for AGN and

stars, the truth catalog is then fed to the image simulator (imSim), capable of

generating raw images from the LSSTCam. Finally, the resulting raw images are

analyzed using the official science pipelines.

The first data release of AGILE corresponds to $1\,{\rm deg}^2$ (21 LSSTCam

detectors) and 3 years of LSST survey in the COSMOS (150.117, +2.205)

deep-drilling field. The underlying truth catalog has an area of 24 deg2, and

extends to $0.2 < z < 5.5$ and host galaxy mass $\log(M/M_\odot) > 8.5$.

The survey baseline is based on baseline v4.0. However, to optimize lightcurve

cadence over depth, we prune the observation sequence by selecting only the

first exposure of each sequence. The total number of visits is $1\,441$.

The LSST Science Pipelines (version 8.0.0) are run on the simulated raw images

in order to produce single-visit calibrated exposures and source catalogs.

Then, the single-visit images are combined to produce the coadded image and

object catalogs. Finally, forced photometry is performed on the single-visit

images based on the object catalog.

With respect to the LSST data product, the AGILE data model follow closely that

of DP0.2 (https://dp0-2.lsst.io/data-products-dp0-2/index.html). LSST Butler

may be used to access both images and catalog. For the access of catalogs we

also provide a sqlite3 database which mimics the functionality and schema of

the TAP service in DP0.2 / DP1.

%% Cell type:markdown id:85b8bc23-e64e-4d0f-bcd8-5e27d4def1a7 tags:

## Demonstration notebooks

We provide the following notebooks demonstrating the usage of AGILE DR1:

- [agile_truth.ipynb](agile_truth.ipynb): explore the underlying truth catalog

- [agile_sed.ipynb](agile_sed.ipynb): explore the measured and truth SEDs

- [agile_lightcurve.ipynb](agile_lightcurve.ipynb): explore the measured forced photometry and truth lightcurves

- [agile_images.ipynb](agile_images.ipynb): explore simulated single-visit and deep coadded images

- [agile_compare_dp1.ipynb](agile_compare_dp1.ipynb): compare AGILE DR1 to DP1 ECDFS

The full AGILE DR1 (incl. catalogs, images, SEDs, lightcurves) is hosted in the

Naples ADHOC cluster. Here, a small subset of the available data is provided

for demonstration purposes.

%% Cell type:markdown id:342ae9fd-3da9-4578-abbf-c44af0573133 tags:

## Requirements for running the notebooks

0. download the demo repository

1. download the AGILE master database file 'master.db' (large file, ~128G)

2. install the python requirements from 'requirements.txt'

3. (required only for using LSST butler) install the LSST Science Pipelines (https://pipelines.lsst.io/#installation)

%% Cell type:markdown id:a2b3ce63-c6d4-4b9b-995b-d73a1feccb37 tags:

## Contact:

- Angela Bongiorno (angela.bongiorno@inaf.it, INAF-OAR)

- Akke Viitanen (akke.viitanen@inaf.it, INAF-OAR, Univ. Geneva)

- Akke Viitanen (akke.viitanen@iki.fi, INAF-OAR, Univ. Geneva)

- Ivano Saccheo (ivano.saccheo@inaf.it, INAF-OAR, Univ. Bristol)

Created by AV on 2025/07/09

%% Cell type:code id:24014b1e-6d87-4234-8b95-0d49a9a7b9f0 tags:

%% Cell type:code id:0f69a013-bb12-43ab-95bb-ae40917b103d tags:

``` python

```

   "metadata": {},

   "outputs": [],

   "source": [

    "from astropy.table import Table\n",

    "import os\n",

    "import sqlite3\n",

    "import pandas as pd\n",

    "import matplotlib.pyplot as plt\n",

    "\n",

    "from astropy.wcs import WCS\n",

    "from astropy import units as u\n",

    "from astropy.coordinates import SkyCoord\n",

    "from astropy.table import Table\n",

    "import matplotlib as mpl\n",

    "import matplotlib.pyplot as plt\n",

    "import numpy as np\n",

    "import pandas as pd\n",

    "import pyvo\n",

    "import os"

    "from scipy.stats import binned_statistic\n",

    "\n",

    "try:\n",

    "    from lsst.daf.butler import Butler\n",

    "except ImportError:\n",

    "    print(\"Warning: could not initialize butler.\")"

   ]

  },

  {

     "name": "stdout",

     "output_type": "stream",

     "text": [

      "CPU times: user 22.5 ms, sys: 27.1 ms, total: 49.6 ms\n",

      "Wall time: 181 ms\n"

      "CPU times: user 69 ms, sys: 37.3 ms, total: 106 ms\n",

      "Wall time: 279 ms\n"

     ]

    },

    {

     "data": {

      "text/html": [

       "<div><i>Table length=5</i>\n",

       "<table id=\"table125919508342320\" class=\"table-striped table-bordered table-condensed\">\n",

       "<table id=\"table133795347070320\" class=\"table-striped table-bordered table-condensed\">\n",

       "<thead><tr><th>objectId</th><th>tract</th><th>patch</th><th>coord_ra</th><th>coord_dec</th><th>refExtendedness</th><th>refBand</th><th>match_id</th><th>u_psfFlux</th><th>u_psfFluxErr</th><th>u_psfFlux_flag</th><th>u_extendedness_flag</th><th>lsst-u_point</th><th>lsst-u_total</th><th>g_psfFlux</th><th>g_psfFluxErr</th><th>g_psfFlux_flag</th><th>g_extendedness_flag</th><th>lsst-g_point</th><th>lsst-g_total</th><th>r_psfFlux</th><th>r_psfFluxErr</th><th>r_psfFlux_flag</th><th>r_extendedness_flag</th><th>lsst-r_point</th><th>lsst-r_total</th><th>i_psfFlux</th><th>i_psfFluxErr</th><th>i_psfFlux_flag</th><th>i_extendedness_flag</th><th>lsst-i_point</th><th>lsst-i_total</th><th>z_psfFlux</th><th>z_psfFluxErr</th><th>z_psfFlux_flag</th><th>z_extendedness_flag</th><th>lsst-z_point</th><th>lsst-z_total</th><th>y_psfFlux</th><th>y_psfFluxErr</th><th>y_psfFlux_flag</th><th>y_extendedness_flag</th><th>lsst-y_point</th><th>lsst-y_total</th></tr></thead>\n",

       "<thead><tr><th>int64</th><th>int64</th><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>str1</th><th>int64</th><th>float64</th><th>float64</th><th>int64</th><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int64</th><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int64</th><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int64</th><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int64</th><th>int64</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>int64</th><th>int64</th><th>float64</th><th>float64</th></tr></thead>\n",

       "<tr><td>4229812435961270187</td><td>9813</td><td>37</td><td>150.45161253533473</td><td>2.769216081333327</td><td>0.0</td><td>i</td><td>6289499</td><td>71545.74178409649</td><td>165.72055349945435</td><td>0</td><td>0</td><td>74.93854924610565</td><td>74.96878496038477</td><td>78667.2370029365</td><td>59.216265814946176</td><td>0</td><td>0</td><td>81.70469527439508</td><td>81.77857248232993</td><td>97789.44687779424</td><td>69.72839269148996</td><td>0</td><td>0</td><td>101.34551160019339</td><td>101.48344499599636</td><td>113879.77475497374</td><td>76.05851038996032</td><td>0</td><td>0</td><td>117.7671065777968</td><td>117.95330310617194</td><td>136216.81501546915</td><td>117.59358476083352</td><td>0</td><td>0</td><td>140.4938753930129</td><td>140.80340532336606</td><td>129648.65355257494</td><td>350.3645937073211</td><td>0</td><td>0</td><td>134.98143143221165</td><td>135.47705503567482</td></tr>\n",

    {

     "data": {

      "application/vnd.jupyter.widget-view+json": {

       "model_id": "00a528eb881840519dc952ed62739dbc",

       "model_id": "17cfd0c66e7b450b9d4ea42753db2e08",

       "version_major": 2,

       "version_minor": 0

      },

    {

     "data": {

      "application/vnd.jupyter.widget-view+json": {

       "model_id": "8c875750f0c94113bdb09a8db93f2dd9",

       "model_id": "1e26189b57af40a59c5bb9aaf5ff1223",

       "version_major": 2,

       "version_minor": 0

      },

    {

     "data": {

      "application/vnd.jupyter.widget-view+json": {

       "model_id": "d0635166e27e440f8e166aa95f4ccaee",

       "model_id": "0ed4e232809c4ddeaa5a257cbc0758a4",

       "version_major": 2,

       "version_minor": 0

      },

    {

     "data": {

      "application/vnd.jupyter.widget-view+json": {

       "model_id": "dfd994f0266c4efda559838fd88866a3",

       "model_id": "64a589e6fe0c4226ba3e24b5a5367229",

       "version_major": 2,

       "version_minor": 0

      },

    {

     "data": {

      "application/vnd.jupyter.widget-view+json": {

       "model_id": "6df4679fb1e0421cb713af51ba187049",

       "model_id": "95a940d4e5ca4b61b4de3eb611cad2fa",

       "version_major": 2,

       "version_minor": 0

      },