move fbin calculation to CatalogStar

%% Cell type:markdown id:70b92b7a-6039-4468-bf49-87dc87c2d965 tags:

# Create an example AGILE truth catalog

This notebooks demonstrates the basic usage of the truth catalog library, and generates an extermely small truth catalog for testing purposes.

In AGILE, each individual class of source (e.g. AGN, galaxy, star) is first contained in its own catalog. The final step in the truth catalog creation is to simply combine these individual source catalogs into one combined catalog that can be used for future image simulations.

The different classes of objects considered are:

1. Galaxies

2. AGNs

3. Stars

4. Binary stars

The following notebook may be accessed [here](https://www.ict.inaf.it/gitlab/akke.viitanen/lsst_inaf_agile/-/blob/a619e4244f56e731d11724f8a70dd5e7d6e1fda4/docs/notebooks/create_truth_catalog.ipynb)

%% Cell type:markdown id:3cbfa609-8950-474a-97f0-9fcefb71c103 tags:

## Initialize

%% Cell type:code id:9759604f-8be1-4a5d-b63a-88da53fab067 tags:

``` python

import os

if not os.path.exists("src/lsst_inaf_agile"):

    os.chdir("../../")

    os.getcwd()

```

%% Cell type:code id:a01c8b5b-8e8e-409d-94c2-185806a70646 tags:

``` python

import logging

import numpy as np

from lsst_inaf_agile.catalog_agn import CatalogAGN

from lsst_inaf_agile.catalog_combined import CatalogCombined

from lsst_inaf_agile.catalog_galaxy import CatalogGalaxy

from lsst_inaf_agile.catalog_star import CatalogStar

from lsst_inaf_agile.egg import Egg

from lsst_inaf_agile.image_simulator import ImageSimulator

from lsst_inaf_agile.merloni2014 import Merloni2014

from lsst_inaf_agile import util

```

%% Cell type:code id:67317893-e15d-43c6-9f8f-94e37b27e3af tags:

``` python

# Setup the dirname

dirname = "data/tests/test_notebook_create_truth_catalog"

```

%% Cell type:markdown id:897ccf7a-3bdc-4ae6-943d-20a7eb2b6c9e tags:

## Create the EGG galaxy catalog

%% Cell type:code id:37224123-c322-4cc8-beab-fe8d56d3455b tags:

``` python

# The following list of keyword arguments may be expanded on.

# Look into 'egg-gencat help'for all the available EGG arguments.

egg_kwargs = Egg.get_example_egg_kwargs(dirname + "/egg.fits")

egg_kwargs

```

%% Cell type:code id:e004eed5-a555-4307-80c2-ebe8460d4ae0 tags:

``` python

egg = Egg(egg_kwargs)

egg.run()

```

%% Cell type:code id:62685aff-dd9b-4054-92b6-5bf4f3e6cb6c tags:

``` python

catalog_egg = Egg.read(egg_kwargs["out"])

```

%% Cell type:code id:cb439284-3a83-45d6-8727-ee749892eb03 tags:

``` python

# Convert to the AGILE galaxy catalog class

catalog_galaxy = CatalogGalaxy(dirname, catalog_egg)

```

%% Cell type:markdown id:f0634979-f64c-40af-ac05-b8ab7095e39d tags:

## Create the AGN catalog

%% Cell type:code id:4f39b065-f97b-4aa7-9f59-8ed05794b724 tags:

``` python

# Create the AGN catalog

kwargs_catalog_agn = dict(

    dirname=dirname,

    catalog_galaxy=catalog_galaxy,

    type_plambda="zou+2024",

    save_sed=1,

    seed=20251005,

    merloni2014=Merloni2014(1, 0, 0.05, 0.95),

)

catalog_agn = CatalogAGN(**kwargs_catalog_agn)

```

%% Cell type:markdown id:1fd2389a-8048-41de-bbc1-b7c8848fb87d tags:

### Examine an example AGN

%% Cell type:code id:6a915c09-565d-4cc3-87ce-c7a1bb2ba24e tags:

``` python

# Select a type1 AGN

select = (catalog_agn["is_agn"] == 1) & (catalog_agn["is_optical_type2"] == 0)

# Select the first AGN by ID

ids = catalog_agn["ID"][select]

my_id = ids[0]

print(f"Found a type1 AGN with ID = {my_id}")

```

%% Cell type:markdown id:5ca1fcb0-67b4-4d9a-b72a-75ee712c4b11 tags:

### Examine an AGN SED

%% Cell type:code id:b00e821a-689e-47d8-a3af-841fef3b261a tags:

``` python

lam, flux = catalog_agn.get_sed(my_id)

plt.loglog(lam, flux)

```

%% Cell type:markdown id:b5c2b317-bae9-4cc7-a221-075c73ae7a77 tags:

### Estimate a single AGN light curve

%% Cell type:code id:c345c7d1-4bcb-439a-98ca-803689db5604 tags:

``` python

# Select a type1 AGN

select = (catalog_agn["is_agn"] == 1) & (catalog_agn["is_optical_type2"] == 1)

# Select the first AGN by ID

my_id = catalog_agn["ID"][select][0]

# Estimate its lightcurve.

# The lightcurve is estimated for 10-years with a cadence of one day.

# The returned flux is in the observer frame.

lc = catalog_agn.get_lightcurve(my_id, "lsst-r")

plt.plot(lc)

plt.xlabel("MJD")

plt.ylabel(r"flux $r$ [uJy]")

```

%% Cell type:markdown id:448f5d18-f961-4410-81e3-b5bbfe23c15c tags:

## Create the star and binary star catalogs

%% Cell type:code id:3b21fd72-661e-4094-b6d5-f09cef1f8403 tags:

``` python

# Create the star catalogs

catalog_star = CatalogStar(dirname, catalog_galaxy, is_binary=False)

catalog_binary = CatalogStar(dirname, catalog_galaxy, is_binary=True)

```

%% Cell type:code id:93b9dbe2-c79e-4c2d-a2f8-74e6195ddeff tags:

``` python

# NOTE: dal Tio+ simulated only 10% of the toal number of binary systems. They suggest a value of

# fbin=0.40 to account for this. To achieve this:

#   1) input star catalog is downsampled to 1 - fbin = 60% of the original size

#   2) the binary catalog is repeated 4 times

# The helper function below does these two steps automatically.

if "_once" not in globals():

    print(f"Old sizes are {len(catalog_star.stars)=}, {len(catalog_binary.stars)=}")

    catalog_star.stars, catalog_binary.stars = util.get_star_binary_fbin(

        catalog_star.stars, catalog_binary.stars, fbin=0.40, nrepeat=4

    catalog_star, catalog_binary = CatalogStar.get_star_binary_fbin(

        catalog_star, catalog_binary, fbin=0.40, nrepeat=4

    )

    _once = True

print(f"New sizes are {len(catalog_star.stars)=}, {len(catalog_binary.stars)=}")

```

%% Cell type:markdown id:5c6ad011-601a-433e-86e5-f67ffb50bc12 tags:

## Create the combined catalog of AGNs, galaxies, and stars

%% Cell type:code id:8901a2f1-86a1-4d43-a32e-6158f8f017a8 tags:

``` python

# NOTE: we set cache=False to prevent an existing catalog file to be loaded from the disk.

catalog_combined = CatalogCombined(

    dirname, catalog_galaxy, catalog_agn, catalog_star, catalog_binary, cache=False

)

```

%% Cell type:markdown id:d0d0a3cc-dbca-4675-b79b-665308717ba5 tags:

### Examine the truth catalog columns

Here we list all the available columns in the truth catalog. The meaning of these columns is described in the appendix of Viitanen+2026.

It is important to note that the truth catalog contains ONE row per object. That is, a single row corresponds to either an AGN, galaxy, or a star. Consequently, not all the different classes of objects have all the different columns available. For example, (host) galaxy stellar mass ('M') is only available for galaxies (and AGNs).

%% Cell type:code id:cd17c16a-2e08-48c8-b45a-46542673d5b2 tags:

``` python

catalog_combined.get_dtype()

```

%% Cell type:markdown id:64b46ad7-07a2-470a-b035-2a690d5dd0ee tags:

## Examine the output files

At this stage, the static truth catalog is completed. To find out which files have been written to disk, we check the output directory.

%% Cell type:code id:61db188b-6ca2-45f9-96e8-971daf27eb6d tags:

``` python

os.listdir(dirname)

```

%% Cell type:markdown id:a0074fd5-14b0-4a53-9c02-f9baa36cfb6f tags:

The explanation of the catalog files is as follows:

+ egg.fits -- the EGG galaxy catalog

+ agn.fits -- the AGN catalog

+ stars.fits -- the full star catalog

+ binaries.fits -- the full binary star catalog

+ catalog.fits -- the combined truth catalog

In addition to:

+ egg-seds* -- databse files needed to generate EGG SEDs (refer to EGG documentation)

+ seds/ -- AGN SEDs in EGG format (refer to EGG documentation)

+ lightcurves/ -- stored lightcurves (see the AGN catalog section above)

Note that by default, all galaxy SEDs only reside in the EGG database. These can be stored on disk on demand, but essentially doubles the disk space used which can be problematic on larger catalogs.

Also, light curves are only generated by request. In case images are simulated, then a subset of light curves will be estimated automatically for the sources within the region of interest.

%% Cell type:markdown id:01298432-fed8-4570-bf87-fbb507302845 tags:

## Example plot: (host) galaxy stellar mass versus redshift

%% Cell type:code id:73045582-05f0-4fb3-8202-e77862cfed06 tags:

``` python

plt.figure(dpi=200)

select = catalog_combined["Z"] > 0

plt.plot(catalog_combined["Z"][select], catalog_combined["M"][select], ".")

plt.xlabel(r"$z$")

plt.ylabel(r"$\log (M\,/\,M_\mathrm{star})$");

```

%% Cell type:markdown id:22e8a779-187b-4482-a750-2d72bfbd41eb tags:

## Example plot: g-band flux distribution

%% Cell type:code id:eb4c0774-7cc7-45e2-b84d-8a21826add84 tags:

``` python

# Example plot: distribution of the observed g-band flux.

plt.figure(dpi=200)

plt.hist(catalog_combined["lsst-g_total"], bins=np.logspace(-3, 3, 61))

plt.loglog()

plt.xlabel(r"lsst-g_total [uJy]")

plt.ylabel("frequency");

```

%% Cell type:markdown id:0bb771ab-a997-4f51-87aa-3fc3826c08db tags:

## A note on the occupation fraction

%% Cell type:markdown id:73415ddf-6699-4d14-8edd-f3f18d7dca7b tags:

In the simulation, it is preliminarily assumed that each galaxy has a SMBH, and

a corresponding value $M_\mathrm{BH}$. Local observations suggest that this

might not be the case, and especially at low values of Mstar, a BH might be

missing altogether. To facilitate for this, in the truth catalog a flag

``has_bh'' is provided. The formal definition of this flag is:

\begin{equation}

    \mathrm{has\_bh} = U < f_\mathrm{occ}(M_\mathrm{star}),

\end{equation}

where $U$ is a uniform random variable $U \sim \mathrm{Unif}(0, 1)$, and

$f_\mathrm{occ}$ follows the observational results of Zou+2025. This column may

be used as a weight for all calculations of statistical distributions. For

example, the galaxy BH mass function can be calculated with and without the

occupation fraction by simply weighting the mass function by unity (every

galaxy is expected to host a BH), or ``has_bh'' (galaxies are expected to host

BHs in accordance with the occupation fraction). An example is provided below.

%% Cell type:code id:385e6d41-6bca-4fea-975e-b8979609a91a tags:

``` python

# Select all galaxies

is_galaxy = catalog_combined["Z"] > 0.0

# Select central BHs

has_bh = catalog_combined["has_bh"]

# Combine the two selections

is_galaxy_and_has_bh = is_galaxy & has_bh

# Do some statistical tests

print(f"Total number of galaxies: {is_galaxy.sum()}")

print(f"Total number of galaxies with BHs: {is_galaxy_and_has_bh.sum()}")

# Plot Mstar vs. occupation fraction

plt.title("BH occupation fraction vs. stellar mass")

plt.plot(catalog_combined["M"][is_galaxy], catalog_combined["occupation_fraction"][is_galaxy], ".")

plt.xlabel(r"$\log \left( M_\mathrm{star} / M_\odot \right)$")

plt.ylabel(r"$f_\mathrm{occ}$")

```

%% Cell type:markdown id:99cf5e38-5d31-4aab-9112-b6ba6b76a0a3 tags:

For AGN, the logic remains the same. To weigh AGN by the occupation fraction,

one would use the product $\mathrm{is\_agn} \times \mathrm{has\_bh}$ instead of

simply $\mathrm{is\_agn}$. Logically, this is equivalent to the condition "has

BH AND BH is active". For $L_\mathrm{X} >

10^{42}\,\mathrm{erg}\,\mathrm{s}^{-1}$, this has a negligible effect on the

AGN population at large, as was investigated in Viitanen+2026.

%% Cell type:code id:54c2f484-11fb-46d8-a9db-a0cca9dc5664 tags:

``` python

# Select galaxies with BHs

has_bh = catalog_combined["has_bh"]

# Select AGNs

is_agn = catalog_combined["is_agn"]

# Select logLX > 42 objects for the sake of an example

is_loglx_gt_42 = catalog_combined["log_LX_2_10"] > 42

# Combine the selections

has_bh_and_is_agn = has_bh & is_agn

has_bh_and_is_agn = has_bh & is_agn & is_loglx_gt_42

print(f"Total number of AGNs: {is_agn.sum()}")

print(f"Total number of BHs with AGNs: {has_bh_and_is_agn.sum()}")

```

%% Cell type:code id:522423f3-dff7-40da-938d-3b3606160cf7 tags:

``` python

```

        )

        return ret

    @staticmethod

    def get_star_binary_fbin(catalog_star, catalog_binary, fbin=0.40, nrepeat=4, seed=1206):

        """

        Return a binary star catalog according to the description of dal Tio+ and the definition of fbin.

        According to dal Tio+ Sec. 3.2:

        (https://iopscience.iop.org/article/10.3847/1538-4365/ac7be6/pdf)

        ``

        Therefore, for the moment, we recommend a fbin value of 0.4, as

        being both most robust (see Dal Tio et al. 2021) and more

        consistent with the way the stellar densities were originally

        calibrated in TRILEGAL. As we simulated only one-tenth of expected

        binaries, the fbin value of 0.4 can be achieved by randomly

        selecting 60% of single stars and by multiplying by 4 times the

        number of binary systems present in the same regions.

        ``

        Parameters

        ----------

        star: CatalogStar

            Input single CatalogStar

        binary: CatalogStar

            Input binary CatalogStar

        fbin: float

            Assumed fbin value. Stars will be downsampled to 1 - fbin of the

            original size

        nrepeat: int

            Number of repeats on binary star catalog.

        seed: int

            Random number seed.

        Returns

        -------

        star2, binary2: tuple[ArrayLike, ArrayLike]

            Down- and upsampled stellar and binary star catalogs.

        Examples

        --------

        >>> from astropy.table import Table

        >>> # Create mock catalogs with sizes (1000, 100)

        >>> star = Table({"a": [0.0] * 1000})

        >>> binary = Table({"a": [0.0] * 100, "b": [0.0] * 100})

        >>> star2, binary2 = get_star_binary_fbin(star, binary)

        >>> len(star2), len(binary2)

        (617, 400)

        >>> # Small catalog will warn about insufficient statistics but succeeds

        >>> star = Table({"a": [0.0] * 5})

        >>> binary = Table({"a": [0.0] * 2, "b": [0.0] * 2})

        >>> star2, binary2 = get_star_binary_fbin(star, binary)

        >>> len(star2), len(binary2)

        (2, 8)

        """

        # Set the seed

        np.random.seed(seed)

        # Downsample the stellar catalog

        is_star = np.random.rand(len(catalog_star.stars)) < 1 - fbin

        # Copy the binary catalog nrepeat times

        binary2 = np.repeat(catalog_binary.stars, nrepeat)

        # Copy over ra/dec/etc from the REMAINING stellar catalog

        # NOTE: in some small catalog cases the number of remaining stars is not

        # enough to sample for the binary catalog. In these cases, set replace=True

        is_not_enough = len(binary2) > (~is_star).sum()

        if is_not_enough:

            logger.warning(

                "Small stellar catalog. " "Can not sample binary stars sufficiently. " "Will use replace=True"

            )

        star2 = np.random.choice(catalog_star.stars[~is_star], size=len(binary2), replace=is_not_enough)

        # Copy over the relevant columns

        for column in catalog_binary.stars.dtype.names:

            if column in catalog_star.stars.dtype.names:

                binary2[column] = star2[column]

        # Update the two catalogs

        catalog_star.stars = catalog_star.stars[is_star]

        catalog_binary.stars = binary2

        catalog_star.catalog = catalog_star.get_catalog()

        catalog_binary.catalog = catalog_binary.get_catalog()

        assert len(catalog_star.stars) == len(catalog_star.catalog)

        assert len(catalog_binary.stars) == len(catalog_binary.catalog)

        return catalog_star, catalog_binary

        is_star = self.catalog.get_is_star()

        n_galaxy, n_star, n_binary = self.catalog.get_number_galaxy_star_binary()

        assert is_galaxy.sum() == n_galaxy

        assert is_star.sum() == n_star + n_binary

        assert is_star.sum() == n_star + n_binary, (is_star.sum(), n_star, n_binary)

        uid = self.catalog["ID"]

        ra = self.catalog["RA"]

    Returns

    -------

    comoving_volume: float

        The comoving volume (Mpc3) corresponding to the input arguments.

        The comoving volume segment in Mpc3 corresponding to the input

        arguments.

    Examples

    --------

    return S1 * np.ma.true_divide(E2_max**idx - E2_min**idx, E1_max**idx - E1_min**idx)

def luminosity_to_flux(wavlen, luminosity, redshift, distance_in_cm, use_igm=True):

def luminosity_to_flux(wavlen, luminosity_nu, redshift, distance_in_cm, use_igm=True):

    """

    Convert luminosity (in erg/s/ang) to flux in uJy. Default distance is 10pc.

    Convert luminosity_nu (in erg/s/Hz) to flux in uJy. Default distance is 10pc.

    Parameters

    ----------

    wavlen: float

        Rest-frame wavelength in angstroms.

    luminosity: float

        Rest-frame luminosity in erg/s/angstrom.

    luminosity_nu: float

        Rest-frame monochromatic luminosity in erg/s/Hz.

    redshift: float

        Redshift of the source.

    distance_in_cm: float

        Luminosity distance in cm.

    use_igm: bool

        Apply reddening by the intergalactic medium?

    Returns

    -------

    flux: float

        Flux in uJy at the given redshift.

    Examples

    --------

    >>> from astropy.cosmology import FlatLambdaCDM

    # To uJy in observed frame

    log_flux = (

        np.log10(luminosity)

        np.log10(luminosity_nu)

        + np.log10(wavlen_observed)

        + np.log10((u.erg / u.s / u.cm**2 / u.Hz).to(u.uJy))

        - np.log10(constants.c.to(u.angstrom / u.s).value)

    return ((1 * u.au / d).si * u.rad).to(u.mas).value

def get_star_binary_fbin(star, binary, fbin=0.40, nrepeat=4, seed=1206):

    """

    Return a binary star catalog according to the description of dal Tio+ and the definition of fbin.

    According to dal Tio+ Sec. 3.2:

    (https://iopscience.iop.org/article/10.3847/1538-4365/ac7be6/pdf)

    "

    Therefore, for the moment, we recommend a fbin value of 0.4, as

    being both most robust (see Dal Tio et al. 2021) and more

    consistent with the way the stellar densities were originally

    calibrated in TRILEGAL. As we simulated only one-tenth of expected

    binaries, the fbin value of 0.4 can be achieved by randomly

    selecting 60% of single stars and by multiplying by 4 times the

    number of binary systems present in the same regions.

    "

    Examples

    --------

    >>> from astropy.table import Table

    >>> star = Table({"a": [0.0, 1.0, 2.0] * 1000})

    >>> binary = Table({"a": [0.0, 1.0, 2.0] * 100, "b": [0.0, 1.0, 2.0] * 100})

    >>> star2, binary2 = get_star_binary_fbin(star, binary)

    >>> len(star2), len(binary2)

    (1785, 1200)

    >>> # Small catalog will warn about insufficient statistics but succeeds

    >>> star = Table({"a": [0.0, 1.0, 2.0] * 5})

    >>> binary = Table({"a": [0.0, 1.0, 2.0] * 2, "b": [0.0, 1.0, 2.0] * 2})

    >>> star2, binary2 = get_star_binary_fbin(star, binary)

    >>> len(star2), len(binary2)

    (10, 24)

    """

    # Set the seed

    np.random.seed(seed)

    # Downsample the stellar catalog

    is_star = np.random.rand(len(star)) < 1 - fbin

    # Copy the binary catalog nrepeat times

    binary2 = np.repeat(binary, nrepeat)

    # Copy over ra/dec/etc from the REMAINING stellar catalog

    # NOTE: in some small catalog cases the number of remaining stars is not

    # enough to sample for the binary catalog. In these cases, set replace=True

    is_not_enough = len(binary2) > (~is_star).sum()

    if is_not_enough:

        logger.warning(

            "Small stellar catalog. " "Can not sample binary stars sufficiently. " "Will use replace=True"

        )

    star2 = np.random.choice(star[~is_star], size=len(binary2), replace=is_not_enough)

    for column in binary.dtype.names:

        if column in star.dtype.names:

            binary2[column] = star2[column]

    return star[is_star], binary2

def _get_ratio_estimated_true(value_estimated: float, value_true: float) -> float:

    """

    Calculate ratio between estimated value and true value.

    The "ratio" is defined as (y_est - y_true) / y_true.

    Parameters

    ----------

    value_estimated: float

        Estimated value.

    value_true: float

        True value.

    Returns

    -------

    ratio: float

        The ratio defined as (value_estimated - value_true) / value_true.

    Examples

    --------

    np.float64(0.0)

    >>> _get_ratio_estimated_true(2.0, 1.0)

    np.float64(1.0)

    >>> _get_ratio_estimated_true(99.0, 0.0)

    masked

    """

    return np.ma.true_divide(np.abs(value_estimated - value_true), value_true)

    Reference is Hoaglin+ 1983. See also Sec. 4.1 of

    https://iopscience.iop.org/article/10.1088/0004-637X/690/2/1236/meta

    Parameters

    ----------

    value_estimated: float

        Estimated value.

    value_true: float

        True value.

    Examples

    --------

    >>> get_sigma_nmad(1.0, 1.00)

    """

    Calculate catastrophic error fraction from the set of estimated / true values.

    Parameters

    ----------

    value_estimated: float

        Estimated value.

    value_true: float