Commit 020ed121 authored by Gaetano Scandariato's avatar Gaetano Scandariato
Browse files

Refactor non-time trend handling and improve residuals plotting

- Removed unused variable `_EIV_TRIGGERED` from logger.
- Updated `_compute_nontime_trend` to return a dictionary of components instead of a single result.
- Introduced `panel_nontime_residuals.py` to render PDF pages showing partial residuals for non-time components.
- Modified `render_panel_overlay` to simplify y-label handling by removing non-time trend checks.
- Deleted `panel_raw.py` as it is no longer needed.
- Enhanced `generate_report` to include non-time component partial residuals for both individual visits and overall.
- Adjusted `prepare_plot_data` to handle non-time trends more effectively, including mapping component names to their X-variables.
- Updated plotting functions to accommodate changes in data structures for non-time trends.
- Improved figure layout management by removing raw data panel references and adjusting axes accordingly.
parent a81d624c
Loading
Loading
Loading
Loading
Loading
+0 −27
Changes for .kilo/panel_alignment_explanation.md: 0 added lines, 27 removed lines.
Original line number Diff line number Diff line
# Alignment between Panel 0 and Panel 1

## Affected Modules
This logic is primarily implemented and managed in:
- `PyAETNA/reporting/plot_data_prep.py`: Handles the pre-computation of detrended data (`overlay_y`) and the preparation of the `PlotContext`.
- `PyAETNA/reporting/panel_raw.py`: Renders Panel 0, showing the raw data and the non-time correction line.
- `PyAETNA/reporting/panel_overlay.py`: Renders Panel 1, showing the detrended data and model components.

## Overview
There is a conceptual and mathematical distinction between the data presented in Panel 0 (Raw Data) and Panel 1 (Model Overlay) regarding how non-time corrections (e.g., polynomial or GP trends based on non-BJD variables) are handled.

## Panel 0: Raw Data and Corrections
Panel 0 is designed to show the unmodified observations alongside the corrections being applied. 
- **The Correction Line**: The line plotted in Panel 0 represents the total correction including the normalization baseline. 
    - For **Radial Velocity (RV)**: $\text{Line} = \text{norm} + \text{correction}$
    - For **Light Curves (LC)**: $\text{Line} = \text{norm} \times (1 + \text{correction})$
- **Purpose**: This allows the user to visually verify that the correction line follows the raw data trends and that the normalization baseline is correctly positioned.

## Panel 1: Model Overlay and Detrending
Panel 1 is aimed at showing the data, models, and confidence bands **after** non-time trends have been removed.
- **Detrended Data**: The data points in Panel 1 are computed by subtracting the non-time correction from the raw data.
    - For **RV**: $y_{\text{overlay}} = y_{\text{raw}} - (\text{norm} + \text{correction})$
    - For **LC**: $y_{\text{overlay}} = y_{\text{raw}} - (\text{norm} \times (1 + \text{correction}))$
- **Confidence Bands**: It is important to note that the confidence bands for the models shown in Panel 1 **still take into account the uncertainties from the non-time trends**. Even though the trends themselves have been removed from the plotted positions of the data and models, their contribution to the overall uncertainty budget is preserved in the bands. This need to be fixed.

## Summary of Misalignment
The corrected data in Panel 1 seems good, but the correction represented in Panel 0 shows a systematic offset wrt the data. The goal is to make Panel 0 shows the nontime correction applied to the data that are then shown in Panel 1. Panel 1, moreover, should show all the remaining components and shouldn't contain any reference to nontime trends.
+0 −46
Changes for PyAETNA/.kilo/psd_plotting_plan.md: 0 added lines, 46 removed lines.
Original line number Diff line number Diff line
# Plan: Enhanced PSD Plotting in PyAETNA

## Objective
Improve the Power Spectral Density (PSD) diagnostic plots in the reports to provide a full variance budget analysis. This allows for a direct comparison between the observed data power, the nominal noise floor, the fitted white noise (jitter), and the GP model's correlated power.

## Proposed Plot Components
For the "GP + noise" panel (Row 1), the following components will be plotted:

1.  **Lomb-Scargle (LS) Periodogram**: The observed power of the detrended data (Black line).
2.  **Nominal Noise Baseline**: A flat line representing the PSD of the nominal data uncertainties $\langle \sigma_{\text{nom}}^2 \rangle$.
3.  **Total White Noise Baseline**: A flat line representing nominal uncertainties plus the MAP jitter $\langle \sigma_{\text{nom}}^2 \rangle + \sigma_{\text{jitter}}^2$. (Shown only if `log_jitter` is finite).
4.  **GP Kernel PSD**: The theoretical PSD of the GP kernel $S_{\text{Kernel}}(f)$. (Shown only if GP is active).
5.  **Total Model PSD**: The sum of the GP Kernel PSD and the Total White Noise Baseline: $S_{\text{Total}}(f) = S_{\text{Kernel}}(f) + S_{\text{WhiteNoise}}$. (Shown only if GP is active).

## Mathematical Logic
The LS PSD is normalized such that $\int S(f) df = \text{Var}(\text{data})$. 
To maintain consistency, the flat noise baselines are calculated as:
$$S_{\text{white}} = \frac{\sigma_{\text{white}}^2}{f_{\text{max}} - f_{\text{min}}}$$
where $\sigma_{\text{white}}^2$ is the variance of the white noise component.

## Implementation Steps

### 1. API Update
- Modify `plot_periodograms` in `PyAETNA/reporting/periodogram_plots.py` to accept `nominal_uncertainties` and `map_jitter`.
- Update the call site in `PyAETNA/reporting/pdf_report.py` to pass these values from the model/data objects.

### 2. Calculation Logic
- Compute $f_{\text{range}} = f_{\text{max}} - f_{\text{min}}$.
- Calculate $S_{\text{nom}} = \text{mean}(\sigma_{\text{nom}}^2) / f_{\text{range}}$.
- Calculate $S_{\text{jitter}} = (\text{mean}(\sigma_{\text{nom}}^2) + \sigma_{\text{jitter}}^2) / f_{\text{range}}$.

### 3. Plotting Updates
- Use `ax.axhline` for the flat baselines.
- Use `ax.plot` for the GP and Total Model curves.
- Implement conditional checks:
    - Jitter baseline $\rightarrow$ `if map_jitter is not None and not np.isinf(log_jitter)`.
    - GP lines $\rightarrow$ `if gp_kernel is not None and gp_x_variable == 'BJD'`.

### 4. Legend & Styling
- Update the legend to clearly label: "LS Periodogram", "Nominal Noise", "Total White Noise", "GP Kernel PSD", and "Total Model PSD".
- Use distinct colors and line styles (e.g., dotted for nominal, dashed for jitter).

## Verification
- Generate a report for a target with a known GP fit.
- Verify that the integral of the "Total Model PSD" visually matches the integral of the LS periodogram.
- Ensure that the noise floor of the LS periodogram aligns with the "Total White Noise" baseline.

PyAETNA/.kilo/step1.md

deleted100644 → 0
+0 −18
Changes for PyAETNA/.kilo/step1.md: 0 added lines, 18 removed lines.
Original line number Diff line number Diff line
# Step 1: Enhance PlotContext and Data Preparation

## Context
`PlotContext` currently aggregates non-time trends into a single entity. To enable detailed partial residual analysis, we need to track individual non-time components separately.

## Purpose
Modify the data preparation pipeline to:
1. Identify each individual non-time component.
2. Retrieve the corresponding observable for each component from `plData.dBasis`.
3. Compute the Maximum A Posteriori (MAP) estimate and the 16-84% confidence bands for each component.

## Files to Modify
- `PyAETNA/reporting/plot_data_prep.py`

## Verification & Tests
- **Component Tracking**: Verify that `PlotContext` now contains a list or dictionary of individual non-time components rather than a single aggregate.
- **Data Integrity**: Check that the array lengths of the computed MAP and confidence bands match the length of the observables.
- **Summation Check**: Verify that the sum of all individual non-time components equals the total non-time trend previously calculated.

PyAETNA/.kilo/step2.md

deleted100644 → 0
+0 −17
Changes for PyAETNA/.kilo/step2.md: 0 added lines, 17 removed lines.
Original line number Diff line number Diff line
# Step 2: Create the Partial Residuals Renderer

## Context
The necessary component-level data is now available in `PlotContext`. We now need a visualization layer to render this data into the final report.

## Purpose
Create a new rendering module `panel_nontime_residuals.py` to generate a PDF page. For each identified non-time component:
1. **Compute Partial Residuals**: Calculate $y_{raw} - (\text{Total Model} - \text{Component})$. This isolates the effect of the specific component.
2. **Visualization**: Plot these partial residuals against the corresponding observable.
3. **Overlay**: Plot the MAP line and the 16-84% confidence bands over the scatter plot to visualize the fit quality for that specific component.

## Files to Create
- `PyAETNA/reporting/panel_nontime_residuals.py` (New File)

## Verification & Tests
- **Visual Inspection**: Generate a test PDF and verify the layout of the partial residuals page.
- **Trend Validation**: Verify that the MAP line correctly follows the trend of the partial residuals for each component.

PyAETNA/.kilo/step3.md

deleted100644 → 0
+0 −16
Changes for PyAETNA/.kilo/step3.md: 0 added lines, 16 removed lines.
Original line number Diff line number Diff line
# Step 3: Integrate into PDF Report

## Context
The data preparation and the rendering module are complete, but the new page is not yet integrated into the main report generation workflow.

## Purpose
Modify the main PDF report logic to trigger the partial residuals renderer.
1. Detect if non-time trends are present in the current fit results.
2. If present, call the `panel_nontime_residuals.py` renderer to insert the partial residuals page into the final PDF document.

## Files to Modify
- `PyAETNA/reporting/pdf_report.py`

## Verification & Tests
- **End-to-End Test**: Run a full fit using a configuration that includes non-time components.
- **PDF Validation**: Open the resulting PDF and verify that the "Partial Residuals" page is present and correctly populated with data.
Loading