Commit 2f003be6 authored by Laura, Jason R.'s avatar Laura, Jason R.
Browse files

Adds DB tests without a live DB and sensor model tests

parent 72f0b1d9
Loading
Loading
Loading
Loading
+18 −36
Original line number Diff line number Diff line
name: Pull-Request-CI
name: CI

on:
  pull_request:
    branches:
      - dev
      - main
  push:
    branches:
      - dev
      - main

env:
  isis-root: /usr/share/miniconda/envs/isis/
@@ -20,29 +20,11 @@ jobs:
      fail-fast: false
      matrix:
       os: [ubuntu-latest]
       python-version: ["3.7"]
    services:
      postgres:
        image: postgis/postgis:10-2.5
        env:
          POSTGRES_PASSWORD: NotTheDefault
          POSTGRES_DB: postgres
        ports:
          - 35432:5432
        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5   
       python-version: ["3.7", "3.8", "3.9", "3.10"]
    defaults:
      run:
        shell: bash -l {0}
    steps:
      - name: ISIS Conda Env
        uses: conda-incubator/setup-miniconda@v2
        with:
          python-version: 3.6
          activate-environment: isis
          channels: usgs-astrogeology, conda-forge
      - name: Install ISIS
        run: |
          conda install -q -y -c usgs-astrogeology isis
      - name: Setup test-resources/
        run: |
          mkdir $GITHUB_WORKSPACE/test-resources/
@@ -51,36 +33,36 @@ jobs:
        run: |
          curl "${{ env.hirise-pds-url }}/PSP_010502_2090_RED5_0.IMG" -o $GITHUB_WORKSPACE/test-resources/PSP_010502_2090_RED5_0.img
          curl "${{ env.hirise-pds-url }}/PSP_010502_2090_RED5_1.IMG" -o $GITHUB_WORKSPACE/test-resources/PSP_010502_2090_RED5_1.img
      - name: Exit isis env
        run: conda deactivate
      - name: Checkout Code
        uses: actions/checkout@v2
        uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 
      - name: Cache conda
        uses: actions/cache@v2
        env:
          # Increase this value to reset cache if etc/example-environment.yml has not changed
          CACHE_NUMBER: 0
          CACHE_NUMBER: 1
        with:
          path: ~/conda_pkgs_dir
          key:
            ${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{
            hashFiles('environment.yml') }}
      - name: Setup ENV
        uses: conda-incubator/setup-miniconda@v2
        uses: conda-incubator/setup-miniconda@3b0f2504dd76ef23b6d31f291f4913fb60ab5ff3
        with:
          miniconda-version: "latest"
          mamba-version: "*"
          miniforge-version: latest  
          use-mamba: true  
          channels: conda-forge  
          channel-priority: strict
          activate-environment: autocnet 
          environment-file: environment.yml  
          use-only-tar-bz2: true
          auto-activate-base: false  
          auto-update-conda: false  
          python-version: ${{ matrix.python-version }}
      - name: config ISIS vars
        run: |
          conda env config vars set ISISROOT=${{ env.isis-root }}
          conda env config vars set ISISDATA=$GITHUB_WORKSPACE/${{ env.isis-data }}
      - name: Install dev requirements
        run: |
          pip install -r test_requirements.txt
      - name: Check build environment
        run: |
          conda list
@@ -89,7 +71,7 @@ jobs:
          python setup.py install
      - name: Test Python Package
        run: |
           pytest --cov-report=xml
           pytest -n 4
      - name: Upload Coverage
        uses: codecov/codecov-action@v1
        with:
+4 −0
Original line number Diff line number Diff line
# ISIS / Testing
print.prt
tests/artifacts/*

# OS X
.DS_Store
._.DS_Store
+1 −18
Original line number Diff line number Diff line
image: code.usgs.gov:5001/astrogeology/autocnet/mambaforge

services:
    - name: code.usgs.gov:5001/astrogeology/autocnet/postgis
      alias: postgres
      entrypoint: ["docker-entrypoint.sh"]
      command: ["postgres"]

variables:
    POSTGRES_HOST: postgres
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: NotTheDefault
    POSTGRES_HOST_AUTH_METHOD: trust

before_script:
  - mamba env create -f environment.yml -n autocnet_env
  - conda init
@@ -27,15 +15,10 @@ unit-test:
  stage: test
  script:
    - mamba install --file test_requirements.txt
    - pip install mock-alchemy
    - wget "https://asc-isisdata.s3.us-west-2.amazonaws.com/autocnet_test_data/B08_012650_1780_XN_02S046W.l1.cal.destriped.crop.cub" -P tests/test_subpixel_match/
    - wget "https://asc-isisdata.s3.us-west-2.amazonaws.com/autocnet_test_data/D16_033458_1785_XN_01S046W.l1.cal.destriped.crop.cub" -P tests/test_subpixel_match/
    - wget "https://asc-isisdata.s3.us-west-2.amazonaws.com/autocnet_test_data/J04_046447_1777_XI_02S046W.l1.cal.destriped.crop.cub" -P tests/test_subpixel_match/
    - psql -h $POSTGRES_HOST -c 'create database template_postgis;' -U postgres ;
    - psql template_postgis -h $POSTGRES_HOST -U postgres -c 'create extension postgis';
    - psql template_postgis -U $POSTGRES_USER -h $POSTGRES_HOST -c 'create extension postgis_topology';
    - psql -d template_postgis -U $POSTGRES_USER -h $POSTGRES_HOST -c 'GRANT ALL ON geometry_columns TO PUBLIC;';
    - psql -d template_postgis -U $POSTGRES_USER -h $POSTGRES_HOST -c 'GRANT ALL ON geography_columns TO PUBLIC;';
    - psql -d template_postgis -U $POSTGRES_USER -h $POSTGRES_HOST -c 'GRANT ALL ON spatial_ref_sys TO PUBLIC;';
    - pytest .

pages:

ARCHITECTURE.md

0 → 100644
+39 −0
Original line number Diff line number Diff line
# Global Map

## Major Data Structures

### Network vs. Non-Network
The major data structures: graph, node, and edge have network versions (e.g., `NetworkNode` and `NetworkCandidateGraph`). These inherit all functionality from the non-network versions and add I/O operations that use network resources. Network resources can be processing queues, databases, and remote job submission through job orchestrators like slurm.

It is preferable to implement analytic capabilities in the non-network versions of the data structures. The goal is that the network versions simply retrieve data and push results to non-memory locations.

### Graphs: CandidateGraph & NetworkCandidateGraph

### Nodes: Node & NetworkNode

### Edges: Edge & NetworkEdge

### ControlNetworks
ISIS compatible control networks are managed as pandas DataFrames within the library. When the control network is persisted to a database (usually via a `NetworkCandidateGraph`), pandas `read_sql` calls are used with the `db_to_df_sql_string` to select a dataframe representation. This data structure can be analyzed, manipulated, and ultimately either written back to the database or to a file (via the `to_isis` method in `plio`).

## Testing
The library is tested via unit and integration tests using pytest. Endeavour to test as much of the software as is feasible, understanding that the code base has been developed to create a product and not unto itself.

Tests are parameritized using the `pytest.ini` file in the repository root. This way, developers are all invoking the tests in a standard way and best practices can be propagated under version control.

### Unit Tests
The majority of the library is tested via unit tests.

### Integration Tests
The integration tests are longer running. Strive to keep integration tests less than 120s each. We assume that tests can run in parallel with at least 4 CPUs (GitHub free runners, as of March 2024 have 4 CPUs on ubuntu). Integration tests do not make use of any external resources. They should make liberal use of mocked REDIS and SQLAlchemy session objects.

## Sensor Models
Sensor models are abstracted through a common interface. That interface is stored in 

# Gotchas
Here are a number of different gotchas in the library that deal with stuff.

## Latitude: Ocentric vs. Ographic
There are two different ways to define latitude. See [the proj docs](https://proj.org/en/9.4/operations/conversions/geoc.html) for a in-depth description of ocentric vs. ographic latitudes. This library is impacted when working with other libraries that default to a particular latitude definition. As of the 1.2.0 release, all latitudes inside this library are ographic, conforming to the PROJ standard. The CSM (usgscsm) working in XYZ/BCBF coordinates, so anytime one converts those to lat/lons, they use PROJ and have ographic latitudes. The ISIS library defaults to ocentric latitudes. So, passing a lat/lon pair into a program like `campt` assumes that that latitude is ocentric. So, the autocnet.camera.sensor_models `ISISSensor` object takes can of transformations to ocentric when a lat/lon pair is passed. That same object defaults to returning ographic latitudes when one queries for ground coordinates. When writing a control network, one writes the XYZ/BCBF coordinates. Therefore, no conversion has to take place. 

When adding any functionality to the library dealing with sensor models and ISIS either (1) use the `ISISSensor` interface or (2) be very careful to convert to/from ographic/ocentric latitudes as appropriate.
 No newline at end of file
+4 −0
Original line number Diff line number Diff line
@@ -35,6 +35,10 @@ release.
## [Unreleased]
### Added
- Ability to choose whether to compute overlaps for a network candidate graph
- Integration tests for end-to-end of an equatorial CTX pair and a mid-latitude CTX trio.

### Changed
- CI on the library now uses a mocked sqlalchemy connection. All tests can now run locally without the need for a supplemental postgres container. This changed removed some non-optimal tests that were testing datbase triggers and database instantiation handled by SQLAlchemy.

### Fixed
- Errors when importing sensor model in `overlap.py`
Loading