Unverified Commit d83087be authored by Ross Beyer's avatar Ross Beyer Committed by GitHub
Browse files

Convert from pysis to kalasiris and rewrite of spatial/isis.py (#573) 

* test(change_detection): Added tests for the functions which call out to ISIS.

* refactor(change_detection): Converted pysis to kalasiris.

* refactor(utils/hirise): Converted pysis to kalasiris.

* refactor(spatial): Converted pysis to kalasiris.

* refactor(acd_cd): Converted from pysis to kalasiris.

* refactor(conda/meta.yaml): Converted from pysis to kalasiris.

* refactor(environment.yml): Converted from pysis to kalasiris.

* refactor(notebooks): Converted from pysis to kalasiris.

* refactor(spatial/isis.py): Converted from pysis to kalasiris.

* refactor(matcher/subpixel.py): Converted from pysis to kalasiris.

* docs(AUTHORS): Added my name.

* docs(README): Added info about needing to separately install ISIS and then set ISIS environment variables in the autocnet environment.

* refactor(utils/hirise.py): Removed unused imports, added some opinionated comments.

* test(utils/hirise.py): Created tests.

* build(Makefile): Added for my own sanity, but also to help download some example products for testing.

* refactor(spatial/isis.py): Minor changes to deal with new pvl, and added tests.

* feat(spatial/isis.py): Rebuilt the three functions, normalized their input and output, and added tests.

Modules in autocnet that used these functions were altered to adapt
to
the new pattern, but they did not have tests before this change, so
it
is unknown if my changes have any other consequences.

* fix(Makefile): Don't auto-run test-resources with "make test" make more optional.

* ci(continuous_integration.yml): I love trying to debug CI via commit.

* ci(continuous_integration.yml): Fiddly YAML syntax.  You'd think I'd be used to counting spaces by now.

* ci(continuous_integration.yml): Need to count in whole document.

* ci(continuous_integration.yml): Need both HiRISE channels for tests, otherwise they skip, but at least that's working.

* fix(subpixel.py): Original code had reversed the output of the functions, but I didn't notice that, so I inadvertently un-reversed it and introduced an error.  Fixed.
parent 136610be

.github/workflows/continuous_integration.yml

+28 −0
Original line number Diff line number Diff line
@@ -8,6 +8,11 @@ on: 
    branches:

      - dev



env:

  isis-root: /usr/share/miniconda/envs/isis/

  isis-data: test-resources/ISISDATA/

  hirise-pds-url: https://hirise-pds.lpl.arizona.edu/PDS/EDR/PSP/ORB_010500_010599/PSP_010502_2090



jobs:

  Build-and-Test:

    runs-on: ${{ matrix.os }}
@@ -29,6 +34,25 @@ jobs: 
      run:

        shell: bash -l {0}

    steps:

      - name: ISIS Conda Env

        uses: conda-incubator/setup-miniconda@v2

        with:

          python-version: 3.6

          activate-environment: isis

          channels: usgs-astrogeology, conda-forge

      - name: Install ISIS

        run: |

          conda install -q -y -c usgs-astrogeology isis

      - name: Setup test-resources/

        run: |

          mkdir $GITHUB_WORKSPACE/test-resources/

          mkdir $GITHUB_WORKSPACE/${{ env.isis-data }}

      - name: Download IMG

        run: |

          curl "${{ env.hirise-pds-url }}/PSP_010502_2090_RED5_0.IMG" -o $GITHUB_WORKSPACE/test-resources/PSP_010502_2090_RED5_0.img

          curl "${{ env.hirise-pds-url }}/PSP_010502_2090_RED5_1.IMG" -o $GITHUB_WORKSPACE/test-resources/PSP_010502_2090_RED5_1.img

      - name: Exit isis env

        run: conda deactivate

      - name: Checkout Code

        uses: actions/checkout@v2

      - name: Cache conda
@@ -53,6 +77,10 @@ jobs: 
          environment-file: environment.yml

          use-only-tar-bz2: true

          python-version: ${{ matrix.python-version }}

      - name: config ISIS vars

        run: |

          conda env config vars set ISISROOT=${{ env.isis-root }}

          conda env config vars set ISISDATA=$GITHUB_WORKSPACE/${{ env.isis-data }}

      - name: Check build environment

        run: |

          conda list

AUTHORS.rst

+2 −1
Original line number Diff line number Diff line
@@ -15,4 +15,5 @@ Development Team 
Contributors

------------



None yet. Why not be the first?
 
`Ross Beyer <https://github.com/rbeyer>`_

Makefile

0 → 100644
+99 −0
Original line number Diff line number Diff line 
.PHONY: clean clean-test clean-pyc clean-build docs help

.DEFAULT_GOAL := help



define BROWSER_PYSCRIPT

import os, webbrowser, sys



from urllib.request import pathname2url



webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1])))

endef

export BROWSER_PYSCRIPT



define PRINT_HELP_PYSCRIPT

import re, sys



for line in sys.stdin:

	match = re.match(r'^([a-zA-Z_-]+):.*?## (.*)$$', line)

	if match:

		target, help = match.groups()

		print("%-20s %s" % (target, help))

endef

export PRINT_HELP_PYSCRIPT



define DOWNLOAD_PYSCRIPT

import urllib.request, sys



urllib.request.urlretrieve(sys.argv[1],sys.argv[2])

endef

export DOWNLOAD_PYSCRIPT



BROWSER := python -c "$$BROWSER_PYSCRIPT"

DOWNLOAD := python -c "$$DOWNLOAD_PYSCRIPT"



help:

	@python -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)



clean: clean-build clean-pyc clean-test ## remove all build, test, coverage and Python artifacts



clean-build: ## remove build artifacts

	rm -fr build/

	rm -fr dist/

	rm -fr .eggs/

	find . -name '*.egg-info' -exec rm -fr {} +

	find . -name '*.egg' -exec rm -f {} +



clean-pyc: ## remove Python file artifacts

	find . -name '*.pyc' -exec rm -f {} +

	find . -name '*.pyo' -exec rm -f {} +

	find . -name '*~' -exec rm -f {} +

	find . -name '__pycache__' -exec rm -fr {} +



clean-test: ## remove test and coverage artifacts

	rm -fr .tox/

	rm -f .coverage

	rm -fr htmlcov/

	rm -fr .pytest_cache

	rm -fr test-resources



lint: ## check style with flake8

	flake8 --max-complexity 10 --ignore E203,E501,W503 autocnet tests



test: ## run tests quickly with the default Python

	pytest autocnet



test-resources: ## Download what we need for testing

	mkdir test-resources

	$(DOWNLOAD) https://hirise-pds.lpl.arizona.edu/PDS/EDR/PSP/ORB_010500_010599/PSP_010502_2090/PSP_010502_2090_RED5_0.IMG test-resources/PSP_010502_2090_RED5_0.img

	$(DOWNLOAD) https://hirise-pds.lpl.arizona.edu/PDS/EDR/PSP/ORB_010500_010599/PSP_010502_2090/PSP_010502_2090_RED5_1.IMG test-resources/PSP_010502_2090_RED5_1.img





test-all: ## run tests on every Python version with tox

	tox



coverage: test  ## check code coverage quickly with the default Python

	coverage report -m

	coverage html

	$(BROWSER) htmlcov/index.html



docs: ## generate Sphinx HTML documentation, including API docs

	rm -f docs/autocnet.rst

	rm -f docs/modules.rst

	sphinx-apidoc -o docs/ autocnet

	$(MAKE) -C docs clean

	$(MAKE) -C docs html

	$(BROWSER) docs/_build/html/index.html



servedocs: docs ## compile the docs watching for changes

	watchmedo shell-command -p '*.rst' -c '$(MAKE) -C docs html' -R -D .



release: dist ## package and upload a release

	twine upload dist/*



dist: clean ## builds source and wheel package

	python setup.py sdist

	python setup.py bdist_wheel

	ls -l dist



install: clean ## install the package to the active Python's site-packages

	python setup.py install

README.md

+28 −4
Original line number Diff line number Diff line
@@ -17,10 +17,34 @@ Is available at: https://usgs-astrogeology.github.io/autocnet/ 
We suggest using Anaconda Python to install Autocnet within a virtual environment.  These steps will walk you through the process.



1. [Download](https://www.continuum.io/downloads) and install the Python 3.x Miniconda installer.  Respond ``Yes`` when prompted to add conda to your BASH profile.

2. Install `mamba`. [`mamba`](https://github.com/mamba-org/mamba) is a fast cross platform package manager that has, in our experience, improved environment solving. The AutoCNet environment is complex and `mamba` is necessary to get a solve. To install `conda install -c conda-forge mamba`.

3. Install the autocnet environment using the supplied environment.yml file: `mamba env create -n autocnet -f environment.yml` 

4. Activate your environment: `conda activate autocnet`

5. If you are doing to develop autocnet or would like to use the bleeding edge version: `python setup.py develop`. Otherwise, `conda install -c usgs-astrogeology autocnet`

2. Install ISIS.   Follow the 

   [instructions to install ISIS](https://github.com/USGS-Astrogeology/ISIS3#installation) in its own conda 

   environment. With that ISIS environment activated, determine the values for the ISIS environment 

   variables, with a command like this (may vary by your shell):



   `printenv | grep ISIS`



   Copy down the values of ISISROOT and ISISDATA.  Exit the ISIS environment. 

   

3. Install `mamba`. [`mamba`](https://github.com/mamba-org/mamba) is a fast cross platform package manager that has, in our experience, improved environment solving. The AutoCNet environment is complex and `mamba` is necessary to get a solve. To install `conda install -c conda-forge mamba`.

4. Install the autocnet environment using the supplied environment.yml file: `mamba env create -n autocnet -f environment.yml` 

5. Activate your environment: `conda activate autocnet`

6. Ensure ISISROOT and ISISDATA are set in the *autocnet* environment.  One way is to

   ```

   conda env config vars set ISISROOT=value-you-wrote-down-from-step-2

   conda env config vars set ISISDATA=value-you-wrote-down-from-step-2

   conda deactivate

   conda activate autocnet

   ```

   If you use Jupyter notebooks, you may need to do the following in a cell before importing 

   *autocnet* modules:

   ```

   import os

   os.environ["ISISROOT"] = "path-you-wrote-down-in-step-2"

   os.environ["ISISDATA"] = "path-you-wrote-down-in-step-2"

   ```

   

7. If you are going to develop autocnet or would like to use the bleeding edge version: `python setup.py develop`. Otherwise, `conda install -c usgs-astrogeology autocnet`



## How to run the test suite locally

autocnet/cg/change_detection.py

+3 −3
Original line number Diff line number Diff line
@@ -26,7 +26,7 @@ import richdem as rd 
import pandas as pd

import geopandas as gpd



import pysis

from kalasiris import specialpixels



from autocnet.utils.utils import bytescale

from autocnet.matcher.cpu_extractor import extract_features
@@ -57,7 +57,7 @@ def image_diff(arr1, arr2): 
    arr1[arr1 == 0] = np.nan

    arr2[arr2 == 0] = np.nan



    isis_null = pysis.specialpixels.SPECIAL_PIXELS['Real']['Null']

    isis_null = specialpixels.Real.Null

    arr1[arr1 == isis_null] = np.nan

    arr2[arr2 == isis_null] = np.nan
@@ -92,7 +92,7 @@ def image_ratio(arr1, arr2): 
    arr1[arr1 == 0] = np.nan

    arr2[arr2 == 0] = np.nan



    isis_null = pysis.specialpixels.SPECIAL_PIXELS['Real']['Null']

    isis_null = specialpixels.Real.Null

    arr1[arr1 == isis_null] = np.nan

    arr2[arr2 == isis_null] = np.nan