Unverified Commit fe91cb73 authored by jlaura's avatar jlaura Committed by GitHub
Browse files

Adds some examples using the refactored API (#442)

* Adds some examples using the refactored API

* Removes bad sample config and fixes README typo

* Updates for comments

* Removes method entirely

* Updates for comments

* Updates for comments to improve docs

* Updates for better grammar

* typos
parent 675b4eae
Loading
Loading
Loading
Loading
+201 −0
Original line number Diff line number Diff line
@@ -27,3 +27,204 @@ We suggest using Anaconda Python to install Autocnet within a virtual environmen
2. Get the Postgresql with Postgis container and run it `docker run --name testdb -e POSTGRES_PASSOWRD='NotTheDefault' -e POSTGRES_USER='postgres' -p 5432:5432 -d mdillon/postgis`
3. create database template_postgis: `docker exec testdb psql -c 'create database template_postgis;' -U postgres`
4. Run the test suite: `pytest autocnet`

## Simple Network Examples:


### Setup a project
This first example imports the NetworkCandidateGraph object, which is used to orchestrate jobs, manage
a database session, and generally work with the images, points, and measures in
a control network.

```
from autocnet.graph.network import NetworkCandidateGraph

# Make an empty NCG
ncg = NetworkCandidateGraph()
# Load the configuration file
ncg.config_from_file('config/demo.yml')

# Populate the nodes/edges from the DB
ncg.from_database()
```

Line by line, this code first imports the network candidate graph, a collection
of nodes and edges that represents a potential control network.
`from autocnet.graph.network import NetworkCandidateGraph`.

Next, a network candidate graph is instantiated. `ncg =
NetworkCandidateGraph()`.

The network candidate graph (or NCG) is assocaited with a collection of
PostGreSQL database tables. We have to initiate the database connection via a
configuration file. An example configuration file is
provided in the config directory. `ncg.config_from_file('config/demo.yml')`

Once configured, the images need to be loaded and the graph of potential
overlapping images generated. We do this with `ncg.from_database()`.

At this point, you have a fully functioning autocnet project using an NCG. The
above snippet assumes that a prepopulated database already exists. Keep reading
to see how AutoCNet supports importing images from an existing image data
store.

### Import images from a data store containing image footprints
Autocnet does not assume where your image footprints are coming from for
initial setup. We do assume that you have a prepopulated database of image
footprints with a `geom` column. Otherwise, you could use any software to
create image footprints and populate the footprint database.

To initialize a project from a data store of image footprints we can do the
following:

```
# These lines are pulled from the example above
from autocnet.graph.network import NetworkCandidateGraph

ncg = NetworkCandidateGraph()
ncg.config_from_file('config/demo.yml')

# Create the connection 
source_db_config = {'username':'jay',
        'password':'abcde',
        'host':'localhost',
        'pgbouncer_port':5432,
        'name':'someothertable'}

# Subset the data store using a spatial query.
geom = 'LINESTRING(145 10, 145 10.25, 145.25 10.25, 145.25 10, 145 10)'
srid = 949900
outpath = '/scratch/some/path/for/data'
query = f"SELECT * FROM ctx WHERE ST_INTERSECTS(geom, ST_Polygon(ST_GeomFromText('{geom}'), {srid})) = TRUE"
ncg.add_from_remote_database(source_db_config, outpath, query_string=query)
```

Here we create an NCG as above. Then we define a new database connection with
the name of the database from which data will be extracted. `source_db_config =
{'username':'jay', 'password':'abcde', 'host':'localhost',
'pgbouncer_port':5432, 'name':'someothertable'}.

In this example, we want to use a spatial query to subset the data. We could
also use an attribute query or some combination. The only restriction is that
the quert string be valid SQL. `geom = 'LINESTRING(145 10, 145 10.25, 145.25
10.25, 145.25 10, 145 10)'`

The PostGIS query requires a valid SRID for the input geometry, so we
explicitly define that here. This is the SRID that the footprints are being
stored in inside of the data store. `srid = 949900` The srid here is a custom
srid that has been added to the data store spatial reference table; the id can
be any arbitrary number as long as it exists in the spatial reference table.

The `add_from_remote_database` call copies the image files in the source
database into a new directory. Here we define that directory. `outpath =
'/scratch/some/path/for/data'`. 

The query string is then constructed: `query = f"SELECT * FROM ctx WHERE
ST_INTERSECTS(geom, ST_Polygon(ST_GeomFromText('{geom}'), {srid})) = TRUE"` 

Finally, our database associated with the NCG is populated and the image data
are copied. `ncg.add_from_remote_database(source_db_config, outpath,
query_string=query)`

### Creating a NCG Using a Filelist
It is also possible to create a NCG and instantiate an associated database from a
list of ISIS cube files that have had footprints created (using *footprintinit*).

```
from autocnet.graph.network import NetworkCandidateGraph

ncg = NetworkCandidateGraph.from_filelist(myimages.lis)
```

This method can take a bit of time to run if the filelist is large as the data
are loaded into the database sequentially and then a spatial overlay operation is performed
to determine how individual images overlap with one another (using the footprints
generated from the a priori sensor pointing.)

### Operations on the NCG: Database Rows
After we have an NCG, we want to perform operations on the graph or on database
rows associated with the graph (e.g., the Points, Measures, or Image Overlaps).
We use a functional approach where an arbitray function can be applied to an
iterable associated with the graph. Here is a concrete example to help
illustrate what this looks like in practice.

```
from autocnet.graph.network import NetworkCandidateGraph

ncg = NetworkCandidateGraph()
ncg.config_from_file('/home/jlaura/autocnet_projects/demo.yml')
ncg.from_database()


# Define a function to govern the distribution of points in the North/South direction
def ns(x):
    from math import ceil
    return ceil(round(x,1)*8)

# Define a function to govern the distribution of points in the East/West direction
def ew(x):
    from math import ceil
    return ceil(round(x,1)*1)

# Pack a set of kwargs into a keyword that the called function is expecting
distribute_points_kwargs = {'nspts_func':ns, 'ewpts_func':ew}

# Apply a function on an iterable
njobs = ncg.apply('spatial.overlap.place_points_in_overlap', 
                  on='overlaps', 
                  cam_type='isis',
                  distribute_points_kwargs=distribute_points_kwargs)
```

Most of the above is either familiar boiler plate or a pair of helper functions
that we want to pass in. The interesting stuff is happening in:

```
njobs = ncg.apply('spatial.overlap.place_points_in_overlap',
                  on='overlaps',
                  cam_type='isis',
                  distribute_points_kwargs=distribute_points_kwargs)
```

Here, we are applying the `spatial.overlap.place_points_in_overlap` function on
an iterable (`overlaps`) with three keyword arguments (that the function is
expecting). The syntax for the function is module.submodule.function_name'.
Where the submodule can be repeated, e.g.,
module.submodule.subsubmodule.function_name.

It is possible to use a similar block to, for example, apply some subpixel
registration algorithm: 

`njobs = ncg.apply('matcher.subpixel.subpixel_register_point', on='points')`

or to apply a second pass subpixel alignment on only measures meeting some
criteria:

```
filters = {'ignore' : True}  # A database filter in the form column name : equality
njobs = ncg.apply('matcher.subpixel.subpixel_register_measure',
                  on='measures',
                  filters=filters)
```

### Operations on the NCG: Nodes and Edges
Just like the above example, it is possible to apply arbitrary functions to
nodes and edges in a NetworkCandidateGraph.

```
ncg = NetworkCandidateGraph()
ncg.config_from_file('/home/jlaura/autocnet_projects/demo.yml')
ncg.from_database()

njobs = ncg.apply('network_to_matches', on='edges')
```

After the standard boilerplate, the `network_to_matches` function is applied to
every edge in the graph. This function takes the points and measures from the
database and expands them so that every edge now has the pairwise
(measure-to-measure) information that is frequently quite useful when using
computer vision techniques. Note that the function to be called is not longer
being specificed with the import path (e.g.,
spatial.overlap.place_points_in_overla-). Note that the function to be called is no longer
being specificed with the import path (e.g., spatial.overlap.place_points_in_overlaps) because 
only Edge or NetworkEdge methods can be called on the autocnet Edge or NetworkEdge objects.
+24 −56
Original line number Diff line number Diff line
@@ -5,14 +5,14 @@ cluster:
    # Which processing queue should be used?
    queue: 'shortall'
    # Location to put <jobid.log> files for cluster jobs
    cluster_log_dir: '/scratch/jlaura/elysium/logs'
    cluster_log_dir: '/logs'
    # What cluster submission tool should be used
    cluster_submission: 'slurm'  # or `pbs`
    # What scratch or temporary area should be used for temporary file creation
    tmp_scratch_dir: '/scratch/jlaura'
    tmp_scratch_dir: '/scratch'

    # The amount of RAM (in MB) to request for jobs
    extractor_memory: 1024
    extractor_memory: 8192
    processing_memory: 1024

### Database Configuration ###
@@ -20,20 +20,25 @@ database:
    # Name of the database management system. For example, postgresql or sqlite
    type: 'postgresql'
    # Username used to log in to the database
    username: 'sample'
    username: 'jay'
    # Password used to log in to the database
    password: 'sample'
    password: 'abcde'
    # Host machine name (or localhost) where the database is running
    host: 'autocnet.wr.usgs.gov'
    host: '127.0.0.1'
    # Port that the database is exposed on
    port: 5432
    port: 8085
    # Port where a proxy that manages database connections is running
    pgbouncer_port: 6543
    pgbouncer_port: 8083
    # The name of the database to connect to.  Tables will be created inside this DB.
    name: 'jelysiumtest' # This needs to be all lowercase for PostGreSQL!
    name: 'deleteme' # This needs to be all lowercase for PostGreSQL!
    # The number of seconds to wait while attemping to connect to the DB.
    timeout: 500

env: 
  conda: 'autocnet'  # The name of a conda environment to initialize for cluster jobs 
  ISISROOT: '/path_to/isisx.y.z'  # PATH to the ISISROOT directory
  ISISDATA: '/usgs/cpkgs/isis3/data'  # PATH to the ISISDATA directory
    
### Pfeffernusse / CSM Configuration ###
pfeffernusse:
    # URL to the service that serves CSM compliant ISDs
@@ -43,25 +48,23 @@ pfeffernusse:
redis:
    # Name of the redis queue. Ensure it is unique to avoid multiple uses
    # of the same redis queue.
    basename: 'elysium'
    basename: 'demo'
    # Hostname (or localhost) where the redis server is running
    host: 'autocnet.wr.usgs.gov'
    host: 'localhost'
    # Port that redis is accepting connections on
    port: '6379'
    port: '8084'
    # The name of the queue that successful job messages are pushed. This
    # name should be unique to avoid collision with other users.
    completed_queue: 'jely:done'
    completed_queue: 'demo:done'
    # The name of the queue used for jobs that need to be started. This
    # name should be unique to avoid collision with other users.
    processing_queue: 'jely:proc'
    processing_queue: 'demo:proc'
    # The name of the queue that currently processing jobs are pushed to. This
    # name should be unique to avoid collision with other users.
    working_queue: 'jely:working'
    working_queue: 'demo:working'

### Spatial Reference Setup ###
spatial:
    # The name of the target body that will be output in ISIS control networks.
    target: 'MARS'
    # The following two values need to be added to the database that is storing
    # the spatial information.
    #
@@ -77,47 +80,12 @@ spatial:
    # for planetary bodies.
    proj4_str: '+proj:longlat +a:3396190 +b:3376200 +no_defs'
    # DEM used to generate height values for projection
    dem: '/scratch/jlaura/mars/molaMarsPlanetaryRadius.cub'
    dem: '$ISISDATA/dems/molaMarsPlanetaryRadius.cub'
    # Target for ISIS
    target: 'MARS'

### Working Directories ###
directories:
    # The directory where VRTs should be created
    vrt_dir: '/sratch/jlaura/elysium/vrt'
    vrt_dir: '/sratch/vrts'
### Algorithms ###
# This section contains definitions for algorithms available
# to AutoCNet. When populated, the values in these blocks
# are parsed on job submission and used as the args/kwargs.
# When an algorithm has multiple entries, e.g., 3 for `ring_match`,
# the algorithm will be run up to 3 times. The first time, it is
# assumed that the strictest parametrization is used. If this
# fails, the next parametrization is used and so forth until either
# the algorithm returns successfully or all parameter combinations
# have been applied. This capability is experimental!
algorithms:
     ring_match:
         - target_points: 25
           tolerance: 0.01
         - target_points: 20
           tolerance: 0.01
         - target_points: 25
           tolerance: 0.02
     compute_fundamental_matrix:
         - tolerance: 0.3
           reproj_threshold: 10
           initial_x_size: 500
           initial_y_size: 500
           corr_x_size: 40
           corr_y_size: 40
         - tolerance: 0.3
           reproj_threshold: 15
           initial_x_size: 500
           initial_y_size: 500
           corr_x_size: 40
           corr_y_size: 40
         - tolerance: 0.25
           reproj_threshold: 20
           initial_x_size: 500
           initial_y_size: 500
           corr_x_size: 40
           corr_y_size: 40