file for workshop notebooks; start of a few notebooks (737f44a6) · Commits · aflab / astrogeology / Autocnet

docs/users/workshops/FY21_workshop/autocnet_intro.ipynb

0 → 100644

+53 −0

Original line number	Diff line number	Diff line
		%% Cell type:code id: tags:

		``` python
		The components of the NetworkCandidateGraph

		The database structures

		Apply & cluster processing

		How to generate a control network purely in autocnet (Use the Phobos)

		Sub-pixel registration

		Cross-instrument matching

		How to ingest an ISIS control network

		Overlap analysis

		Outlier detection

		Custom cost functions
		```

		%% Cell type:markdown id: tags:

		# AutoCNet Intro
		AutoCNet is a suite of functions that parallelize network generation and analyze the health of networks. Networks within AutoCNet are represented with an [undirected graph](https://en.wikipedia.org/wiki/Graph_(discrete_mathematics)), called a NetworkCandidateGraph.

		It leverages a database for presistent storage of the images, points, and measures of a network. The use of this database allows for quick access of each elements geometries and how those geometries relate with one another.

		%% Cell type:markdown id: tags:

		# Components of The NetworkCandidateGraph

		%% Cell type:markdown id: tags:

		# Database Table Structures
		This includes:
		- camera
		- costs
		- edges
		- images
		- keypoints
		- matches
		- measures
		- overlay
		- points

		%% Cell type:code id: tags:

		``` python
		```

docs/users/workshops/FY21_workshop/jupyterhub_ipython.ipynb

0 → 100644

+223 −0

Original line number	Diff line number	Diff line
		%% Cell type:markdown id: tags:

		# Anaconda Environments
		Anaconda environments are a collection of python packages that are installed into an isolated environment and can be selectively accessed through activation of that environment. They are particularly helpful because various versions of a program or various combinations of programs in isolated environments do not effect those in another environment.

		For example, ASC internally creates anaconda environments for each new release, and release candidate of the ISIS software. If a user would like to access any particular version of ISIS, they would type `conda activate isisx.y.z`, if later they wanted to access a different version they could `conda deactivate & conda activate isisu.v.w` without worrying about cross containination of environment variables.

		%% Cell type:markdown id: tags:

		### Where are they located?
		There are internal environments and personal environments. Environments available to ASC internal users include ISIS release builds and select apps (autocnet, jupyter, plio, qgis, etc). You can view which environments area available to your user profile, along with the location of the installed apps using:

		%% Cell type:code id: tags:

		``` python
		! conda env list \| head
		```

		%% Output

		# conda environments:
		#
		autocnet_Jaffine /home/ladoramkershner/miniconda3/envs/autocnet_Jaffine
		autocnet_local /home/ladoramkershner/miniconda3/envs/autocnet_local
		csm_local /home/ladoramkershner/miniconda3/envs/csm_local
		isis4c_prints /home/ladoramkershner/miniconda3/envs/isis4c_prints
		isis_custom_06.05.20 /home/ladoramkershner/miniconda3/envs/isis_custom_06.05.20
		qgis /home/ladoramkershner/miniconda3/envs/qgis
		swig_csm /home/ladoramkershner/miniconda3/envs/swig_csm
		autocnet /usgs/apps/anaconda/envs/autocnet

		%% Cell type:markdown id: tags:

		The asterisk ('\*') indicates the currently active anaconda environment.

		%% Cell type:markdown id: tags:

		### Creating a New Environment
		__I am thinking of having the user create a new autocnet_uid environment, then uploading it to a python kernel__

		An anaconda environment can be created using the conda env create command

		```
		conda env create -n [name] -f [path/to/autocnet/environment.yml]
		```

		The -n argument is name that the new conda environment will be assigned. The -f argument points to a file that dictates which conda packages will be installed at the creation of the new environment. In this case, there is an environment.yml file located in the root folder of autocnet repository.

		Write this function below filling in the two arguments.

		%% Cell type:code id: tags:

		``` python
		! [insert function]
		```

		%% Cell type:markdown id: tags:

		Show the `conda list` command (see what is avaiable in the conda environment)

		%% Cell type:code id: tags:

		``` python
		```

		%% Cell type:markdown id: tags:

		### What Packages are Available in an Environment?
		A user can either produce a general list of all packages in a conda environment

		%% Cell type:code id: tags:

		``` python
		!conda list \| head
		```

		%% Output

		# packages in environment at /usgs/apps/anaconda/envs/jupyter:
		#
		asn1crypto 0.22.0 py35_0 conda-forge
		astropy 2.0.1 np111py35_1 conda-forge
		batchspawner 0.0.1.dev0 <pip>
		bkcharts 0.2 py35_0 conda-forge
		bleach 1.5.0 py35_0 conda-forge
		bokeh 0.12.6 py35_0 conda-forge
		boto 2.47.0 py35_0
		ca-certificates 2017.7.27.1 0 conda-forge

		%% Cell type:markdown id: tags:

		or check if a certain package is in the environment

		%% Cell type:code id: tags:

		``` python
		!conda list jupyterhub
		```

		%% Output

		# packages in environment at /usgs/apps/anaconda/envs/jupyter:
		#
		jupyterhub 0.7.2 py35_0 conda-forge

		%% Cell type:markdown id: tags:

		# Notebook kernels
		Jupyter kernels are conda environments that can be accessed within a jupyter notebook
		- Click on 'Kernel' tab in top menu
		- Go to change 'Change Kernel' and look through the options

		### Where are they located?
		These jupyter kernels are located in user specific directories:

		%% Cell type:code id: tags:

		``` python
		uid="ladoramkershner" # insert user id here
		!ls /home/$uid/.local/share/jupyter/kernels/
		```

		%% Output

		adtm_jl autocnet_jaffine cowboy_borpbop kernel.json
		autocnet autocnet_local csm_local rotations

		%% Cell type:markdown id: tags:

		### How to Upload New Kernel?
		```
		python -m ipykernel install --user --name myenv --display-name "Python (myenv)"
		```

		--name should be one of the environment listed by 'conda env list'

		--display-name is how you want it to appear in the jupyter notebook

		%% Cell type:code id: tags:

		``` python
		python -m ipykernel install --user --name myenv --display-name "Python (myenv)"
		```

		%% Cell type:markdown id: tags:

		### Check Successful Upload

		%% Cell type:code id: tags:

		``` python
		uid="ladoramkershner" # insert user id here
		!ls /home/$uid/.local/share/jupyter/kernels/
		```

		%% Cell type:markdown id: tags:

		# How to start-up a notebook
		In order to access python modules (or functions from a python module) they must be explicitly loaded into the notebook.

		%% Cell type:code id: tags:

		``` python
		import numpy as np
		import blah from BLAH
		import BLAH
		```

		%% Cell type:markdown id: tags:

		## Cells
		Cells are isolated blocks of code that can be run individually. Although the code is sequester within a cell, the variables created in a cell can be accessed else were. For example,

		%% Cell type:code id: tags:

		``` python
		a = 2+2
		```

		%% Cell type:code id: tags:

		``` python
		print(a)
		```

		%% Output

		4

		%% Cell type:markdown id: tags:

		## Documenting

		%% Cell type:code id: tags:

		``` python
		```

		%% Cell type:code id: tags:

		``` python
		```

		%% Cell type:markdown id: tags:

		# Killing Notebooks
		Notebooks are hosted on the nebula cluster, unless specifically shut down, the job running the jupyter notebook will continue running even if jupyter hub fails. It is important to cancel your jupyter jobs after you are finished or you will get made fun of.

		%% Cell type:code id: tags:

		``` python
		!squeue -u ladoramkershner
		```

		%% Output

		JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
		21438177 longall jupyter- ladoramk R 8-07:33:17 1 neb15

		%% Cell type:markdown id: tags:


		`scancel [jobid]`

docs/users/workshops/FY21_workshop/network_generation.ipynb

0 → 100644

+185 −0

Original line number	Diff line number	Diff line
		%% Cell type:markdown id: tags:

		# Set up

		%% Cell type:code id: tags:

		``` python
		import os
		os.environ['ISISROOT'] = '/usgs/cpkgs/anaconda3_linux/envs/isis4.2.0'

		import ctypes
		from ctypes.util import find_library
		ctypes.CDLL(find_library('usgscsm'))

		from pysis import isis

		from autocnet.graph.network import NetworkCandidateGraph
		from autocnet.io.db.model import Measures, Points, Images

		import matplotlib.pyplot as plt
		import pandas as pd
		```

		%% Cell type:code id: tags:

		``` python
		# Morning or Evening?
		morning = 'true'
		kw_dict = {'true': 'morning',
		'false': 'evening'}
		kw = kw_dict[morning]
		```

		%% Cell type:code id: tags:

		``` python
		config_path = f'/home/ladoramkershner/projects/config_autocnet/phobos/phobos_workshop_network_generation.yml'
		```

		%% Cell type:code id: tags:

		``` python
		from autocnet.config_parser import parse_config
		config = parse_config(config_path)
		print('db name: ', config['database']['name'])
		log_dir = config['cluster']['cluster_log_dir']
		print('log dir: ', log_dir)
		print('ISISROOT: ', config['env']['ISISROOT'])
		print('conda env: ', config['env']['conda'])
		```

		%% Cell type:markdown id: tags:

		# Read in

		%% Cell type:markdown id: tags:

		### First Run - File List

		%% Cell type:code id: tags:

		``` python
		filelist = f'/scratch/ladoramkershner/phobos.lis'

		ncg = NetworkCandidateGraph()
		ncg.config_from_file(config_path)
		ncg.add_from_filelist(filelist, clear_db=True)
		```

		%% Cell type:markdown id: tags:

		### All Subsequent Runs

		%% Cell type:code id: tags:

		``` python
		ncg = NetworkCandidateGraph()
		ncg.config_from_file(config_path)
		ncg.from_database()
		```

		%% Cell type:code id: tags:

		``` python
		print('Cluster queue length:')
		print('before -> ', ncg.queue_length)
		```

		%% Cell type:code id: tags:

		``` python
		print('Cleaning up cluster queue:')
		ncg.queue_flushdb()
		print('after -> ', ncg.queue_length)
		```

		%% Cell type:markdown id: tags:

		### Visualize Graph

		%% Cell type:code id: tags:

		``` python
		ncg.plot()
		```

		%% Cell type:markdown id: tags:

		# Place Points in Overlap

		%% Cell type:code id: tags:

		``` python
		# Make sure the output log directory exists
		ppio_log_dir = log_dir.replace('logs', 'ppio_logs')
		print(ppio_log_dir)

		if not os.path.exists(ppio_log_dir):
		os.mkdir(ppio_log_dir)

		print('Exists: ', os.path.exists(ppio_log_dir))
		```

		%% Cell type:code id: tags:

		``` python
		def ns(x):
		from math import ceil
		return ceil(round(x,1)*8)

		def ew(x):
		from math import ceil
		return ceil(round(x,1)*1)

		distribute_points_kwargs = {'nspts_func':ns, 'ewpts_func':ew, 'method':'classic'}

		njobs = ncg.apply('spatial.overlap.place_points_in_overlap',
		on='overlaps',
		walltime='00:20:00',
		cam_type='isis',
		distribute_points_kwargs=distribute_points_kwargs,
		log_dir=ppio_log_dir,
		arraychunk=100,
		chunksize=2248)
		print(njobs)
		```

		%% Cell type:markdown id: tags:

		### ppio: Check queue -> reapply (if needed)

		%% Cell type:code id: tags:

		``` python
		redis_orphans = ncg.queue_length
		print("jobs left on the queue: ", redis_orphans)
		```

		%% Cell type:code id: tags:

		``` python
		# job_array = ncg.apply('spatial.overlap.place_points_in_overlap',
		# chunksize=redis_orphans,
		# arraychunk=None,
		# walltime='00:30:00',
		# log_dir=ppio_log_dir,
		# reapply=True)
		# print(job_array)
		```

		%% Cell type:markdown id: tags:

		### ppio: Write out Network

		%% Cell type:code id: tags:

		``` python
		path = '/scratch/ladoramkershner/kaguya/2021_mosaics/reiner_gamma/morning/control'
		cnet = 'reiner_gamma_morning_ns8_ew1.net'
		ncg.to_isis(os.path.join(path,cnet)) # write out
		```

		%% Cell type:markdown id: tags:

		I am going to do the point registration in isis, autocnet has had work on their subpixel registration techniques and I am not sure which one to use at the moment