Unverified Commit cb121663 authored Feb 25, 2025 by Adam Paquette Committed by GitHub Feb 25, 2025

GTiff Reading and Writing (#5618)



* replaced custom min/max with std::max/std::min

* left a gdal reference in

* Initial changes for a GdalIoHandler and creating an external DN cube

* added write + tests

* Initial move away two flags on the cube to using the LabelAttachment to define label state

* Addressed PR feedback

* Added external gdal support to cube

* Changed Tiff type to GTiff

* Renamed imageIoHandler to ImageIoHandler

* Added pixeltype to gdaliohandler constructor

* Added gdal pixel type translation and missing special pixels for various ISIS pixel types

* Small fixes to get things working

* Removed prints and added -1s back to read and write

* Changed gdal reads to use the raw buffer and correctly apply the ISIS pixel type

* Added initial pixel conversions and tests for writing

* Final tests for pixel type reading and writing

* Added tiff fixtures

* Transfered pixel type to gdal handler in cube

* Silenced unsued return warning

* Reused file handler in TiffFixture

* added virtual band checks

* Added default compression for tiffs

* Removed max again

* Opened up the environment to get a work env

* Fixed boolean comparisons against enum

* Initial stab at fixing the buffer copy

* Adds msk file writting for tiffs

* Added boundary check for buffers

* Added fixture function to write a M x N x O tiff

* Reverted commented out Index access function Exceptions

* Default tiff DNs to the noDataValue

* Cleaned up buffer boundary handling for reads and writes

* Added gdal io tests for reading outside of the image

* Updated shadow functional tests

* Properly handled pointers for mac and linux

* Removed commented out line

* Addressed OS dependent shadow failures

* Removed read contraints on ecubes

* Moved CubeIoHandler getters into public space

* Handled gdal warnings in GdalIoHandler and its tests

* Fixed compiler warnings in TiffFixture

* Reverted data cube truncation

* gdal json pvl parsing

* Update Pvl.cpp

* Fixed gdal json to pvl parsing

* Updated cube tests with tiff read/write changes

* Moved Tiff creation into a function on Cube and made Tiff Cube metadata read an explicit constructor

* Added PVL to Json export for GDAL metadata

* Fixed PVLKeyword toJson writing arrays

* Added string conversion for GTiff

* Extracted PVL read json lambda into static function

* Enforced Cube carry/manage GdalDataset object and made GdalIoHandler and Blob use it

* Fixed tiff fixtures alpha bands

* Seperated ISD spice contructor and init function from default init function

* Fixed blob reading and writing from hex

* Fixed error message when reading a gdal blob

* Fixed setting dataset ownership correctly on file constructor

* Fixed closing the dataset if we don't find the USGS metadata

* Fixed cube test expected label size

* Changed cube blob reading apps to access the blobs via the cube

* Small fixes

* Fixed issue with cam2map requesting buffers completely outside of the tiff

* Applied cube label reading to catlab

* Added blob read exception for GdalRead

* Exposed IException vector length

* Handled GDALs nodatavalue

* Moved GDAL pvl label construction into the Cube

* Commented out mask creation

* Forced label reading to come from the cube in ctxcal

* Fixed data ownership when reading metadata from GTiff

* Finalized mapping group generation from spatial reference

* Fix reading URLs in qview and blob reading

* Added a scaling factor to the ISIS buffer classes

* Updated data iohandlers and viewportbuffer to handle scaled buffers

* Finished cube open/create methods to handle GTiffs and Cubes

* Exposed scale on bufferfill action

* Updated tests when trying to open a bogus cube

* Allowed qview file dialog to open cloud volumes

* Minor buffer indexing fix

* Updated GTiff related tests

* Added GeoTiff tests

* Converted blob tests to Gtests

* Updated Cube to handle detached and external correctly for GTiff format

* Allow apps to handle other PVL files and fix failing blob tests

* Updated cube manager tests truth

* Fixed hexstream conversions on linux

* Fix unsigned int comparison in GdalIoHandler

* Made TestCubeAttachLineScanTableFromIsd use the temp directory fixture

* Actually fix hex reading and writing

* Set Predictor for GeoTIFFs in ISIS

* Removed unused variables causing compiler warnings

* Set GDAL format from driver

* Removed getting driver name, caused errors on AWS Linux

* Updated changelog

---------

Co-authored-by: Kelvin <krodriguez@usgs.gov>

parent c94c2e8c

CHANGELOG.md

+1 −0

Original line number	Diff line number	Diff line
		@@ -39,6 +39,7 @@ release.
		- Added adjusted XYZ point coordinate sigmas to the points.csv jigsaw output file. Modified
		ctest FunctionalTestJigsawApollo to validate this output. [#5710](https://github.com/DOI-USGS/ISIS3/issues/5710)
		- Added OFFBODY and OFFBODYTRIM parameters to cam2cam. Added tests and updated documentation. [#3602] (https://github.com/DOI-USGS/ISIS3/issues/3602)
		- Added support for reading, writing, and viewing GeoTIFFs in ISIS. [#5618](https://github.com/DOI-USGS/ISIS3/pull/5618)

		### Changed

environment.yml

+9 −8

Original line number	Diff line number	Diff line
		@@ -5,8 +5,8 @@ dependencies:
		- ale =0.10.0,<1
		- aom
		- armadillo
		- boost >=1.78.0,<1.79
		- boost-cpp >=1.78.0,<1.79
		- boost >=1.76.0,<1.82
		- boost-cpp >=1.76.0,<1.82
		- blas
		- usgs-astrogeology::bullet
		- bz2file
		@@ -22,6 +22,7 @@ dependencies:
		- ffmpeg
		- geos >=3.12, <3.13
		- geotiff
		- gdal >=3.5 ,<3.9.0a0
		- gmp
		- graphviz
		- conda-forge::gsl >=2.6, <2.7
		@@ -34,10 +35,10 @@ dependencies:
		- kalasiris
		- krb5
		- libgdal >=3.7,<3.8
		- libopencv >=4.7,<4.8
		- libopencv >=4.5,<4.8
		- libpng >=1.6.34,<1.7
		- libprotobuf
		- libtiff >=4.5,<4.6
		- libtiff >=4.5,<4.8
		- libxml2
		- make
		- mesalib
		@@ -47,17 +48,17 @@ dependencies:
		- nlohmann_json=3.11.2
		- ninja >=1.7.2,<1.8
		- nn
		- opencv >=4.7,<4.8
		- opencv >=4.5,<4.8
		- openssl >=3.1.2,<3.2
		- pcl >=1.13,<1.14
		- plio
		- protobuf
		- python >=3.10
		- python_abi >=3.10
		- python >=3.9
		- python_abi >=3.9
		- pytest
		- rclone
		- qhull
		- qt-main>=5.15.8, <5.16
		- qt-main>=5.15, <5.16
		- qwt <6.3.0
		- sqlite >=3.46.0,<3.47
		- suitesparse <7.7.0

gtest @ f8d7d77c

Original line number	Diff line number	Diff line
		Subproject commit b10fad38c4026a29ea6561ab15fc4818170d1c10
		Subproject commit f8d7d77c06936315286eb55f8de22cd23c188571

isis/CMakeLists.txt

+2 −1

Original line number	Diff line number	Diff line
		@@ -267,6 +267,7 @@ find_package(CSM 3.0.3.3 REQUIRED)
		find_package(CSPICE 65 REQUIRED)
		find_package(Eigen REQUIRED)
		find_package(Embree 3.13.0 REQUIRED)
		find_package(GDAL REQUIRED CONFIG)
		find_package(GeoTIFF 2 REQUIRED)
		SET(GSL_ROOT_DIR $ENV{CONDA_PREFIX})
		find_package(GSL 2.2.1 REQUIRED)
		@@ -353,7 +354,7 @@ foreach (_variableName ${_variableNames})
		endforeach()

		# add target based linkages to ALLLIBS variable
		list(APPEND ALLLIBS pantor::inja sensorutilities protobuf::libprotobuf embree)
		list(APPEND ALLLIBS pantor::inja sensorutilities protobuf::libprotobuf embree GDAL::GDAL)

		# Sometimes we add the same lib more than once (especially with LIBDIRS)
		list(REMOVE_DUPLICATES ALLLIBDIRS)

isis/src/apollo/apps/apollopanstitcher/apollopanstitcher.cpp

+3 −3

Original line number	Diff line number	Diff line
		@@ -715,11 +715,11 @@ namespace Isis {
		att.setFileFormat( panC[0]->format() );
		att.setByteOrder( panC[0]->byteOrder() );
		att.setPixelType( panC[0]->pixelType() );
		if (panC[0]->labelsAttached()) {
		att.setLabelAttachment(AttachedLabel);
		if (panC[0]->labelsAttached() == Cube::AttachedLabel) {
		att.setLabelAttachment(Cube::AttachedLabel);
		}
		else {
		att.setLabelAttachment(DetachedLabel);
		att.setLabelAttachment(Cube::DetachedLabel);
		}

		//define an output cube