Merge pull request #18 from jessemapel/VersionHistory (762420a4) · Commits · aflab / astrogeology / ISIS3

isis/src/control/objs/ControlNetVersioner/ControlNetVersioner.h

+234 −33

Original line number	Diff line number	Diff line
		@@ -49,6 +49,8 @@ namespace Isis {
		/**
		* @brief Handle various control network file format versions.
		*
		* <h3>Overview</h3>
		*
		* This class is used to read all versions of control networks and write out
		* the most recent version in Pvl and protobuf format. When reading a
		* control net file, the ControlNeVersioner is initialized with the
		@@ -59,23 +61,29 @@ namespace Isis {
		* formats. ControlNet can then interface with this class and only has to
		* work with the current ControlPoint object.
		*
		* <h3>Reading Control Network Files</h3>
		*
		* The read routine is as follows:
		* 1. Read the Pvl file header.
		* 2. Determine if the network is stored in Pvl or protobuf format
		* 3. Determine the version of the network
		* 4. Read in the general ControlNet information such as network
		* <ol>
		* <li>Read the Pvl file header</li>
		* <li>Determine if the network is stored in Pvl or protobuf format</li>
		* <li>Determine the version of the network</li>
		* <li>Read in the general ControlNet information such as network
		* description, last modification date, etc. For Pvl networks, this
		* information is in a PvlObject at the start of the network. For
		* protobuf objects this information is stored in a header protobuf
		* message after the Pvl file header.
		* 5. For each control point do the following:
		* a. Read the control point into the appropriate ControlPointV####
		* object.
		* b. If the ControlPointV#### object is not the most recent version,
		* upgrade it to the latest version.
		* c. Convert the final ControlPointV#### object into a ControlPoint
		* object and store it in the ControlNetVersioner.
		* message after the Pvl file header.</li>
		* <li>For each control point do the following:
		* <ol type="a">
		* <li>Read the control point into the appropriate ControlPointV####
		* object.</li>
		* <li>If the ControlPointV#### object is not the most recent version,
		* upgrade it to the latest version.</li>
		* <li>Convert the final ControlPointV#### object into a ControlPoint
		* object and store it in the ControlNetVersioner.</li>
		* </ol>
		* </li>
		* </ol>
		*
		* Once the ControlNet file is read into the ControlNetVersioner, the
		* ControlPoints can be accessed through the takeFirstPoint method. This
		@@ -85,28 +93,33 @@ namespace Isis {
		* information about the control network can be accessed directly from the
		* ControlNetVersioner.
		*
		* <h3>Writing Control Network Files</h3>
		*
		* The protobuf file write routine is as follows:
		* 1. Copy the general ControlNet information such as network description,
		* last modification date, etc. into the ControlNetVersioner.
		* 2. Copy the pointers to the ControlPoints and store them in the
		* <ol>
		* <li>Copy the general ControlNet information such as network description,
		* last modification date, etc. into the ControlNetVersioner.</li>
		* <li>Copy the pointers to the ControlPoints and store them in the
		* ControlNetVersioner. The ControlNetVersioner does not assume
		* ownership of the ControlPoints when it does this. The ControlNet
		* or what the ControlNet got the points from retains ownership.
		* 3. Write a 65536 byte blank header to the file.
		* 4. Write the general ControlNet information to a protobuf message header
		* after the blank header.
		* 5. For each control point do the following:
		* a. Convert the control point into a protobuf message.
		* b. Write the size of the protobuf message to the file.
		* c. Write the protobuf message to the file.
		* 6. Write a Pvl header into the original blank header at the start of the
		* file. This header contains a flag indicating the network is a
		* or what the ControlNet got the points from retains ownership.</li>
		* <li>Write a 65536 byte blank header to the file.</li>
		* <li>Write the general ControlNet information to a protobuf message header
		* after the blank header.</li>
		* <li>For each control point do the following:
		* <ol type="a">
		* <li>Convert the control point into a protobuf message.</li>
		* <li>Write the size of the protobuf message to the file.</li>
		* <li>Write the protobuf message to the file.</li>
		* </ol>
		* </li>
		* <li>Write a Pvl header into the original blank header at the start of the
		* file. This header contains: a flag indicating the network is a
		* protobuf formatted network, the version of the format, the byte
		* offset and size of the protobuf header, the byte offset and
		* size of the block of protobuf control points, and general
		* information about the control network.
		*
		* information about the control network.</li>
		* </ol>
		*
		* Once the ControlNetVersioner is initialized from a file or a ControlNet,
		* a Pvl formatted version of the control network can be created by the
		@@ -114,27 +127,32 @@ namespace Isis {
		* Pvl format. From here, the Pvl network can be written to a file with
		* Pvl::write(filename).
		*
		* <h3>Modifying the Control Network File Format</h3>
		*
		* If the control network file format is changed the following changes need
		* to be made to ControlNetVersioner and its supporting classes:
		*
		*<ul>
		* <li>
		* New containers need to be added to interface with the new format. These
		* containers should try to match the format of the data in the file, then
		* the versioner will convert from that format to ControlNet, ControlPoint,
		* ControlMeasure, and ControlMeasureLogData. General information about the
		* control network should be stored in the ControlNetHeaderV#### structs.
		* Data for control points should be stored in ControlPointV#### objects.
		*
		* </li>
		* <li>
		* If a new control point container was created, code needs to be added to
		* create ControlPoint objects from them. A new createPoint method that
		* takes the new container should be added to do this. The createMeasure
		* method should also be changed to create ControlMeasures from the new
		* containers.
		*
		* </li>
		* <li>
		* If a new header container was created, code needs to be added to store its
		* information in the ControlNetVersioner. A new createHeader method that
		* takes the new header container should be added to do this.
		*
		* </li>
		* <li>
		* New code needs to be added to update the previous containers to the new
		* containers. Updating control point containers should happen in the new
		* ControlPointV#### container class. It should have a constructor that
		@@ -144,18 +162,199 @@ namespace Isis {
		* Updating header containers should happen in the createHeader method that
		* takes the previous version. If the headers become more complicated, this
		* may need to change to match how control point containers are updated.
		*
		* </li>
		* <li>
		* New methods need to be added to read the new file format. Methods for
		* reading Pvl formatted files and protobuf formatted files need to be
		* added; they should match the naming convention of readPvlV#### and
		* readProtobufV#### respectively.
		*
		* </li>
		* <li>
		* New methods need to be added to write out the new file format. The write
		* method should be changed to write out the new protobuf format. If
		* a new header container is added, the writeHeader method should be
		* changed to write the new protobuf header to the file. If a new control
		* point container is added, the writeFirstPoint method should be changed
		* to write a new protobuf control point to the file.
		* </li>
		* <li>
		* Update the documentation on this class under the <b>Control Network File
		* Format History</b> heading. This should include a description of the new
		* file format and how it differs from the previous version.
		* </li>
		*</ul>
		*
		* <h3>Control Network File Format History</h3>
		*
		* Prior to the creation of this versioning class, which was released with
		* ISIS 3.2.2, all control network files were Pvl formatted text files.
		* Reading and writing these files was handled by the ControlNet, ControlPoint,
		* and ControlMeasure classes. As the file format was changed, those
		* classes were modified to account for the new format. Because of this,
		* the history of the control network file format prior to ISIS 3.2.2
		* is not well documented.
		*
		* The following are descriptions of the different control network file
		* format versions that are currently supported. Each description also
		* describes how that version differs from the previous.
		*
		* <b>Version 1</b>
		*
		* This version maintains backwards compatibility with all files created
		* prior to versioning. If a control network file does not have a version
		* flag, then it is assumed to be a version 1 file. Because this version
		* supports all files created prior to ISIS 3.2.2, there is no standardized
		* Pvl format associated with it.
		*
		* Originally, there was no version 1 binary format. When version 2 was
		* added, version 1 was changed to use the version 2 binary format.
		*
		* <b>Version 2</b>
		*
		* This version was the first to have a standardized format. The following
		* were standardized with this format:
		* <ul>
		* <li>The Held flag was replaced by point types. Points could be either
		* Tie points or Ground points. Points that were previously flagged as
		* Held were changed to Ground points.</li>
		* <li>A posteriori was replaced with adjusted in several keyword
		* names.</li>
		* <li>The a priori and adjusted ground points were changed from
		* (latitude, longitude, radius) format to body fixed (X, Y, Z) format.
		* </li>
		* <li>Ground point sigmas were replaced with covariance matrices.</li>
		* <li>Latitude, longitude, and radius constrained flags were added.</li>
		* <li>Estimated measures were renamed to Candidate measures.</li>
		* <li>Unmeasured measures were renamed to Candidate measures, had their
		* line and sample set to 0, and were flagged as ignored.</li>
		* <li>Automatic, ValidatedManual and AutomaticPixel measures were
		* renamed to RegisteredPixel measures.</li>
		* <li>ValidatedAutomatic and AutomaticSubPixel measures were renamed to
		* RegisteredSubPixel measures.</li>
		* <li>ErrorSample and ErrorLine were renamed to SampleResidual and
		* LineResidual respectively.</li>
		* <li>Diameter, ZScore, and ErrorMagnitude were no longer saved in
		* measures.</li>
		* </ul>
		*
		* Version 2 was the first version to support a binary protobuf format.
		* Version 1 was retroactively changed to use the version 2 binary format.
		* Version 2 binary control network files consist of three parts:
		* <ol>
		* <li>
		* <em>Pvl File Header:</em> The file starts with a Pvl formatted header
		* that contains offsets to the binary components of the file and the
		* version number of the file. This header may also contain general
		* information about the control network that is only for user reference
		* is not used when reading the file.
		* </li>
		* <li>
		* <em>Protobuf Core:</em> After the Pvl header is the protobuf core that
		* contains the majority of the network data. This is a hierarchical
		* structure with general network information such as the network
		* description at the top level. Below that is the control point
		* information. The lowest level contains the control measure information.
		* This structure is defined by <em>ControlNetFileProtoV0001.proto</em>.
		* </li>
		* <li>
		* <em>Protobuf Log Data:</em> The final component of the file contains
		* the control measure log data. This is structured the same as the
		* protobuf core. So, the log data for the i<sup>th</sup> measure in the
		* j<sup>th</sup> point in the core is in the i<sup>th</sup> measure of
		* the j<sup>th</sup> point in this structure. This structure is defined
		* by <em>ControlNetLogDataProtoV0001.proto</em>.
		* </li>
		* </ol>
		*
		* <b>Version 3</b>
		*
		* This version was created to avoid file size limits imposed by the
		* version 2 binary file format. Protobuf messages are limited to 2GB for
		* security reasons. So, version 2 binary files can only contain 2GB of
		* information in their Protobuf Core. In version 3, the Protobuf Core was
		* changed from a single message to a header message and individual
		* messages for each control point. This way, the Protobuf Core could
		* contain an arbitrary amount of information as long as any single point
		* does not exceed 2GB. At the same time, control measure log data was
		* moved into the Protobuf Core so that each measure contains its own log
		* data.
		*
		* Version 3 binary control network files are formatted as follows:
		* <ul>
		* <li>
		* <em>pvl File Header:</em> The Pvl Header in version 3 binary files
		* is the same as the Pvl Header in version 2 binary files, except it has
		* offsets to the Protobuf Header and Protobuf Core instead of the
		* Protobuf Core and Protobuf Log Data.
		* </li>
		* <li>
		* <em>Protobuf Header:</em> The binary component of version 3 binary
		* control network files starts with a protobuf message header that
		* contains general information about the network and the size of each
		* control point's protobuf message. The size of each control point
		* message is required to parse the Protobuf Core because protobuf
		* messages are not self-delimiting. This structure is defined by
		* <em>ControlNetFileHeaderV0002.proto</em>.
		* </li>
		* <li>
		* <em>Protobuf Core:</em> Immediately after the Protobuf Header is
		* the Protobuf Core which contains all of the control point and control
		* measure information. This is structured as consecutive protobuf
		* messages where each message contains all of the information for a
		* control point and its control measures. Because protobuf messages
		* are not self-delimiting, the size of each message must be known prior
		* to parsing the Protobuf Core. The control point messages are defined
		* by <em>ControlPointFileEntryV0002.proto</em>.
		* </li>
		* </ul>
		*
		* Version 3 also further differentiated control point types. Control
		* points that had their latitude, longitude, and/or radius constrained
		* were changed from Tie points to Constrained points.
		*
		* <b>Version 4</b>
		*
		* This version was created when Ground and Tie control points were renamed
		* to Fixed and Free respectively. Version 4 Pvl control network files are
		* identical to version 3 Pvl control network files, except for the new
		* control point type values. When version 4 was created, the .proto file
		* that defined control point protobuf messages was modified to allow for
		* the new control point type names and the old names were flagged as
		* deprecated. So, version 3 and version 4 binary control network files are
		* formatted exactly the same.
		*
		* <b>Version 5</b>
		*
		* This version was created to allow for progressive reading and writing of
		* binary control network files. Previous versions required the entire
		* contents of binary control network files to be read into memory before
		* the ControlNet, ControlPoint, and ControlMeasure objects could be
		* created. Version 5 was created to allow binary control network files
		* to be read one control point at a time. Similarly, previous versions
		* required all of the information in the control network to be copied into
		* protobuf structures before any of it could be written to a file.
		*
		* Version 5 Pvl control network files are identical to version 4 Pvl
		* control network files.
		*
		* Version 5 binary control network files are formatted the same as version
		* 4 binary control network files except for how they store the sizes of
		* the control point messages in the Protobuf Core. In a version 5 binary
		* control network file, each control point message is prepended by an
		* unsigned, 32 bit, LSB, integer (c++ uint32_t) that contains the size of
		* the message. This design was modeled after the delimited read and write
		* functionality in the Java protobuf library. Additionally, the Protobuf
		* Header contains the number of control points in the network instead of
		* a list of all the control point message sizes. The structure of the
		* Protobuf Header is defined by <em>ControlNetFileHeaderV0005.proto</em>.
		* The structure of the protobuf messages in the Protobuf Core is defined
		* by <em>ControlPointFileEntryV0002.proto</em>, the same as in version 4.
		*
		* Starting with version 5, the naming scheme for .proto files was changed
		* to use the same version number as the control net file format. So, the
		* new .proto file defining the Protobuf Header was named
		* <em>ControlNetFileHeaderV0005.proto</em> instead of
		* <em>ControlNetFileHeaderV0003.proto</em>.
		*
		* @ingroup ControlNetwork
		*
		@@ -199,6 +398,8 @@ namespace Isis {
		* @history 2018-01-12 Adam Goins - Added the ControlPoint radii to the header to avoid
		* Target::GetRadii calls to speed up createPoint().
		* @history 2018-01-12 Adam Goins - Added Progress during reads.
		* @history 2018-01-27 Jesse Mapel - Fixed some documentation formatting. Added a section
		* describing the different file format versions.
		*/
		class ControlNetVersioner {

isis/src/control/objs/ControlNetVersioner/ControlPointV0001.h

+20 −3

Original line number	Diff line number	Diff line
		@@ -36,9 +36,25 @@ namespace Isis {
		* @brief A container for the information stored in a version 1 ControlPoint.
		*
		* A wrapper around the version 1 protobuf serialization of a ControlPoint. It allows for reading
		* ControlPoints serialized as both PvlObjects and protobuf messages. In order to simplify the
		* upgrade process from version 1 to version 2, the data is always stored in a protobuf message
		* after being read.
		* ControlPoints serialized as both PvlObjects and protobuf messages.
		*
		* This class is designed to be compatible with all Pvl formats created prior
		* to the use of versioning. So, the PvlObject constructor has to work with
		* several different formats. Hence, several different keywords are checked
		* for the same value.
		*
		* Because this version supports several different formats, there is no
		* standardized set of keywords, but all version 1 Pvl control networks
		* have the same high level structure. Control points are represented by
		* objects contained in the ControlNetwork object. Control measures are
		* represented by groups contained in the control point objects.
		*
		* Once read in, the data is always stored in a protobuf message regardless
		* of the source. This is done to optimize reading binary control network
		* files. Because the protobuf format for version 1 control points is
		* identical to the version 2 format, control points read from a version 1
		* file are automatically "converted" to version 2. This also makes the
		* version 1 to 2 upgrade process as simple as passing a pointer.
		*
		* @ingroup ControlNetwork
		*
		@@ -48,6 +64,7 @@ namespace Isis {
		* @history 2017-12-18 Jesse Mapel - Original version.
		* @history 2017-12-21 Adam Goins - Changed Pvl constructor to take PvlObject.
		* @history 2017-12-21 Jesse Mapel - Improved documentation.
		* @history 2017-01-27 Jesse Mapel - More documentation improvements.
		*/
		class ControlPointV0001 {
		public:

isis/src/control/objs/ControlNetVersioner/ControlPointV0002.h

+116 −10

File changed.

Preview size limit exceeded, changes collapsed.

isis/src/control/objs/ControlNetVersioner/ControlPointV0003.h

+117 −2

File changed.

Preview size limit exceeded, changes collapsed.