Support WITH + fix a lot the case-sensitivity management. WARNING: the signature (63241bc6) · Commits · Sonia Zorba / vollt

identifiers_rules.md

0 → 100644

+218 −0

Original line number	Diff line number	Diff line
		_Date: 25th Sept. 2019_
		_Author: Grégory Mantelet_

		# Identifiers' rules in ADQL-Lib

		## Definitions

		These rules apply to any kind of identifier (e.g. table, column).

		To simplify explanations, we will consider here that an identifier is composed of 3 pieces of information:

		- adqlName _- string_
		- adqlCaseSensitive _- boolean_
		- dbName _- string_

		`adqlName` and `dbName` are never stored in their qualified (e.g. the table prefix inside a full column name) or their delimited (e.g. `"aTable"` is the delimited form of `aTable`) form ; surrounded double quotes (and escaped double quotes: `""`) and prefixes are used/checked at initialization but always discarded for storage.

		If `dbName` is not specified, it is the same as `adqlName`.

		`adqlCaseSensitive` is set to `true` only if `adqlName` was delimited at initialization.

		Rules about how to build this 3-tuple depend on the origin of the identifier (e.g. `TAP_SCHEMA`, subquery, CTE) ; these rules are detailed below.

		Let's now see how to write ADQL and SQL queries with a such 3-tuple...

		## Identifiers in ADQL

		In ADQL queries, an identifier MUST be delimited only in the following cases:

		* if a ADQL/SQL reserved keyword

		```sql
		-- declared table identifier: adqlName=`distance`
		SELECT ... FROM distance -- INCORRECT because `distance` is a reserved keyword
		SELECT ... FROM "distance" -- CORRECT
		```

		* if not a regular ADQL identifier

		```sql
		-- declared table identifier: adqlName=`2do`
		SELECT ... FROM 2do -- INCORRECT because `2do` is starting with a digit
		SELECT ... FROM "2do" -- CORRECT
		```

		* if ambiguous with another identifier of the same type

		```sql
		-- declared column identifiers: adqlName=`id` in `table1` and adqlName=`id` in `table2`
		SELECT id FROM table1, table2 -- INCORRECT because the column `id` exists in both tables
		SELECT table1.id FROM table1, table2 -- CORRECT
		```



		In any other case, the identifier _MAY_ be delimited, but if not, you are free to write it in upper-/lower-/mixed-case.

		If the identifier is declared in a CASE-SENSITIVE way, it MUST be respected when delimited in the ADQL query.

		If the identifier is CASE-INSENSITIVE, its delimited ADQL version MUST be all in lower-case.

		Then, the following ADQL queries are perfectly allowed:

		```sql
		-- declared table: adqlName=`aTable`, adqlCaseSensitive=`false`
		SELECT ... FROM atable -- OK
		SELECT ... FROM ATABLE -- OK
		SELECT ... FROM "atable" -- OK (because lower-case if not declared CS)
		SELECT ... FROM "aTable" -- INCORRECT

		-- declared table: adqlName=`aTable`, adqlCaseSensitive=`true`
		SELECT ... FROM atable -- OK
		SELECT ... FROM ATABLE -- OK
		SELECT ... FROM "atable" -- INCORRECT
		SELECT ... FROM "aTable" -- OK
		```

		## SQL translation

		_In this part, we will consider PostgreSQL as SQL target._

		The `dbName` of an identifier is _always_ considered as _case-sensitive_. So, it will _always_ be written delimited in SQL queries.

		Examples of SQL queries:

		```sql
		-- declared table: adqlName=`aTable`, dbName=-
		-- ADQL: SELECT ... FROM atable
		SELECT ... FROM "aTable"

		-- declare table: adqlName=`aTable`, dbName=`DBTable`
		-- ADQL: SELECT ... FROM atable
		SELECT ... FROM "DBTable"
		```

		## Automatic column aliases in SQL

		To ensure having the expected labels in SQL query results, aliases are automatically added (if none is specified) to all items of the `SELECT` clause.

		As `dbName`s, these default aliases are considered as case sensitive.

		They are built using the `adqlName` of the aliased identifiers. If this name is _not case-sensitive_, the alias will be written in lower-case. But if _case-sensitive_, it is written exactly the same.

		Examples of SQL queries:

		```sql
		-- declared table: adqlName=`aTable`, dbName=`DBTable`
		-- declared columns in `aTable`:
		-- * adqlName=`ColCS`, adqlCaseSensitive=`false`, dbName=`dbCol1`
		-- * adqlName=`ColCI`, adqlCaseSensitive=`true`, dbName=`dbCol2`
		-- ADQL: SELECT colcs, colci FROM atable
		SELECT "dbCol1" AS "colcs", "dbCol2" AS "ColCI" FROM "DBTable"
		```



		## Schemas/Tables/Columns declared in `TAP_SCHEMA`

		* `adqlName` = `TAP_SCHEMA.(schemas.schema_name\|tables.table_name\|columns.column_name)`
		* `dbName` = `TAP_SCHEMA.(schemas\|tables\|columns).dbName` or if NULL `adqlName`

		Examples with `TAP_SCHEMA.tables`:

		\| In TAP_SCHEMA.tables \| In ADQL-Lib \|
		\| ---------------------------------------------- \| ------------------------------------------------------------ \|
		\| table_name=`aTable`, dbName=_null_ \| adqlName=`aTable`, adqlCaseSensitive=`false`, dbName=_null_ \|
		\| table_name=`schema.aTable`, dbName=_null_ \| adqlName=`aTable`, adqlCaseSensitive=`false`, dbName=_null_ \|
		\| table_name=`"aTable"`, dbName=_null_ \| adqlName=`aTable`, adqlCaseSensitive=`true`, dbName=_null_ \|
		\| table_name=`schema."aTable"`, dbName=_null_ \| adqlName=`aTable`, adqlCaseSensitive=`true`, dbName=_null_ \|
		\| table_name=`aTable`, dbName=`DBTable` \| adqlName=`aTable`, adqlCaseSensitive=`false`, dbName=`DBTable` \|
		\| table_name=`aTable`, dbName=`"DBTable"` \| adqlName=`aTable`, adqlCaseSensitive=`false`, dbName=`DBTable` \|
		\| table_name=`aTable`, dbName=`schema.DBTable` \| adqlName=`aTable`, adqlCaseSensitive=`false`, dbName=`schema.DBTable` \|
		\| table_name=`aTable`, dbName=`schema."DBTable"` \| adqlName=`aTable`, adqlCaseSensitive=`false`, dbName=`schema."DBTable"` \|

		## Tables from subqueries (i.e. `FROM` and `WITH`)

		_Reminder: in ADQL, a subquery declared as table in the `FROM` clause and a CTE declared in the `WITH` clause MUST always be aliased/named._

		* `adqlName` = _subquery's alias/CTE's name_
		* `dbName` = `adqlName`

		If the alias/name is delimited (i.e. case sensitive), `adqlCaseSensitive` will be set to `true` and surrounding double quotes are removed from `adqlName` .

		If the alias/name is not delimited, `adqlName` is set to the alias put into lower-case, and `adqlCaseSensitive` is `false`.

		Examples:

		```sql
		--
		-- Subqueries in FROM:
		--
		SELECT ... FROM (SELECT * FROM atable) AS t1
		SELECT ... FROM (SELECT * FROM atable) AS T1
		-- => adqlName=`t1`, adqlCaseSensitive=`false`, dbName=`t1`

		SELECT ... FROM (SELECT * FROM atable) AS "T1"
		-- => adqlName=`T1`, adqlCaseSensitive=`true`, dbName=`T1`

		--
		-- CTEs in WITH:
		--
		WITH t1 AS (SELECT * FROM atable) SELECT ... FROM t1
		WITH T1 AS (SELECT * FROM atable) SELECT ... FROM t1
		-- => adqlName=`t1`, adqlCaseSensitive=`false`, dbName=`t1`

		WITH "T1" AS (SELECT * FROM atable) SELECT ... FROM t1
		-- => adqlName=`T1`, adqlCaseSensitive=`true`, dbName=`T1`
		```

		## Columns of a (sub)query

		* If _NOT aliased_:

		* `adqlName` = _original's `adqlName`_
		* `adqlCaseSensitive` = _original's `adqlCaseSensitive`_
		* `dbName` = _original's `dbName`_



		* If _aliased_:

		* `adqlName` = _alias in lower-case if not delimited, exact same alias otherwise_
		* `adqlCaseSensitive` = `true` _if alias is delimited_, `false` _otherwise_
		* `dbName` = `adqlName`



		Examples:

		```sql
		-- declared column: adqlName=`aColumn`, adqlCaseSensitive=`false`, dbName=`DBCol`
		SELECT acolumn FROM atable
		SELECT AColumn FROM atable
		-- => the declared column

		-- declared column: adqlName=`aColumn`, adqlCaseSensitive=`true`, dbName=`DBCol`
		SELECT acolumn FROM atable
		SELECT AColumn FROM atable
		-- => the declared column

		-- declared column: adqlName=`aColumn`, adqlCaseSensitive=`true`, dbName=`DBCol`
		SELECT acolumn AS myColumn FROM atable
		-- => adqlName=`mycolumn`, adqlCaseSensitive=`false`, dbName=`mycolumn`
		SELECT acolumn AS "myColumn" FROM atable
		-- => adqlName=`myColumn`, adqlCaseSensitive=`true`, dbName=`myColumn`

		```

		_Note: The new representation of an aliased column has different ADQL and DB names, but the other metadata (e.g. datatype, UCD, ...) of the original column are copied as such._

		## Duplicated output columns

		_The term 'output columns' means here the columns written in the output format (e.g. VOTable). They are not the columns represented as a 3-tuple in this document._

		TODO

src/adql/db/CheckContext.java

0 → 100644

+65 −0

Original line number	Diff line number	Diff line
		package adql.db;

		/*
		* This file is part of ADQLLibrary.
		*
		* ADQLLibrary is free software: you can redistribute it and/or modify
		* it under the terms of the GNU Lesser General Public License as published by
		* the Free Software Foundation, either version 3 of the License, or
		* (at your option) any later version.
		*
		* ADQLLibrary is distributed in the hope that it will be useful,
		* but WITHOUT ANY WARRANTY; without even the implied warranty of
		* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
		* GNU Lesser General Public License for more details.
		*
		* You should have received a copy of the GNU Lesser General Public License
		* along with ADQLLibrary. If not, see <http://www.gnu.org/licenses/>.
		*
		* Copyright 2019 - UDS/Centre de Données astronomiques de Strasbourg (CDS)
		*/

		/**
		* State of the {@link DBChecker} at one recursion level inside an ADQL query.
		*
		* <p>
		* An instance of this class aims to list columns and Common Table Expressions
		* (i.e. CTE - temporary tables defined in the WITH clause) available inside
		* a specific ADQL (sub-)query.
		* </p>
		*
		* @author Grégory Mantelet (CDS)
		* @version 2.0 (10/2019)
		* @since 2.0
		*/
		public class CheckContext {

		/** List of available CTEs at this level. */
		public final SearchTableApi cteTables;

		/** List of available columns (of all tables). */
		public final SearchColumnList availableColumns;

		/**
		* Create a context with the given list of CTEs and columns.
		*
		* @param cteTables All available CTEs.
		* <i>Replaced by an empty list, if NULL.</i>
		* @param columns All available columns.
		* <i>Replaced by an empty list, if NULL.</i>
		*/
		public CheckContext(final SearchTableApi cteTables, final SearchColumnList columns) {
		this.cteTables = (cteTables == null ? new SearchTableList() : cteTables);
		this.availableColumns = (columns == null ? new SearchColumnList() : columns);
		}

		/**
		* Create a deep copy of this context.
		*
		* @return Deep copy.
		*/
		public CheckContext getCopy() {
		return new CheckContext(cteTables.getCopy(), new SearchColumnList(availableColumns));
		}

		}

src/adql/db/DBChecker.java

+1042 −987

File changed.

Preview size limit exceeded, changes collapsed.

src/adql/db/DBColumn.java

+34 −6

Original line number	Diff line number	Diff line
		@@ -35,7 +35,20 @@ package adql.db;
		public interface DBColumn {

		/**
		* Gets the name of this column (without any prefix and double-quotes).
		* Gets the name of this column.
		*
		* <i>
		* <p><b>Notes:</b>
		* The returned ADQL name is:
		* </p>
		* <ul>
		* <li>non-empty/NULL</li>
		* <li>non-delimited (i.e. not between double quotes),</li>
		* <li>non-prefixed (i.e. no table/schema/catalog name)</li>
		* <li>and in the same case as provided at initialization (even if not case
		* sensitive).</li>
		* </ul>
		* </i>
		*
		* @return Its ADQL name.
		*/
		@@ -54,17 +67,31 @@ public interface DBColumn {
		public boolean isCaseSensitive();

		/**
		* Gets the name of this column in the "database".
		* Gets the name of this column in the "database" (e.g. as it should be used
		* in SQL queries).
		*
		* <i>
		* <p><b>Notes</b>
		* The returned DB name is:
		* </p>
		* <ul>
		* <li>non-empty/NULL</li>
		* <li>non-delimited (i.e. not between double quotes),</li>
		* <li>non-prefixed (i.e. no table/schema/catalog name)</li>
		* <li>and in the EXACT case as it MUST be used.</li>
		* </ul>
		*
		* @return Its DB name.
		*/
		public String getDBName();

		/**
		* <p>Get the type of this column (as closed as possible from the "database" type).</p>
		* Get the type of this column (as closed as possible from the "database"
		* type).
		*
		* <p><i>Note:
		* The returned type should be as closed as possible from a type listed by the IVOA in the TAP protocol description into the section UPLOAD.
		* <p><i><b>Note:</b>
		* The returned type should be as closed as possible from a type listed by
		* the IVOA in the TAP protocol description into the section UPLOAD.
		* </i></p>
		*
		* @return Its type.
		@@ -76,7 +103,8 @@ public interface DBColumn {
		/**
		* Gets the table which contains this {@link DBColumn}.
		*
		* @return Its table or <i>null</i> if no table is specified.
		* @return Its table
		* or NULL if no table is specified.
		*/
		public DBTable getTable();

src/adql/db/DBIdentifier.java

0 → 100644

+344 −0

Original line number	Diff line number	Diff line
		package adql.db;

		/*
		* This file is part of ADQLLibrary.
		*
		* ADQLLibrary is free software: you can redistribute it and/or modify
		* it under the terms of the GNU Lesser General Public License as published by
		* the Free Software Foundation, either version 3 of the License, or
		* (at your option) any later version.
		*
		* ADQLLibrary is distributed in the hope that it will be useful,
		* but WITHOUT ANY WARRANTY; without even the implied warranty of
		* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
		* GNU Lesser General Public License for more details.
		*
		* You should have received a copy of the GNU Lesser General Public License
		* along with ADQLLibrary. If not, see <http://www.gnu.org/licenses/>.
		*
		* Copyright 2019- UDS/Centre de Données astronomiques de Strasbourg (CDS)
		*/

		/**
		* Generic implementation of any kind of ADQL/DB identifier.
		*
		* <p>
		* It already implements functions getting and setting the ADQL and DB names
		* of the interfaces {@link DBTable} and {@link DBColumn}. Thus, it guarantees
		* that all DB... identifiers will behave the same way when manipulating their
		* ADQL and DB names.
		* </p>
		*
		* @author Grégory Mantelet (CDS)
		* @version 2.0 (09/2019)
		* @since 2.0
		*
		* @see DBTable
		* @see DBColumn
		*/
		public abstract class DBIdentifier {

		/** Regular expression of a delimited identifier (i.e. an identifier between
		* double quotes ; an inner double quote is escaped by doubling it). */
		private final static String REGEXP_DELIMITED = "\"(\"\"\|[^\"])*\"";

		/** Name (not delimited, not prefixed) to use in ADQL queries.
		* <p><i><b>Important:</b> It must never be NULL.</i></p> */
		protected String adqlName = null;

		/** A flag indicating if the ADQL name is case sensitive or not (i.e. if it
		* must be delimited or not in an ADQL query). */
		protected boolean adqlCaseSensitive = false;

		/** Name (not delimited, not prefixed) of this identifier in the "database".
		* This name must be used, for example, while translating an ADQL query into
		* SQL.
		* <p><i><b>Note:</b> It may be NULL. In such case, {@link #getDBName()}
		* must return {@link #adqlName}.</i></p> */
		protected String dbName = null;

		/**
		* Create an identifier with the given ADQL name.
		*
		* <p>
		* In this constructor, the DB name is not set. Thus, {@link #getDBName()}
		* will return the same as {@link #getADQLName()}.
		* </p>
		*
		* <p><i><b>Note:</b>
		* If the given name is delimited, the surrounding double quotes will be
		* removed and {@link #isCaseSensitive()} will return <code>true</code>.
		* </i></p>
		*
		* @param adqlName The ADQL and DB name of this identifier.
		* <i>It may be delimited and/or qualified.</i>
		*
		* @throws NullPointerException If the given name is NULL or empty.
		*
		* @see #setADQLName(String)
		*/
		protected DBIdentifier(final String adqlName) throws NullPointerException {
		setADQLName(adqlName);
		}

		/**
		* Create an identifier with the given ADQL and DB names.
		*
		* <p>
		* In this constructor, the DB name is not set. Thus, {@link #getDBName()}
		* will return the same as {@link #getADQLName()}.
		* </p>
		*
		* <p><i><b>Note:</b>
		* If the given name is delimited, the surrounding double quotes will be
		* removed and {@link #isCaseSensitive()} will return <code>true</code>.
		* </i></p>
		*
		* @param adqlName The ADQL and DB name of this identifier.
		* <i>It may be delimited and/or qualified.</i>
		*
		* @throws NullPointerException If the given name is NULL or empty.
		*
		* @see #setADQLName(String)
		* @see #setDBName(String)
		*/
		protected DBIdentifier(final String adqlName, final String dbName) throws NullPointerException {
		setADQLName(adqlName);
		setDBName(dbName);
		}

		/**
		* Get the ADQL version of this identifier.
		*
		* <p>
		* This name is neither delimited, nor prefixed.
		* To determine whether it should be delimited in an ADQL query, use
		* {@link #isCaseSensitive()}.
		* </p>
		*
		* <p><i><b>Note:</b>
		* The returned string is never empty or NULL.
		* </i></p>
		*
		* @return The name to use in ADQL queries.
		*/
		public String getADQLName() {
		return adqlName;
		}

		/**
		* Set the ADQL version of this identifier.
		*
		* <p>
		* If the given name is delimited, the surrounding double quotes will be
		* removed and case sensitivity will be set to <code>true</code>
		* (i.e. {@link #isCaseSensitive()} will return <code>true</code>).
		* </p>
		*
		* <p><i><b>Note:</b>
		* The given name must not be prefixed.
		* </i></p>
		*
		* <p><i><b>WARNING:</b>
		* If the given name is NULL or empty (even after removal of surrounding
		* double quotes, if delimited), this function will immediately throw an
		* exception.
		* </i></p>
		*
		* @param newName New ADQL version of this identifier.
		*
		* @throws NullPointerException If the given name is NULL or empty.
		*
		* @see #isDelimited(String)
		* @see #normalize(String)
		*/
		public void setADQLName(final String newName) throws NullPointerException {
		boolean adqlCS = isDelimited(newName);
		String normName = normalize(newName);

		if (normName == null)
		throw new NullPointerException("Missing ADQL name!");

		this.adqlName = normName;
		this.adqlCaseSensitive = adqlCS;
		}

		/**
		* Tell whether the ADQL version of this identifier is case sensitive or
		* not.
		*
		* <p>
		* If case sensitive, the ADQL name must be written between double quotes
		* (and all inner double quotes should be doubled).
		* </p>
		*
		* @return <code>true</code> if case sensitive,
		* <code>false</code> otherwise.
		*/
		public boolean isCaseSensitive() {
		return adqlCaseSensitive;
		}

		/**
		* Set the case sensitivity of the ADQL version of this identifier.
		*
		* <p>
		* Setting the case sensitivity to <code>true</code> will force the
		* delimited form of the ADQL name (i.e. it will be written between
		* double quotes).
		* </p>
		*
		* @param caseSensitive <code>true</code> to declare the ADQL name as case
		* sensitive,
		* <code>false</code> otherwise.
		*/
		public void setCaseSensitive(final boolean caseSensitive) {
		this.adqlCaseSensitive = caseSensitive;
		}

		/**
		* Get the database version of this identifier.
		*
		* <p>This name is neither delimited, nor prefixed.</p>
		*
		* <p>In an SQL query, this name should be considered as case sensitive.</p>
		*
		* <p><i><b>Note:</b>
		* The returned string is never empty or NULL.
		* </i></p>
		*
		* @return The real name of this identifier in the "database".
		*/
		public String getDBName() {
		return (dbName == null) ? getADQLName() : dbName;
		}

		/**
		* Set the database version of this identifier.
		*
		* <p>
		* If the given name is delimited, the surrounding double quotes will be
		* removed.
		* </p>
		*
		* <p><i><b>Note 1:</b>
		* The given name should not be prefixed.
		* </i></p>
		*
		* <p><i><b>Note 2:</b>
		* If the given name is NULL or empty (even after removal of surrounding
		* double quotes if delimited), {@link #getDBName()} will return the same
		* as {@link #getADQLName()}.
		* </i></p>
		*
		* @param newName The real name of this identifier in the "database".
		*
		* @see #normalize(String)
		*/
		public void setDBName(final String newName) {
		dbName = normalize(newName);
		}

		/**
		* Tell whether the given identifier is delimited (i.e. within the same pair
		* of double quotes - <code>"</code>).
		*
		* <i>
		* <p>The following identifiers ARE delimited:</p>
		* <ul>
		* <li><code>"a"</code></li>
		* <li><code>""</code> (empty string ; but won't be considered as a
		* valid ADQL name)</li>
		* <li><code>" "</code> (string with spaces ; but won't be considered as a
		* valid ADQL name)</li>
		* <li><code>"foo.bar"</code></li>
		* <li><code>"foo"".""bar"</code> (with escaped double quotes)</li>
		* <li><code>""""</code> (idem)</li>
		* </ul>
		* </i>
		*
		* <i>
		* <p>The following identifiers are NOT considered as delimited:</p>
		* <ul>
		* <li><code>"foo</code> (missing ending double quote)</li>
		* <li><code>foo"</code> (missing leading double quote)</li>
		* <li><code>"foo"."bar"</code> (not the same pair of double quotes)</li>
		* </ul>
		* </i>
		*
		* @param ident Identifier that may be delimited.
		*
		* @return <code>true</code> if the given identifier is delimited,
		* <code>false</code> otherwise.
		*/
		public static boolean isDelimited(final String ident) {
		return ident != null && ident.trim().matches(REGEXP_DELIMITED);
		}

		/**
		* Normalize the given identifier.
		*
		* <p>This function performs the following operations:</p>
		* <ol>
		* <li>Remove leading and trailing space characters.</li>
		* <li>If the resulting string is empty, return NULL.</li>
		* <li>If {@link #isDelimited(String) delimited}, remove the leading and
		* trailing double quotes.</li>
		* <li>If the resulting string without leading and trailing spaces is
		* empty, return NULL.</li>
		* <li>Return the resulting string.</li>
		* </ol>
		*
		* @param ident The identifier to normalize.
		*
		* @return The normalized string,
		* or NULL if NULL or empty.
		*
		* @see #denormalize(String, boolean)
		*/
		public static String normalize(final String ident) {
		// Return NULL if empty:
		if (ident == null \|\| ident.trim().length() == 0)
		return null;

		// Remove leading and trailing space characters:
		String normIdent = ident.trim();

		// If delimited, remove the leading and trailing ":
		if (isDelimited(normIdent)) {
		normIdent = normIdent.substring(1, normIdent.length() - 1).replaceAll("\"\"", "\"");
		return (normIdent.trim().length() == 0) ? null : normIdent;
		} else
		return normIdent;
		}

		/**
		* De-normalize the given string.
		*
		* <p>
		* This function does something only if the given string is declared as
		* case sensitive. In such case, it will surround it by double quotes.
		* All inner double quotes will be escaped by doubling them.
		* </p>
		*
		* <p><i><b>Note:</b>
		* If the given string is NULL, it will be returned as such (i.e. NULL).
		* </i></p>
		*
		* @param ident The identifier to de-normalize.
		* @param caseSensitive <code>true</code> if the given identifier is
		* considered as case sensitive,
		* <code>false</code> otherwise.
		*
		* @return The de-normalized identifier.
		*
		* @see #normalize(String)
		*/
		public static String denormalize(final String ident, final boolean caseSensitive) {
		if (caseSensitive && ident != null)
		return "\"" + ident.replaceAll("\"", "\"\"") + "\"";
		else
		return ident;
		}

		}