/7.2/concepts/Recent content in Concepts on Kinetica DocsHugo -- gohugo.ioen/7.2/concepts/aggregation/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/aggregation/Execution Pivot Unpivot Memory-Only Tables and Persistence Sharding and Replication Limitations General Aggregate Memory-Only Tables Kinetica is able to aggregate data via the native APIs and SQL using various aggregate expressions. The native API endpoints related to aggregation: /aggregate/convexhull /aggregate/groupby /aggregate/histogram /aggregate/kmeans /aggregate/minmax /aggregate/minmax/geometry /aggregate/statistics /aggregate/statistics/byrange /aggregate/unique /aggregate/unpivot To read more about aggregation in SQL, see Aggregation. Execution In SQL, to find the minimum, maximum, & average trip distances, as well as the average passenger count for each vendor per year from the taxi data set (weeding out data with errant trip distances):/7.2/concepts/array/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/array/Array Type Create Table Insert Data Retrieve Data Array Functions Scalar Functions Examples ARRAY_ITEM ARRAY_LENGTH ARRAY_TO_STRING STRING_TO_ARRAY Aggregation/Transposition Functions Examples ARRAY_AGG ARRAY_AGG_DISTINCT UNNEST_JSON_ARRAY Examples Array Type The array data type provides the capability to store a list of primitive data type values in a single column. All arrays in Kinetica are one-dimensional and 1-based. They are also unbounded, though the number of items in the array can be specified upon table/column creation for compatibility with other databases./7.2/concepts/collections/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/collections/Collections are the precursors to schemas, introduced in Kinetica v7.1. Any collections in prior versions will become schemas after upgrade, and any tables & views not contained within collections will be placed in the home schema. Operations on collections will no longer be supported after upgrade. See Schemas for details./7.2/concepts/column_compression/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/column_compression/Column compression is the application of a standard compression algorithm to a persisted column's data. Column data will be compressed on disk for faster loading into memory at the cost of the decompression processing time on that loaded data. Once in memory, the data will remain uncompressed until it is no longer needed. Any inserting or updating of data in memory would result in column data being compressed and written to disk depending on which of these conditions is reached first:/7.2/concepts/credentials/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/credentials/Managing Credentials Creating a Credential Provider-Specific Syntax Altering a Credential Removing a Credential Showing a Credential Updating Credential Permissions A credential is an authentication object for a resource or repository that is external to the database. It consists of the credentials used to authenticate to that external system, with the goal of providing an authentication token to any database object that may need to connect to that system. A credential name must adhere to the standard naming criteria./7.2/concepts/cube/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/cube/The cube operation takes n number of columns and produces 2n aggregates. For example, given a cube on columns A, B, & C, it computes the requested aggregates for the following combinations of columns: {ABC} - unique A B C triplets {AB } - unique A B pairs {A C} - unique A C pairs { BC} - unique B C pairs {A  } - unique A values { B } - unique B values {  C} - unique C values {   } - all values The cube operation is an aggregate function that can be invoked natively in the options map of the /aggregate/groupby endpoint./7.2/concepts/data_sinks/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/data_sinks/Managing Data Sinks Creating a Data Sink Consumer-Specific Syntax A data sink is reference object for a data target that is generally external to the database. It consists of the location & connection information to that target. A data sink can make use of a credential object for storing remote authentication information. A data sink name must adhere to the standard naming criteria. Each data sink exists within a schema and follows the standard name resolution rules for tables./7.2/concepts/data_sources/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/data_sources/Managing Data Sources Creating a Data Source Provider-Specific Syntax Limitations A data source is reference object for a data set that is external to the database. It consists of the location & connection information to that external source, but doesn't hold the names of any specific data sets/files within that source. A data source can make use of a credential object for storing remote authentication information. A data source name must adhere to the standard naming criteria./7.2/concepts/dictionary_encoding/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/dictionary_encoding/Dictionary encoding is a data compression technique that can be applied to individual columns. It will store each unique value of a column in memory and associate each record with its corresponding unique value. This eliminates the storage of duplicate values in a column, reducing the overall memory & disk space required to hold the data. Dictionary encoding is most effective on columns with low cardinality; the fewer the number of unique values within a column, the greater the reduction in memory usage./7.2/concepts/except/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/except/An except is a representation of all rows in one data set (table or view) that do not appear in another. Note An except is somewhat analogous to creating a table from a SQL EXCEPT of two tables. See CREATE TABLE ... AS and EXCEPT for details. An except is performed via the /create/union endpoint, using the except or except_all mode: Except -- all unique rows that exist in one data set, but not the other Except All -- all rows (including duplicates) that exist in one data set, but not the other Note/7.2/concepts/expressions/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/expressions/Kinetica has native API support for a variety of expressions, which are used as inputs while querying data (for supported SQL expressions, see Queries (SQL) ). These native API expressions can involve one or more constants (both numeric and string) and table columns. The expressions follow certain constraints based on where they are used, but all the expressions should follow the basic guidelines outlined below. Important Use parentheses liberally to ensure correct order-of-operations./7.2/concepts/external_tables/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/external_tables/Types Data File Formats Table Features Data Sources Ingestion Mode Refresh on Start Error Mode Creating an External Table An external table is a database object whose source data is located in one or more files external to the database. The source data can be located in either of the following locations: in KiFS on a remote system, accessible via a data source External tables are created via the /create/table/external native API call./7.2/concepts/filtered_views/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/filtered_views/A filtered view is a queryable filter of a table or another view. A filtered view can be created by filtering an existing table or view using the /filter endpoint. Note For SQL views, see CREATE VIEW. Given the need to minimize memory usage, filtered views are given a default time-to-live, after which they will expire and be removed from memory. Each access of the filtered view causes the remaining time-to-live to be reset./7.2/concepts/full_text_search/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/full_text_search/The SQL FILTER_BY_STRING function, as well as the /filter/bystring endpoint, enables the use of several string-based search modes, including full text search via the search mode. The syntax and restrictions for this search follow, including examples of applying different search criteria. In order for a full text search to be applied to a column, the following requirements must be met: Full text search must be enabled by setting enable_text_search to true in the /opt/gpudb/core/etc/gpudb./7.2/concepts/grouping_sets/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/grouping_sets/The grouping sets operation calculates aggregates over any user-specified dimensions. It offers much more flexibility and customizability over the Cube and Rollup functions, while allowing both as possible dimensions over which aggregation can be performed. For example, given grouping sets of B, (D,E), & (ROLLUP(A,C)), it computes the requested aggregates for the following combinations of columns: Group B: { B   } - unique B values Group (D,E): {   DE} - unique D E pairs Group ROLLUP(A,C): {A C  } - unique A C pairs {A    } - unique A values {     } - all values The grouping sets operation is an aggregate function that can be invoked natively in the options map of the /aggregate/groupby endpoint./7.2/concepts/indexes/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/indexes/An index is used to improve the performance of data access within the system. There are several types of indexes: Primary Key Index Relational Index Column Index Low-Cardinality Index Chunk Skip Index Geospatial Index CAGRA Index HNSW Index Primary Key Index A primary key index is created by default when a table is created with a primary key specified. It will match the primary key in nature, in that it will be a single-column index if the primary key is on a single column and a multi-column composite index if the primary key is a composite primary key./7.2/concepts/intersect/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/intersect/An intersect is a representation of all rows that appear in both of a pair of specified data sets (tables or views). Note An intersect is somewhat analogous to creating a table from a SQL INTERSECT of two tables. See CREATE TABLE ... AS and INTERSECT for details. An intersect is performed via the /create/union endpoint, using the intersect or intersect_all mode: Intersect -- all unique rows that exist in both specified data sets Intersect All -- all rows (including duplicates) that exist in both specified data sets Note/7.2/concepts/iteration/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/iteration/Overview Examples Geospatial Point Extraction - Variable Iteration Line Segment Extraction - Variable Iteration Endpoint Extraction - Fixed Iteration Non-Geospatial Letter Counting - Variable Iteration Data Duplication - Fixed Iteration Limitations Overview Kinetica supports arbitrary iteration over records within a data set for the purpose of generating 0 to N result records per iterated record. The primary use case for this is in returning sub-elements of a given geospatial column as individual records; e./7.2/concepts/jdbc_drivers/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/jdbc_drivers/Database Example Connection String Amazon Athena jdbc:awsathena://AwsRegion=us-east-1;S3OutputLocation=s3://mybucket/queryoutput/; Amazon Redshift jdbc:redshift://localhost:5439/mydb Amazon DSQL jdbc:postgresql://localhost:5432/mydb Apache Derby jdbc:derby://localhost:1527/mydb;create=true Apache Druid jdbc:avatica:remote:url=http://localhost:8082/druid/v2/sql/avatica/ Apache Hive jdbc:hive2://localhost:10000/default Apache Ignite jdbc:ignite:thin://localhost:10800 Apache Pinot jdbc:pinot://<HOST>:8099/query?controller=http://<CONTROLLER>:9000 ClickHouse jdbc:clickhouse://localhost:8123/mydb CockroachDB jdbc:postgresql://localhost:26257/defaultdb?sslmode=require DuckDB jdbc:duckdb:/path/to/mydb.duckdb Elasticsearch (SQL) jdbc:es://localhost:9200 Exasol jdbc:exa:localhost:8563;schema=MYDB Firebird (Jaybird) jdbc:firebirdsql://localhost:3050/path/to/mydb.fdb Google BigQuery jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=myproject;OAuthType=3; Google Cloud Spanner jdbc:cloudspanner:/projects/<PROJECT_ID>/instances/<INSTANCE_ID>/databases/<DATABASE_ID> Greenplum jdbc:greenplum://localhost:5432;DatabaseName=mydb H2 jdbc:h2:~/mydb HSQLDB jdbc:hsqldb:hsql://localhost/mydb IBM Db2 jdbc:db2://localhost:50000/mydb Informix jdbc:informix-sqli://localhost:1526/mydb:INFORMIXSERVER=server MariaDB jdbc:mariadb://localhost:3306/mydb Microsoft SQL Server jdbc:sqlserver://localhost:1433;databaseName=mydb MongoDB (Atlas SQL) jdbc:mongodb://localhost:27017/mydb?/7.2/concepts/joins/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/joins/Join Execution Types Native Joins SQL Joins Join Distribution Types Node-Local Joins Distributed Joins Equality-Based Join Rules Inequality-Based Join Rules Creating a Join View Examples Inner Join Outer Join Operating on a Join View Examples SQL Conversion Example Performance Optimization Limitations & Cautions Memory Implications Operations on Underlying Tables Kinetica supports the SQL concept of joining data sets, via join views, connecting related records between two or more tables./7.2/concepts/json/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/json/JSON Type Create Table Insert Data Retrieve Data JSON Operators Example JSON Functions Scalar Functions Aggregation Functions Examples JSON_ARRAYAGG JSON_ARRAYAGG_DISTINCT UNNEST_JSON_ARRAY Examples UNNEST_JSON_OBJECT Examples JSON Type The JSON data type provides the capability to store native JSON in a single column and apply JSON-specific functions to it. Create Table SQL Python SQL 1 CREATE TABLE example.js ( j JSON ) Python 1 2 3 4 5 6 table_name = 'example./7.2/concepts/materialized_views/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/materialized_views/Refresh Manual Refresh Manual Mode On Change Mode On Query Mode Periodic Mode Creating a Materialized View Examples Limitations and Cautions Kinetica supports the concept of a materialized view, which is a greater-performant query across one or more tables. For details on interfacing with materialized views from SQL, see CREATE MATERIALIZED VIEW. A materialized view name must adhere to the standard naming criteria. Each materialized view exists within a schema and follows the standard name resolution rules for tables./7.2/concepts/tables_memory_only/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/tables_memory_only/Memory-only tables are not persisted by default like a regular table. You can add, modify, or delete records in a memory-only table. A memory-only table name must adhere to the standard naming criteria. Each memory-only table exists within a schema and follows the standard name resolution rules for persisted tables. The following scenarios/endpoints will create a memory-only table (unless persistence is specified): Using the /aggregate/groupby, /aggregate/unique, or /aggregate/unpivot endpoints with the result_table option specified Projection -- created from /create/projection Union (also includes intersect and except operations) -- created from /create/union Creating a table using /create/table with the is_result_table option set to true Considerations Ingestion speed is better than persisted tables, as there are no disk writes./7.2/concepts/nulls/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/nulls/Setting Nullability SQL REST C++ C# Java JavaScript Node.js Python Using Nulls Examples Aggregates and Nulls Example Using nulls begins with setting nullability of type fields at creation time. Each database API has a language-specific means for assigning nullability when creating a type. Null values and nullability are also supported for SQL. Setting Nullability Setting nullability is possible using three different methods: GAdmin Native API (using the /create/type endpoint) SQL The process for specifying nullability varies between the different API languages and SQL; the process for each language is outlined below./7.2/concepts/pivot/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/pivot/The pivot operation works much the same as the SQL concept: "rotating" and expanding single columns of values into multiple columns (one for each value), creating a wider but shorter table. It is nearly the opposite of the unpivot operation, though it will merge duplicates and drop null values, which unpivot cannot undo. You can perform pivot operations using the pivot and pivot_values parameters in the options map of /aggregate/groupby or via the SQL PIVOT function./7.2/concepts/projections/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/projections/A projection is a concept unique to Kinetica. A projection is created through the /create/projection endpoint and represents a set of columns from and/or column expressions applied to a source data set (table or view). Projections are memory-only tables by default, but can be persisted (like a table) using the persist option. While projections can simply represent a whole-cloth cross-section of an existing table, filtering clauses can be applied before creating the column to create an entirely new data set./7.2/concepts/rollup/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/rollup/The rollup operation takes n number of columns and produces n + 1 aggregates. For example, given a rollup on columns A, B, & C, it computes the requested aggregates for the following combinations of columns: {ABC} - unique A B C triplets {AB } - unique A B pairs {A  } - unique A values {   } - all values The rollup operation is an aggregate function that can be invoked natively in the options map of the /aggregate/groupby endpoint./7.2/concepts/schemas/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/schemas/A schema is a container for all database objects, providing namespacing for them. Namespacing occurs at the schema level and at the category level, meaning that: One schema could contain a table with the same name as a table in another schema. One schema could contain a data source and data sink with the same name. One schema cannot contain an external table and a view with the same name./7.2/concepts/table_monitors/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/table_monitors/Overview Configuration Local ZMQ Local Table External Consumer Managing Table Monitors Limitations & Cautions Example Overview A table monitor (SQL Stream) is an entity that can be created to watch a source table for inserts, updates, or deletes, and relay notifications for consumption by interested clients to one of the following targets: a local ZMQ message queue (default) a local database table an external Apache Kafka broker an external webhook A table monitor is bound to a single type of table operation to monitor, assigned when created, and cannot be changed to monitor other types afterwards./7.2/concepts/tables/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/tables/Overview Name Resolution Naming Criteria Example Distribution Sharding Random Sharding Replication Partitioning Range Interval List Hash Series Primary Keys Purpose Designation Relation to Shard Key Relation to Foreign Key Index Creation Soft Primary Keys Purpose Designation Relation to Shard Key Relation to Foreign Key Index Creation Shard Keys Purpose Designation Relation to Primary Key Creation Foreign Keys Purpose Designation Relation to Primary Key Relation to Shard Key Creation Overview The concept of tables is at the heart of Kinetica interactions./7.2/concepts/ttl/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/ttl/Time-to-live (TTL) is a configurable number of minutes since last access before a table or view will expire and be removed from the database. A TTL of -1 will direct that an entity not expire. If a table or view is assigned a TTL, its expiration timer will be reset to the value of its configured TTL each time it is accessed. For instance, a view with a five minute TTL will have its life extended by five minutes from the time it is accessed every time it is accessed./7.2/concepts/types/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/types/Type Label Type Definition Column Types Column Properties Data Types Data Handling Data Keys Data Replacement A type is analogous to a traditional database definition for a table. Before data can be stored in Kinetica, a type must be specified for that data. For every type, Kinetica assigns a unique GUID. Kinetica will use the same GUID for all types with identical characteristics. Every type in Kinetica consists of the following:/7.2/concepts/unions/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/unions/A union is a representation of all rows that appear in any of a set of specified data sets (tables or views). Note A union is somewhat analogous to creating a table from a SQL UNION of two or more tables. See CREATE TABLE ... AS and UNION for details. A union is performed via the /create/union endpoint, using the union or union_all mode: Union -- retains all unique rows from the specified data sets Union All -- retains all rows from the specified data sets Note/7.2/concepts/unpivot/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/unpivot/The unpivot operation works much the same as the SQL concept: "rotating" a set of column headers (and the associated values) into rows to create a more normalized table that's longer and narrower. It is nearly the opposite of the pivot operation, though it will not be able to unmerge records and restore nulls that were aggregated by the pivot operation. You can perform unpivot operations using /aggregate/unpivot or via the SQL UNPIVOT function./7.2/concepts/views/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/views/ Views can be organized into three categories: Filtered Views -- created from: /filter /filter/byarea /filter/bybox /filter/bygeometry /filter/bylist /filter/byradius /filter/byrange /filter/byseries /filter/bystring /filter/bytable /filter/byvalue Join views -- created from: /create/jointable Materialized Views -- created from: /create/materializedview/7.2/concepts/window/Mon, 01 Jan 0001 00:00:00 +0000/7.2/concepts/window/Function Details Function Type Aggregate Ranking Window Specification Frame Clauses Range Frames Rows Frames Frame Bounds Examples Limitations Kinetica supports the SQL concept of windows. It can do so through the /create/projection and /get/records/bycolumn endpoints. The window function is specified as a column name in the column_name parameter. Note For performing window functions in SQL, see Window. A window projection can be created from any table or view. If the source data set is replicated, the resulting window projection will also be replicated.