/7.1/aws/concepts/Recent content in Concepts on Kinetica DocsHugo -- gohugo.ioen/7.1/aws/concepts/aggregation/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/aggregation/Execution Pivot Unpivot Memory-Only Tables and Persistence Sharding and Replication Limitations General Aggregate Memory-Only Tables Kinetica is able to aggregate data via the native APIs and SQL using various aggregate expressions. The native API endpoints related to aggregation: /aggregate/convexhull /aggregate/groupby /aggregate/histogram /aggregate/kmeans /aggregate/minmax /aggregate/minmax/geometry /aggregate/statistics /aggregate/statistics/byrange /aggregate/unique /aggregate/unpivot To read more about aggregation in SQL, see Aggregation. Execution In SQL, to find the minimum, maximum, & average trip distances, as well as the average passenger count for each vendor per year from the taxi data set (weeding out data with errant trip distances):/7.1/aws/concepts/credentials/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/credentials/Managing Credentials Creating a Credential Provider-Specific Syntax Altering a Credential Removing a Credential Showing a Credential Updating Credential Permissions A credential is an authentication object for a resource or repository that is external to the database. It consists of the credentials used to authenticate to that external system, with the goal of providing an authentication token to any database object that may need to connect to that system. A credential name must adhere to the standard naming criteria./7.1/aws/concepts/cube/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/cube/The cube operation takes n number of columns and produces 2n aggregates. For example, given a cube on columns A, B, & C, it computes the requested aggregates for the following combinations of columns: {ABC} - unique A B C triplets {AB } - unique A B pairs {A C} - unique A C pairs { BC} - unique B C pairs {A  } - unique A values { B } - unique B values {  C} - unique C values {   } - all values The cube operation is an aggregate function that can be invoked natively in the options map of the /aggregate/groupby endpoint./7.1/aws/concepts/data_sinks/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/data_sinks/Managing Data Sinks Creating a Data Sink Provider-Specific Syntax A data sink is reference object for a data target that is external to the database. It consists of the location & connection information to that external target. A data sink can make use of a credential object for storing remote authentication information. A data sink name must adhere to the standard naming criteria. Each data sink exists within a schema and follows the standard name resolution rules for tables./7.1/aws/concepts/data_sources/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/data_sources/Managing Data Sources Creating a Data Source Provider-Specific Syntax Limitations A data source is reference object for a data set that is external to the database. It consists of the location & connection information to that external source, but doesn't hold the names of any specific data sets/files within that source. A data source can make use of a credential object for storing remote authentication information. A data source name must adhere to the standard naming criteria./7.1/aws/concepts/dictionary_encoding/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/dictionary_encoding/Dictionary encoding is a data compression technique that can be applied to individual columns of the following effective types: int long date char1 - char256 It will store each unique value of a column in memory and associate each record with its corresponding unique value. This eliminates the storage of duplicate values in a column, reducing the overall memory & disk space required to hold the data. Dictionary encoding is most effective on columns with low cardinality; the fewer the number of unique values within a column, the greater the reduction in memory usage./7.1/aws/concepts/except/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/except/An except is a representation of all unique rows in one data set (table or view) that do not appear in another. See EXCEPT for the SQL version of this operation. An except is performed via the /create/union endpoint using the except or except_all mode: Except -- all unique rows that exist in one data set, but not the other Except All -- all rows (including duplicates) that exist in one data set, but not the other You can only perform an except two data sets, and the columns between the two must have similar data types./7.1/aws/concepts/expressions/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/expressions/Kinetica has native API support for a variety of expressions, which are used as inputs while querying data (for supported SQL expressions, see Queries (SQL) ). These native API expressions can involve one or more constants (both numeric and string) and table columns; however, expressions cannot be applied to store-only columns. The expressions follow certain constraints based on where they are used, but all the expressions should follow the basic guidelines outlined below./7.1/aws/concepts/external_tables/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/external_tables/Types Data File Formats Table Features Data Sources Ingestion Mode Refresh on Start Error Mode Creating an External Table An external table is a database object whose source data is located in one or more files external to the database. The source data can be located in either of the following locations: in KiFS on a remote system, accessible via a data source External tables are created via the /create/table/external native API call./7.1/aws/concepts/filtered_views/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/filtered_views/A filtered view is a queryable filter of a table or another view. A filtered view can be created by filtering an existing table or view using the /filter endpoint. Filtered views can be updated to insert the missing data points for a series from the underlying table using /update/records/byseries. Given the need to minimize memory usage, filtered views are given a default time-to-live, after which they will expire and be removed from memory./7.1/aws/concepts/full_text_search/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/full_text_search/The SQL FILTER_BY_STRING function, as well as the /filter/bystring endpoint, enables the use of several string-based search modes, including full text search via the search mode. The syntax and restrictions for this search follow, including examples of applying different search criteria. In order for a full text search to be applied to a column, it must be a string type and have the text_search column property applied. The following effective types under the string base type are text-searchable:/7.1/aws/concepts/grouping_sets/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/grouping_sets/The grouping sets operation calculates aggregates over any user-specified dimensions. It offers much more flexibility and customizability over the Cube and Rollup functions, while allowing both as possible dimensions over which aggregation can be performed. For example, given grouping sets of B, (D,E), & (ROLLUP(A,C)), it computes the requested aggregates for the following combinations of columns: Group B: { B   } - unique B values Group (D,E): {   DE} - unique D E pairs Group ROLLUP(A,C): {A C  } - unique A C pairs {A    } - unique A values {     } - all values The grouping sets operation is an aggregate function that can be invoked natively in the options map of the /aggregate/groupby endpoint./7.1/aws/concepts/indexes/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/indexes/An index is used to improve the performance of data access within the system. There are several types of indexes: Primary Key Index Relational Index Column Index Chunk Skip Index Geospatial Index Primary Key Index A primary key index is created by default when a table is created with a primary key specified. It will match the primary key in nature, in that it will be a single-column index if the primary key is on a single column and a multi-column composite index if the primary key is a composite primary key./7.1/aws/concepts/intersect/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/intersect/An intersect is a representation of all rows that appear in both of a pair of specified data sets (tables or views). See INTERSECT for the SQL version of this operation. An intersect is performed via the /create/union endpoint using the intersect or intersect_all mode: Intersect -- all unique rows that exist in both specified data sets Intersect All -- all rows (including duplicates) that exist in both specified data sets You can only perform an intersect two data sets, and the columns between the two must have similar data types./7.1/aws/concepts/iteration/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/iteration/Overview Examples Geospatial Point Extraction - Variable Iteration (in SQL) Line Segment Extraction - Variable Iteration (in Python) Endpoint Extraction - Fixed Iteration (in SQL) Non-Geospatial Letter Counting - Variable Iteration (in SQL) Data Duplication - Fixed Iteration (in Python) Limitations Overview Kinetica supports arbitrary iteration over records within a data set for the purpose of generating 0 to N result records per iterated record. The primary use case for this is in returning sub-elements of a given geospatial column as individual records; e./7.1/aws/concepts/joins/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/joins/Join Execution Types Native Joins SQL Joins Join Distribution Types Node-Local Joins Distributed Joins Equality-Based Join Rules Inequality-Based Join Rules Creating a Join View Examples Inner Join Outer Join Operating on a Join View Examples SQL Conversion Example Performance Optimization Limitations & Cautions Memory Implications Operations on Underlying Tables Kinetica supports the SQL concept of joining data sets, via join views, connecting related records between two or more tables./7.1/aws/concepts/materialized_views/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/materialized_views/Refresh Manual Refresh Manual Mode On Change Mode On Query Mode Periodic Mode Creating a Materialized View Examples Limitations and Cautions Kinetica supports the concept of a materialized view, which is a greater-performant query across one or more tables. For details on interfacing with materialized views from SQL, see CREATE MATERIALIZED VIEW. A materialized view name must adhere to the standard naming criteria. Each materialized view exists within a schema and follows the standard name resolution rules for tables./7.1/aws/concepts/tables_memory_only/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/tables_memory_only/Memory-only tables are not persisted by default like a regular table. You can add, modify, or delete records in a memory-only table. A memory-only table name must adhere to the standard naming criteria. Each memory-only table exists within a schema and follows the standard name resolution rules for persisted tables. The following scenarios/endpoints will create a memory-only table (unless persistence is specified): Using the /aggregate/groupby, /aggregate/unique, or /aggregate/unpivot endpoints with the result_table option specified Projection -- created from /create/projection Union (also includes intersect and except operations) -- created from /create/union Creating a table using /create/table with the is_result_table option set to true Considerations Ingestion speed is better than persisted tables, as there are no disk writes./7.1/aws/concepts/merge_records/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/merge_records/Merging Records is similar to a union in that multiple tables or views are merged together to create a new merge records table. Since a table is created by the merge records operation, records can be added, modified, or deleted. If you want to duplicate the records in a table or view, merging a table or view with itself is also possible, but the data set will need to be listed twice in the source_table_names parameter./7.1/aws/concepts/nulls/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/nulls/Setting Nullability SQL REST C++ C# Java JavaScript Node.js Python Using Nulls Examples Aggregates and Nulls Example Using nulls begins with setting nullability of type fields at creation time. Each database API has a language-specific means for assigning nullability when creating a type. Null values and nullability are also supported for SQL. Setting Nullability Setting nullability is possible using three different methods: Workbench Native API (using the /create/type endpoint) SQL The process for specifying nullability varies between the different API languages and SQL; the process for each language is outlined below./7.1/aws/concepts/pivot/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/pivot/The pivot operation works much the same as the SQL concept: "rotating" and expanding single columns of values into multiple columns (one for each value), creating a wider but shorter table. It is nearly the opposite of the unpivot operation, though it will merge duplicates and drop null values, which unpivot cannot undo. You can perform pivot operations using the pivot and pivot_values parameters in the options map of /aggregate/groupby or via the SQL PIVOT function./7.1/aws/concepts/projections/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/projections/A projection is a concept unique to Kinetica. A projection is created through the /create/projection endpoint and represents a set of columns from and/or column expressions applied to a source data set (table or view). Projections are memory-only tables by default, but can be persisted (like a table) using the persist option. While projections can simply represent a whole-cloth cross-section of an existing table, filtering clauses can be applied before creating the column to create an entirely new data set./7.1/aws/concepts/rollup/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/rollup/The rollup operation takes n number of columns and produces n + 1 aggregates. For example, given a rollup on columns A, B, & C, it computes the requested aggregates for the following combinations of columns: {ABC} - unique A B C triplets {AB } - unique A B pairs {A  } - unique A values {   } - all values The rollup operation is an aggregate function that can be invoked natively in the options map of the /aggregate/groupby endpoint./7.1/aws/concepts/schemas/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/schemas/A schema is a container for all database objects. It provides namespacing for several categories of those contained database objects: Tables ( tables, external tables, views ) Credentials Data sinks Data sources Graphs Memory-only tables created by various operations Table monitors Namespacing occurs at the schema level and at the category level, meaning that: One schema could contain a table with the same name as a table in another schema./7.1/aws/concepts/table_monitors/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/table_monitors/Overview Configuration Local ZMQ External Consumer Managing Table Monitors Limitations & Cautions Example Overview A table monitor (SQL Stream) is an entity that can be created to watch a target table for inserts, updates, or deletes, and relay notifications for consumption by interested clients to one of the following targets: a local ZMQ message queue (default) an external Apache Kafka broker an external webhook A table monitor is bound to a single type of table operation to monitor, assigned when created, and cannot be changed to monitor other types afterwards./7.1/aws/concepts/tables/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/tables/Overview Name Resolution Naming Criteria Example Distribution Sharding Random Sharding Replication Partitioning Range Interval List Hash Series Primary Keys Purpose Designation Relation to Shard Key Relation to Foreign Key Index Creation Shard Keys Purpose Designation Relation to Primary Key Creation Foreign Keys Purpose Designation Relation to Primary Key Relation to Shard Key Creation Overview The concept of tables is at the heart of Kinetica interactions. A table is a data container associated with a specific type (set of columns & properties), much like tables in other database platforms./7.1/aws/concepts/ttl/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/ttl/Time-to-live (TTL) is a configurable number of minutes since last access before a table or view will expire and be removed from the database. A TTL of -1 will direct that an entity not expire. If a table or view is assigned a TTL, its expiration timer will be reset to the value of its configured TTL each time it is accessed. For instance, a view with a five minute TTL will have its life extended by five minutes from the time it is accessed every time it is accessed./7.1/aws/concepts/types/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/types/Type Label Type Definition Column Types Column Properties Data Types Data Handling Data Keys Data Replacement A type is analogous to a traditional database definition for a table. Before data can be stored in Kinetica, a type must be specified for that data. For every type, Kinetica assigns a unique GUID. Kinetica will use the same GUID for all types with identical characteristics. Every type in Kinetica consists of the following:/7.1/aws/concepts/unions/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/unions/Kinetica supports the concept of set union, which is a representation of all rows that appear across a group of specified data sets (tables or views). See UNION for the SQL version of this operation. A union is performed using the /create/union endpoint, using one of the following modes: Union -- retains all unique rows from the specified data sets Union Distinct -- alias for Union Union All -- retains all rows from the specified data sets Merge Views -- merge two or more filtered views (or filtered views of filtered views) of the same base data set into a new view./7.1/aws/concepts/unpivot/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/unpivot/The unpivot operation works much the same as the SQL concept: "rotating" a set of column headers (and the associated values) into rows to create a more normalized table that's longer and narrower. It is nearly the opposite of the pivot operation, though it will not be able to unmerge records and restore nulls that were aggregated by the pivot operation. You can perform unpivot operations using /aggregate/unpivot or via the SQL UNPIVOT function./7.1/aws/concepts/views/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/views/ Views can be organized into three categories: Filtered Views -- created from: /filter /filter/byarea /filter/bybox /filter/bygeometry /filter/bylist /filter/byradius /filter/byrange /filter/byseries /filter/bystring /filter/bytable /filter/byvalue Join views -- created from: /create/jointable Materialized Views -- created from: /create/materializedview/7.1/aws/concepts/window/Mon, 01 Jan 0001 00:00:00 +0000/7.1/aws/concepts/window/Function Details Function Type Aggregate Ranking Window Specification Frame Clauses Range Frames Rows Frames Frame Bounds Examples Limitations Kinetica supports the SQL concept of windows. It can do so through the /create/projection and /get/records/bycolumn endpoints. The window function is specified as a column name in the column_name parameter. A window projection can be created from any table or view. If the source data set is replicated, the resulting window projection will also be replicated.