Treasure Data uses the same convention as Relational Database Management Systems (RDBMSs) for managing data sets: Databases and Tables. In addition, a Schema can be associated with tables.
Unlike traditional warehousing platforms, Treasure Data allows users to store-first, then schema-later. Schema can be changed at any time, with no cost.
- Basic knowledge of Treasure Data, including the toolbelt.
- A table with some data. You can find an example in the Getting Started guide.
Schema Changes Are Painful
Conventional warehousing platforms are schema dependent, supporting an assumptive analytics model. In this model, data elements forecasted to yield insights are defined in advance, as well as structure of the data store schema.
Performance considerations are also important in initial design and the analyst must have knowledge of the underlying structure to insure query performance. When new columns are added to the table, the schema must change.
Big Data analysis however, is largely non-assumptive. The analyst seeks hidden patterns, relationships or events in the data that were not obvious from the outset. The user must be able to query the data whereever it is stored and without the burden of performance considerations—and exploration can create requirements for new records to support the analysis trail.
In this model schema, dependence adds a significant tax that can become prohibitive.
The TD Approach: Store-First, Schema-Later
Unlike traditional warehousing platforms, TD users can assign schema even after importing data to a table. This means that you can add or remove fields at any time.
This system is much more flexible, and schema changes no longer take days of work.
Default Schema (time and v)
When a table is created, it has two fields:
- time: The time that each entry was generated, in int64 UNIX time
- v: Each log entry, stored as map<string, string>
When we look up the value of a database entry, we must address the information using the format: v[‘field1’].
Furthermore, data values can be of various data types, and must be identified first. This is inefficient, and now unrecommended except for testing.
$ td query -w -d testdb "SELECT v['user'], COUNT(1) AS cnt FROM www_access WHERE v['action']='login' GROUP BY v['user'] ORDER BY cnt DESC"
For daily uses, using custom schema is strongly recommended.
Setting Custom Schema
In general, the default schema is acceptable. But a custom schema makes queries shorter, and greatly improves performance. To set a schema to a table, we can use the ‘td schema:set’ command, as shown below:
$ td schema:set <database> <table> <column_name>:<type>...
|Schemas currently support column names consisting of lowercase alphabets, numbers, and "_" only.|
Treasure Data supports the following data types:
In the “Getting Started” guide, we set up a table with ‘user’ (int type) and ‘action’ (string type) columns. For this example, the schema would be added as follows.
$ td schema:set testdb www_access action:string labeles:'array<string>' user:int
You can now query this table with defined column names.
$ td query -w -d testdb "SELECT user, COUNT(1) AS cnt FROM www_access WHERE action='login' GROUP BY user ORDER BY cnt DESC"
For queries, we leverage the Hive Query Language.
For queries, we leverage the Presto Query Language.
|double||double||decimal(Hive 0.11.0 and later)|
|Convert to string or int||boolean||boolean|
|string||varchar||string or varchar (Hive 0.11.0 and later)|
|string or Convert to long||date||string|
|string or Convert to long||timestamp||timestamp|
You can refer to the open source documentation as well: