Arm Treasure Data uses the same convention as Relational Database Management Systems (RDBMSs) for managing data sets: Databases and Tables. In addition, a Schema can be associated with tables.
Unlike traditional warehousing platforms, Treasure Data allows users to store-first, then schema-later. Schema can be changed at any time, with no cost.
- Basic knowledge of Treasure Data, including the toolbelt.
- A table with some data. You can find an example in the Quickstart: Query article.
Schema Changes Are Painful
Conventional warehousing platforms are schema dependent, supporting an assumptive analytics model. In this model, data elements forecasted to yield insights are defined in advance, as well as structure of the data store schema.
Performance considerations are also important in initial design and the analyst must have knowledge of the underlying structure to insure query performance. When new columns are added to the table, the schema must change.
Big Data analysis however, is largely non-assumptive. The analyst seeks hidden patterns, relationships or events in the data that were not obvious from the outset. The user must be able to query the data whereever it is stored and without the burden of performance considerations—and exploration can create requirements for new records to support the analysis trail.
In this model schema, dependence adds a significant tax that can become prohibitive.
The TD Approach: Store-First, Schema-Later
Unlike traditional warehousing platforms, TD users can assign schema even after importing data to a table. This means that you can add or remove fields at any time.
This system is much more flexible, and schema changes no longer take days of work.
Default Schema (time and v)
When a table is created, it has two fields:
- time: The time that each entry was generated, in int64 UNIX time
- v: Each log entry, stored as map<string, string>
When we look up the value of a database entry, we must address the information using the format: v[‘field1’].
Furthermore, data values can be of various data types, and must be identified first. This is inefficient, and now unrecommended except for testing.
$ td query -w -d testdb "SELECT v['user'], COUNT(1) AS cnt FROM www_access WHERE v['action']='login' GROUP BY v['user'] ORDER BY cnt DESC"
For daily uses, using custom schema is strongly recommended.
Setting Custom Schema
In general, the default schema is acceptable. But a custom schema makes queries shorter, and greatly improves performance. To set a schema to a table, we can use the ‘td schema:set’ command, as shown below:
$ td schema:set <database> <table> <column_name>:<type>...
|Schemas currently support column names consisting of lowercase alphabets, numbers, and "_" only.|
Treasure Data supports the following data types:
In the “Getting Started” guide, we set up a table with ‘user’ (int type) and ‘action’ (string type) columns. For this example, the schema would be added as follows.
$ td schema:set testdb www_access action:string labeles:'array<string>' user:int
You can now query this table with defined column names.
$ td query -w -d testdb "SELECT user, COUNT(1) AS cnt FROM www_access WHERE action='login' GROUP BY user ORDER BY cnt DESC"
When I query my table, I get “null” for some of my integer values. Is this a bug?
Is your integer greater than or equal to 2e64? If it is, then you need to cast it as
decimal like this:
SELECT CAST(my_huge_number AS DECIMAL) FROM tbl...
For queries, we leverage the Hive Query Language.
For queries, we leverage the Presto Query Language.
|Convert to string or int||boolean||boolean|
|string||varchar||string or varchar|
|string or Convert to long||date||string|
|string or Convert to long||timestamp||timestamp|
You can refer to the open source documentation as well: