Data Connector for Apptopia
This Data Connector allows you to import Apptopia Data Source objects into Arm Treasure Data.
Prerequisites
- Basic knowledge of Treasure Data
- Basic knowledge of Apptopia
Use TD Console
Create a new connection
Go to Treasure Data Connections, search and select Apptopia.

Edit your Apptopia client and secret information, click Continue
and give your connection a name:

Click Create Connection
.
Create a new transfer
When you click Create Connection
, you are taken to My Connections tab. Look for the connection you created and click New Transfer
.

The following dialog opens. Edit the details and click Next
.

Preview your data. If you wish to change anything, click Advanced Settings
or click Next
.

In Advanced Settings
you can change some options such as rate limits:

Select the database and table where you want to transfer the data, as shown the following dialog:

Specify the schedule of the data transfer and click Start Transfer
:

You will see the new data transfer in progress, listed under the My Input Transfers
tab, and a corresponding job will be listed in the Jobs section.
Command Line
Install ‘td’ command v0.11.9 or later
You can install the newest Treasure Data Toolbelt.
$ td --version 0.15.0
Create Configuration File
Prepare configuration file (for eg: load.yml
) like below, with your Apptopia account access information.
in: type: apptopia client: xxxxxxxx secret: xxxxxxxx target: publisher_performance (required, see Appendix B) store: itunes_connect (required, see Appendix C) start_date: 2017-01-01 (required for all targets except `category` and `country`) end_date: 2017-02-01 (required for all targets except `category` and `country`) requests_per_minute_limit: 300 (optional, 1000 by default, see Appendix F) out: mode: replace
This example shows an Apptopia Publisher Performance
Data Source:
client
: Apptopia client id.secret
: Apptopia secret.target
: Apptopia entity object to be imported.- See Appendix B for the list of available
target
.
- See Appendix B for the list of available
store
: Desired mobile market to fetch data- See Appendix C for the list of available
store
.
- See Appendix C for the list of available
start_date
: From which date (yyyy-MM-dd
) product data to be imported. This field is required when fetching product usage (target
isproduct_usage
) or product sale (target
isproduct_sales
) with in-app-purchase breakdown (breakdown
hasiap
).end_date
: To which date (yyyy-MM-dd
) product data to be imported. This field is optional and is automatically adjusted to maximum 60 days sincestart_date
limited by current date.requests_per_minute_limit
: Limit number of API calls per minute- See Appendix D for how to use these options
For more details on available out
modes, see Appendix A
(optional): Preview data to import
You can preview data to be imported using the command td connector:preview
.
$ td connector:preview load.yml +------------+------------------+--------------------+---- | id:string | store:string | country_iso:string | ... +------------+------------------+--------------------+---- | 420233213 | "itunes_connect" | US | | jp.td.app | "google_play" | JP | +------------+------------------+--------------------+----
Execute Load Job
Submit the load job. It may take a couple of hours depending on the data size. Users need to specify the database and table where their data are stored.
It is recommended to specify --time-column
option, because Treasure Data’s storage is partitioned by time (see also data partitioning) If the option is not given, the Data Connector will choose the first long
or timestamp
column as the partitioning time. The type of the column specified by --time-column
must be either of long
and timestamp
type.
If your data doesn’t have a time column you can add it by using add_time
filter option. For more details, see add_time filter plugin.
$ td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column updated_date
The preceding command assumes that you already created database(td_sample_db) and table(td_sample_table). If the database or the table does not exist in TD, the command will not succeed, so create the database and table manually or use --auto-create-table
option with td connector:issue
command to auto create the database and table:
$ td connector:issue load.yml --database td_sample_db --table td_sample_table --time-column updated_date --auto-create-table
You can assign Time Format column to the "Partitioning Key" by "--time-column" option. |
Scheduled execution
You can schedule periodic Data Connector execution for periodic Apptopia import. We take great care in distributing and operating our scheduler in order to achieve high availability. By using this feature, you no longer need a cron
daemon on your local datacenter.
Create the schedule
A new schedule can be created using the td connector:create
command. The name of the schedule, cron-style schedule, the database and table where their data will be stored, and the Data Connector configuration file are required.
$ td connector:create \ daily_apptopia_import \ "10 0 * * *" \ td_sample_db \ td_sample_table \ load.yml
The `cron` parameter also accepts the options: `@hourly`, `@daily` and `@monthly`. |
By default, schedule is setup in UTC timezone. You can set the schedule in a timezone using -t or --timezone option. The `--timezone` option supports only extended timezone formats like 'Asia/Tokyo', 'America/Los_Angeles' etc. Timezone abbreviations like PST, CST are *not* supported and may lead to unexpected schedules. |
List the Schedules
You can see the list of currently scheduled entries by td connector:list
.
$ td connector:list +-----------------------+--------------+----------+-------+--------------+-----------------+-----------------------------+ | Name | Cron | Timezone | Delay | Database | Table | Config | +-----------------------+--------------+----------+-------+--------------+-----------------+-----------------------------+ | daily_apptopia_import | 10 0 * * * | UTC | 0 | td_sample_db | td_sample_table | {"type"=>"apptopia", ... } | +-----------------------+--------------+----------+-------+--------------+-----------------+-----------------------------+
Show the Setting and History of Schedules
td connector:show
shows the execution setting of a schedule entry.
% td connector:show daily_apptopia_import Name : daily_apptopia_import Cron : 10 0 * * * Timezone : UTC Delay : 0 Database : td_sample_db Table : td_sample_table
td connector:history
shows the execution history of a schedule entry. To investigate the results of each individual execution, use td job <jobid>
.
% td connector:history daily_apptopia_import +--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+ | JobID | Status | Records | Database | Table | Priority | Started | Duration | +--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+ | 578066 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-18 00:10:05 +0000 | 160 | | 577968 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-17 00:10:07 +0000 | 161 | | 577914 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-16 00:10:03 +0000 | 152 | | 577872 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-15 00:10:04 +0000 | 163 | | 577810 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-14 00:10:04 +0000 | 164 | | 577766 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-13 00:10:04 +0000 | 155 | | 577710 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-12 00:10:05 +0000 | 156 | | 577610 | success | 10000 | td_sample_db | td_sample_table | 0 | 2015-04-11 00:10:04 +0000 | 157 | +--------+---------+---------+--------------+-----------------+----------+---------------------------+----------+ 8 rows in set
Delete the Schedule
td connector:delete
removes the schedule.
$ td connector:delete daily_apptopia_import
Appendix
A) Modes for out
plugin
You can specify file import mode in out
section of load.yml.
append (default)
This is the default mode and records are appended to the target table.
in: ... out: mode: append
replace (In td 0.11.10 and later)
This mode replaces data in the target table. Any manual schema changes made to the target table remains intact with this mode.
in: ... out: mode: replace
B) Available targets
Target | Description |
---|---|
app_metadata | Application metadata |
app_performance | Application performance |
app_ranks | Application ranking |
app_sdks_metadata | Application SDKs metadata |
category | List of categories |
category_rank_lists | Raw ranks top charts for each category |
category_featured_lists | Featured applications for each category |
category_new_releases | New app releases for each category |
country | Supported countries |
publisher_metadata | Publisher metadata |
publisher_performance | Publisher performance |
sdk_metadata | SDK metadata |
C) Available markets
Store | Description |
---|---|
itunes_connect | Appple Store |
google_play | Google Play Market |
D) Rate Limits
There is requests per minute rate limit in Apptopia. This rate limit is auto refreshed after a certain number of seconds.
If you have multiple transfers under the same Apptopia account, you can control the rate limit usage of each transfer via requests_per_minute_limit
in advanced settings as long as the total limit does not exceed your account limit. For example, assume that your account has quotas as 1000 calls/minute
, if you create 2 transfers, e.g. app_performance & publisher_performance targets, you could use 600 rpm for app_performance transfer and the rest (400 rpm) for publisher_performance transfer.
Comments
0 comments
Please sign in to leave a comment.