This is a summary of new Arm Treasure Data features and improvements introduced in the February 1st, 2020 release. If you have any product feature requests, submit them to feedback.treasuredata.com.
Optionally, you can view a video summary of our February releases:
Workflow Utilization Dashboard
The Workflow Utilization dashboard shows your recent and historical use of workflows in Treasure Data. You can see session success rates for user-defined workflows, including custom script workflows. Use the dashboard to view the number of workflow run attempts and duration of workflow runs. You can also monitor your resources by viewing run hours and remaining resource hours for custom scripts.
The Workflow Utilization dashboard is available. Contact your Customer Success Representative for feature access.
With the Workflow Utilization board feature enabled, navigate to the Control Panel. Click Utilization > Workflow Utilization.
Workflow Operator Updates
You can use the following operators for more efficient queries:
s3_wait>: This operator enables a timeout option to terminate the service waiting on data to land in the s3 bucket.
td_result_export>: This operator has been updated with a result_connection parameter. Results can be exported to a specific destination.
For more information, see a list of all Workflow operators.
Workflow Message Helps to Prevent Workflow Overwrites
When you are editing a workflow at the same time that others are editing the same workflow, you risk applying your update to an older workflow revision. In TD Console, the following message alerts you to prevent overwrite.
Improved Usability for Import Integrations
Arm Treasure Data offers some new features that improve your experience when creating and editing data transfers for sources. Changes include the following capabilities:
After you configure your data connector, you can:
save it and run the import immediately
save the configured data connector to run at a later time
For every new data transfer, you can select a database and table, instead of a default database and table being selected for you. This is particularly useful when you already have a similar, existing data transfer and want the specified location for the imported data to be different.
You can view all data connectors for import through a new interface:
Google Sheet Export Re-Authentication Required
You must re-authenticate to access Google Drive metadata. Refer to Google Sheets - Export.
The current version of the Google Sheets V3 connector is scheduled to be deprecated early March 2020. Before deprecation, you must re-authenticate all your existing Google Sheet Authentications. Failure to reauthenticate results in a disruption of your Google sheets jobs. For migrating instructions, see the Appendix in Google Sheets - Export.
Twitter Audience Insights
Twitter deprecated the Twitter Ads API endpoint that supports audience insight on January 30th 2020. As a result, the Twitter Audience Insights Integration will be deprecated 02/05/2020.
Coming Soon: Microsoft Azure BLOB Storage Improvement
The incremental loading option is available for access through Arm TD Console. An issue that causes duplicate file ingestion has been fixed. Contact TD Support to ensure that the configuration of your jobs are up to date with the new implementation.
Read more about the import data connector for Microsoft Azure BLOB Storage.
Google Cloud Storage Improvement
The data connector for Google Cloud Storage enables import of the contents of .tsv and .csv files stored in your GCS bucket. You can use the data connector to handle multiple byte characters, specified in the last_path parameter, properly.
AWS Redshift Use of Serial Copy Mode
The export data connector for Redshift supports serial COPY mode. Use serial COPY mode if you encounter the following error when exporting data to Redshift:
> Detail: Process xxxx waits for AccessExclusiveLock on relation xxxx of database xxxx; blocked by process xxxx”
Otherwise, the serial COPY mode option is not recommended because of the negative impact to performance.
Microsoft SQL Server Improvement
Bug Fixes and Other Improvements
Issues that now have the correct behavior in TD Console include the following:
- When an account owner sends a request to a user to reset their password, an email is correctly sent to the user, directing them to reset their password, and providing a link to TD Console.
- In the query editor, when you are in a saved query and you click Save As, you are asked for name for the new query and redirected to a new query page. Your edits are saved to the new query and don’t affect the original query.
Recommend Upgrading to TD Toolbelt v0.16.8
If you use the td import: command, we recommend that you upgrade your version of the TD Toolbelt. For instructions on how to validate your version and upgrade, see TD Toolbelt for CLI Use.
Upgrading resolves import failures.