This article explains how to send job results directly to your OneDrive account.
You can use this same connector to import One Drive files into Treasure Data. See Import from One Drive.
Prerequisites
- Basic knowledge of Treasure Data, including the toolbelt
- A OneDrive or SharePoint account
- Authorized Treasure Data account access
Use the TD Console to create your connection
Create a new connection
When you configure a data connection, you provide authentication to access the integration. In Treasure Data, you configure the authentication and then specify the source information.
Go to Integrations Hub -> Catalog and search and select OneDrive.
The following dialog opens.
Access to Treasure Data OneDrive requires OAuth2 authentication. The authentication requires that users manually connect their Treasure Data account to their respective OneDrive account.
To authenticate, complete the following procedure:
Click on Click here to connect to a new account.
Log into your OneDrive account in the popup window and grant access to the Treasure Data app.
You will be redirected back to TD Console. Repeat the first step (Create a new connection) and choose your new OAuth connection.
Name your new OneDrive Connection. Click Done.
Configure Output Results to the Data Connection
In this step, you create or reuse a query. In the query, you configure the data connection.
Go to the TD Console query editor page.
Access the query that you plan to use to export data.
Compile the query, if you need to.
Specify the Result Export Target
Click Export Results.
Type the connection name in the search box to filter and select your OneDrive connection.
Edit all the information.
OneDrive Account Plan: there are two account plans:
- Business: you must specify domain, relative path and folder path.
- Personal: you must specify the folder path and whether it is a shared folder or not.
Domain Name: domain of the folder.
Relative Path: the relative path of the user's folder in OneDrive or SharePoint site.
Folder Path: directory of the folder in OneDrive or SharePoint.
File Name: destination file name.
This is a shared folder: select if the folder is shared with you by someone
Format: file extension.
Compression: indicate whether the result is to be compressed or not. Available options: None, gz, and bzip2.
Header line: select if the exported data has the column name as the header line.
Null String: use this value to represent NULL values. Available options:
- Default. select if an empty string ( '' ) is used to represent the Null value for file format CSV, and \N is used for the TSV format.
- Empty string
- \N NULL
- null
End-of-line character: the character at the end of lines. Available options are CRLF, LF, and CR
Quote Policy: available options:
- ALL. select if all values are enclosed by double quotes ("").
- MINIMAL. select if any value that contains an embedded quote (") is presented with a consecutive pair of quotes (""). The MINIMAL quote policy is applied to CSV.
- NONE. select if no escape for embedded quote is applied. By default, NONE is applied to TSV file format.
Max Retries: when an error occurs due to a OneDrive server error or network error, the number of retries attempted before aborting the upload.
Initial Retry Time Wait In Millies: when error occurs, initial waiting time for a retry, double the waiting time after each retry.
Max Retry Wait In Millies: The upper limit waiting time for a retry.
Execute the Query
Save the query with a name and run, or just run the query.
After the query has run successfully, the query result are automatically imported into the specified Container destination.
Example of a Query
SELECT email, first_name, last_name, region, age, gender, website FROM (
VALUES ('kate1@gmail.com', 'Kate', 'Tiny', 'Asia', '41 and above', 'female', 'google.com'),
('ronan@gmail.com', 'R', 'P', 'Americas', '21-30', 'male', 'google.com'),
('michelle@gmail.com', 'M', 'C', 'EMEA', '31-40', 'male', 'facebook.com')
) tbl (email, first_name, last_name, region, age, gender, website);
Optional: Use of Scheduled Jobs for Output
You can use Scheduled Jobs with Result Output, to periodically write the output result to a target destination that you specify.
Optional: Configure Export Results in Workflow
Within Treasure Workflow, you can specify the use of this data connector to output data.
timezone: UTC
_export:
td:
database: sample_datasets
+td-result-into-one-drive:
td>: queries/sample.sql
result_connection: your_connection_name
result_settings:
refresh_token:
account_type:
replace_existing:
file_name:
folder_path:
domain_name:
server_relative_path:
is_shared_folder:
replace_existing:
format:
compression:
header_line:
null_string:
newline:
quote_policy:
Click here for more information on using data connectors in the workflow to export data.
Use the CLI to create your connection
Install ‘td’ command
Install the Treasure Data Toolbelt.
For On-demand Jobs
Add the OneDrive result output destination by using the -r
/ --result
option for the td query
command:
td query -d test_db -w 'SELECT id, via FROM table1' --type presto -r '{"type":"one_drive", "refresh_token":"***","account_type":"personal","domain_name":null,"server_relative_path":null,"folder_path":"/abc","is_shared_folder":false,"file_name":"test_file","replace_existing":null,"format":"csv","compression":"","header_line":true,"null_string":"default","newline":"CRLF","quote_policy":null,"retry_count":"5","retry_initial_wait_millis":"1000","max_retry_wait_millis":"300000"}'
For Scheduled Jobs
Add the OneDrive result output destination by using the -r
/ --result
option for the td sched:create
command:
td sched:create every_6_mins "*/6 * * * *" -d test_db -w 'SELECT id, via FROM table1' --type presto -r '{{"type":"one_drive", "refresh_token":"***","account_type":"personal","domain_name":null,"server_relative_path":null,"folder_path":"/abc","is_shared_folder":false,"file_name":"test_file","replace_existing":null,"format":"csv","compression":"","header_line":true,"null_string":"default","newline":"CRLF","quote_policy":null,"retry_count":"5","retry_initial_wait_millis":"1000","max_retry_wait_millis":"300000"}'
Frequently Asked Questions
Q: How do I export to a folder that someone shared with me?
Business account plan:
You need the domain name, server relative path, and folder path. Go to the Shared With Me tab on the browser:
Click to open the shared folder and make note of the domain name, relative path and folder path as described in Domain Name and Relative Path.
Personal account plan
You must set the field: This is a shared folder (is_shared_folder) to true, and drill down to the shared folder as shown in the image.
Click the shared folder f1 and navigate to the f2 target folder to import
Input the folder path param as shown: folder_path: /f1/f2 (note: folder f0 is not shared with you)
Q: How can I export file to a SharePoint collections?
You need a Domain Name, server Relative Path and Folder Path to import files from SharePoint collections. See Domain Name and Relative Path
Appendix
Domain Name and Relative Path
Domain Name and URL Relative Path are required to access your OneDrive business account plan, including your personal folder, SharePoint site collections, and folders shared with you.
Open the folder that you want to import on the browser. Then get the Domain Name and Relative Path as shown:
Personal folder
SharePoint collection
Folder shared with me
Get your own refresh_token
To issue jobs from CLI, you need client_id, client_secret, and refresh_token.
Register a developer account, and go to Azure App Registration Portal https://portal.azure.com/#home. On the search box, search for App Registration
Click New Registration
Enter App Name, select Account Type and enter Redirect URI as shown. Click Register.
From the App Permissions tab, click Add a Permission
From Request API permissions window, select Microsoft Graph -> Delegated permissions and add permissions as shown:
From the Certificates & secrets tab, click on New client secret, select Never and Add.
A new client secret key is created as shown:
From the Overview tab, you see the client id as shown:
Replace the client_id in this URL and open it on a browser.
https://login.microsoftonline.com/common/oauth2/v2.0/authorize?client_id=xxxxxx&response_type=code&redirect_uri=http://localhost&response_mode=query&scope=offline_access files.readwrite.all openid sites.readwrite.all&state=1234
Accept the Permission Request window.
Copy the code from the address bar:
Replace the code, client_id, client_secret and run a curl command to get the access_token and refresh_token
curl -X POST \
https://login.microsoftonline.com/common/oauth2/v2.0/token \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'client_id={xxxxxx}&client_secret={xxxxx}&grant_type=authorization_code&scope=offline_access%20files.readwrite.all%20openid%20sites.readwrite.all&code={xxxxxx}&redirect_uri=http%3A%2F%2Flocalhost'
Requested Permissions
OneDrive Connector will ask you for following permissions if you are using personal account:
Permission
|
Description
|
Note |
---|---|---|
User.Read | Allows users to sign-in to the app, and allows the app to read the profile of signed-in users. It also allows the app to read basic company information of signed-in users. | |
Files.ReadWrite.AppFolder | Allows the app to read, create, update, and delete files in the application's folder. | Application's folder is a dedicated, special folder for "ARM TreasureData OneDrive Connector" app see What is an App Folder |
Files.ReadWrite.All | Allows the app to read, create, update, and delete all files the signed-in user can access. | The import and export Connector shares the same client application and only performs read, create and update but not delete files. |
offline_access | Allows the app to read and update user data, even when they are not currently using the app. | Use for schedule import/export |
Business/work account requests extra permission
Permission
|
Description
|
Note |
---|---|---|
Sites.ReadWrite.All | Read and write all groups, Edit or delete items in all site collections | only performs read, create and update but not delete files. |
Note: Although the Delete permission is granted by default as part of the Sites.ReadWrite.All permission, the data connector does not delete files on Microsoft OneDrive.
Comments
0 comments
Please sign in to leave a comment.