Python Custom Scripting Tutorial
Python scripts can be run from TD Workflow or Digdag, using the Python operator py>. You can create your workflows for TD using the TD Console or from the command line.
For the workflow to run the Python script, you must specify a Docker image. When the workflow task starts, a new Docker container is created based on the specified Docker image. Docker allows the Python script to execute in the container in an isolated environment.
Running this tutorial takes about 30 minutes and does not require that you have prior experience with Python or Docker images.
- Download and install the TD Toolbelt and the TD Toolbelt Workflow module. For more information, see TD Workflow Quickstart.
- Python 3.6.8 and Anaconda 5.3.0 with Python 3 are supported. Your Python code must be compatible with those versions.
- Install and configure Docker.
- Basic Knowledge of Treasure Workflow's syntax
See examples, for basics such as:
- How to call functions
- How to pass parameters to functions
- How to use environment variables
- How to import functions
Reading and Writing Data from Treasure Data
The examples show how to read data in Treasure Data into a Dataframe, manipulate data, and write it back to Treasure Data as a table.
- You can copy or clone the entire repository.
- Navigate to:
- From the command line, type
lsto verify that you are in the correct directory. You should see the following:
README.md other_scripts scripts simple.dig
- Push the simple examples to your TD environment by typing the following:
td wf push simple-example
This runs the simple.dig workflow and uploads the simple-example to TD.
To verify that the sample was added to TD:
- Open TD Console.
- Navigate to Workflows.
- Search for simple.
- Double click the simple workflow to open up the editor. For example:
To run the Workflow
- Click New Run.
Or to run it from the command line
d wf start simple-example simple --session now
To validate the workflow job run
- From the TD Console, navigate to the workflow editor.
- Select Run History.
- If there are multiple instances of the job, select one to open the job history. From here you can view at what time the job ran, audit logs, and other helpful diagnostic information about the job.