Adding a Custom Python Script to your Workflow
Python scripts can be run from TD Workflow, using the Python operator (py>). The workflow definition can be created using the TD Console or using TD Workflow from the command line.
In the workflow definition, specify a Docker image to use for running the Python script. When the workflow task starts, a new Docker container is created based on the specified docker image. Then, the Python script is executed in the container in an isolated environment.
- Download and install the TD Toolbelt and the TD Toolbelt Workflow module. For more information, see TD Workflow Quickstart.
- Python 3.6.8 and Anaconda 5.3.0 with Python 3 are supported. Your Python code must be compatible with those versions.
- Install and configure Docker.
- Basic Knowledge of Treasure Workflow's syntax
See examples, for basics such as:
- How to call functions
- How to pass parameters to functions
- How to use environment variables
- How to import functions
Creating a workflow that runs a Python script from the TD Console
- Open your workflow definition, Data Workbench > Workflows.
- Create a new workflow or open and edit an existing workflow.
- Add the ‘py>’ operator and specify a Docker image that you want to use. For example: # +py_custom_code_on_.
Creating a workflow that runs a Python script from the command line
- Add a workflow definition .dig file and Python script to the workflow directory.
- Create or find a Docker image that you want to use for the ‘py>’ operator in the .dig file.
- Add syntax similar to the following to your workflow dig file to add the py> operator and specify the Docker image.
- Add the py> to the workflow to call a Python method from the script.
For example, in the following workflow, “python:3.6.8stretch” is the Docker image for your Python script:
# One can use secrets as well.
# See https://support.treasuredata.com/hc/en-us/articles/360001266788-Workflows-Secrets-Management