TD Workflows can be used with version control tools like Git and continuous integration tools such as CircleCI. In this tutorial we’ll go through how to set up a Continuous Deployment pipeline using GitHub and CircleCI.
This tutorial assumes that you’re already somewhat familiar with TD Workflows. If not, please check out the onboarding tutorial first.
For this tutorial, you will need:
If you have not already done so, create a git repository on GitHub with your workflow project. For more information on how to use Git and GitHub, see the GitHub Bootcamp.
Add a CircleCI configuration file to the project. The file should be named
circle.yml, be in the top directory of the project and have the following contents. Then commit and push to GitHub.
machine: timezone: UTC java: version: oraclejdk8 dependencies: override: - mkdir -p ~/bin/ - curl -o ~/bin/digdag -L "https://dl.digdag.io/digdag-latest" - chmod +x ~/bin/digdag test: override: - digdag check deployment: production: branch: master commands: - > digdag push $(basename $(git rev-parse --show-toplevel)) -e api-workflow.treasuredata.com -X client.http.headers.authorization="TD1 $TD_API_KEY" -r `date -u +"%Y-%m-%dT%H:%M:%SZ"`-`git rev-parse HEAD`
Note that the above configuration specifies that CircleCI should deploy changes on the
master branch only. This ensures that changes on other branches and pull requests are not deployed until they are merged to
For more information on CircleCI configuration, see the CircleCI configuration documentation.
Now open the CircleCI dashboard, click “Add Projects” on the left hand side and select your account. The GitHub repository containing your TD Workflow project should be listed. Click “Build Project” on the right hand side in order to start the continuous deployment pipeline.
CircleCI will now perform the first run of the pipeline, which will fail because we have not yet configured TD credentials.
Configure TD Credentials
In order for workflows to be pushed to Treasure Data, you will need to configure a TD API Key for the project.
- Open the CircleCI dashboard again. The workflow project should be listed on the left.
- Click the workflow project and then “Project Settings” in the upper right corner.
- Click “Environment Variables” under “Build Settings” on the left hand side.
TD_API_KEYinto the “Name” field and your TD api key in the “Value” field.
- Click “Save variables”.
- Open the CircleCI dashboard again and click the failed workflow project build.
- Click “Rebuild” in the upper right corner.
CircleCI should now successfully push your workflow to Treasure Data and will do this every time you push a change to GitHub.