TD Workflows can be used with version control tools like Git and continuous integration tools such as CircleCI. In this tutorial, we’ll go through how to set up a Continuous Deployment pipeline using GitHub and CircleCI.
This tutorial assumes that you’re already somewhat familiar with TD Workflows. If not, review the Workflows Introductory tutorial.
For this tutorial, you will need:
If you have not already done so, create a git repository on GitHub with your workflow project. For more information on how to use Git and GitHub, see the GitHub Bootcamp.
Add a CircleCI configuration file to the project. The file should be named
.circleci/config.yml, be in the top directory of the project, and have the following contents. Then commit and push to GitHub.
version: 2.1 executors: my_executor: docker: - image: circleci/openjdk:8 jobs: digdag: executor: my_executor steps: - checkout - run: name: Setup command: | mkdir -p ~/bin/ curl -o ~/bin/digdag -L "https://dl.digdag.io/digdag-latest" chmod +x ~/bin/digdag echo 'export PATH="$HOME/bin:$PATH"' >> $BASH_ENV - run: name: Check config command: digdag check - run: name: Push project command: digdag push $WF_PROJECT -e api-workflow.treasuredata.com -X client.http.headers.authorization="TD1 $TD_API_KEY" -r `date -u +"%Y-%m-%dT%H:%M:%SZ"`-`git rev-parse HEAD` workflows: version: 2 build-n-deploy: jobs: - digdag: filters: branches: only: - master
Note that the above configuration specifies that CircleCI should deploy changes on the
master branch only. This ensures that changes on other branches and pull requests are not deployed until they are merged to
For more information on CircleCI configuration, see the CircleCI configuration documentation.
Now open the CircleCI dashboard, click “Add Projects” on the left hand side and select your account. The GitHub repository containing your TD Workflow project should be listed. Click “Build Project” on the right hand side in order to start the continuous deployment pipeline.
CircleCI will now perform the first run of the pipeline, and will fail because we have not yet configured TD credentials.
Configure TD Credentials
In order for workflows to be pushed to Arm Treasure Data, you must configure a TD API Key for the project.
- Open the CircleCI dashboard again. The workflow project should be listed on the left.
- Click the workflow project and then “Project Settings” in the upper right corner.
- Click “Environment Variables” under “Build Settings” on the left hand side.
TD_API_KEYinto the “Name” field and your TD api key in the “Value” field.
- Click “Save variables”.
- Open the CircleCI dashboard again and click the failed workflow project build.
- Click “Rebuild” in the upper right corner.
CircleCI should now successfully push your workflow to Treasure Data and will do this every time you push a change to GitHub.