Treasure Data provides Server-Side Agent called Treasure Agent (td-agent), to collect server-side logs and events. This article explains 4 steps to streamingly import the data from Ruby applications, through Treasure Agent.

Prerequisites

 

• Basic knowledge of Ruby, Gems, and Bundler.
• Basic knowledge of Treasure Data.
• Ruby 1.9 or higher (for local testing).

 

What is Treasure Agent?

First of all, Treasure Agent (td-agent) needs to be installed on your application servers. Treasure Agent is an agent program sits within your application servers, focusing on uploading application logs to the cloud.

 

The td-logger-ruby library enables Ruby applications to post records to their local Treasure Agent. Treasure Agent in turn receives the records, buffers them, and uploads the data to the cloud every 5 minutes. Because the daemon runs on a local node, the logging latency is negligible.

 

How to install Treasure Agent?

This video demonstrates how to install Treasure Agent in 3 minutes.

 

Step 1: Installing Treasure Agent

To install Treasure Agent (td-agent), please execute one of the command below based on your environment. The agent program will be installed automatically by using the package management software for each platform like rpm/deb/dmg.

RHEL/CentOS 5,6,7

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

Ubuntu & Debian

# 14.04 Trusty (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
# 12.04 Precise
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-precise-td-agent2.sh | sh
# 10.04 Lucid
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-lucid-td-agent2.sh | sh

# Debian Squeeze (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-squeeze-td-agent2.sh | sh
# Debian Wheezy (64bit only)
$ curl -L https://toolbelt.treasuredata.com/sh/install-debian-wheezy-td-agent2.sh | sh

Amazon Linux

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh

MacOS X 10.11+

$ open 'https://packages.treasuredata.com/2/macosx/td-agent-2.3.0-0.dmg'

Windows Server 2012+

Windows installation needs multiple steps to follow. Please go to this documentation.

• Installing Treasuer Agent using .msi Installer (Windows)

 

Opscode Chef (repository)

$ echo 'cookbook "td-agent"' >> Berksfile
$ berks install

AWS Elastic Beanstalk is also supported. Windows is currently NOT supported.

 

Step 2: Modifying /etc/td-agent/td-agent.conf

Next, please specify your API key by setting the apikey option in your /etc/td-agent/td-agent.conf file.

# Input from Logging Libraries
<source>
  type forward
  port 24224
</source>

# Treasure Data Output
<match td.*.*>
  type tdlog
  endpoint api.treasuredata.com
  apikey YOUR_API_KEY
  auto_create_table
  buffer_type file
  buffer_path /var/log/td-agent/buffer/td
  use_ssl true
</match>

Please restart your agent once these lines are in place.

# Linux
$ sudo /etc/init.d/td-agent restart

# MacOS X
$ sudo launchctl unload /Library/LaunchDaemons/td-agent.plist
$ sudo launchctl load /Library/LaunchDaemons/td-agent.plist

td-agent will now accept data via port 24224, buffer it (var/log/td-agent/buffer/td), and automatically upload it into the cloud.

 

Step 3: Using td-logger-ruby

First, add the ‘td’ gem to your Gemfile.

gem 'td', "~> 0.10.6"

Next, please initialize and post the records as shown below.

# Initialize
require 'td'
TreasureData::Logger.open_agent('td.test_db', :host=>'localhost', :port=>24224)

# Example1: login event
TD.event.post('login', {:uid=>123})

# Example2: follow event
TD.event.post('follow', {:uid=>123, :from=>'TD', :to=>'Heroku'})

# Example3: pay event
TD.event.post('pay',
              {:uid=>123, :item_name=>'Stone of Jordan',
               :category=>'ring', :price=>100, :count=>1})

Step 4: Confirming Data Import

First, please execute the program above.

$ ruby test.rb

Sending a SIGUSR1 signal will flush td-agent’s buffer; upload will start immediately.

# Linux
$ kill -USR1 `cat /var/run/td-agent/td-agent.pid`

# MacOS X
$ sudo kill -USR1 `sudo launchctl list | grep td-agent | cut -f 1`

From Web Console

To confirm that your data has been uploaded successfully, check your dataset from the web browser HERE.

From CLI

Or, please issue the td tables command if you have a CLI client.

$ td tables
+------------+------------+------+-----------+
| Database   | Table      | Type | Count     |
+------------+------------+------+-----------+
| test_db    | login      | log  | 1         |
| test_db    | follow     | log  | 1         |
| test_db    | pay        | log  | 1         |
+------------+------------+------+-----------+

Production Deployments

Use Rack-based Server Deployments

We recommend that you use unicorn, thin, mongrel, etc. Other setups have not been fully validated.

 

High-Availablability Configurations of td-agent

For high-traffic websites (more than 5 application nodes), we recommend using a high availability configuration of td-agent. This will improve data transfer reliability and query performance.

• High-Availability Configurations of td-agent

 

Monitoring td-agent

Monitoring td-agent itself is also important. Please refer to this document for general monitoring methods for td-agent.

• Monitoring td-agent

Next Steps

We offer a schema mechanism that is more flexible than that of traditional RDBMSs. For queries, we leverage the Hive Query Language.

• Schema Management
• Hive Query Language
• Programatic Access with REST API and its Bindings