Load Testing with Tsung

class: center, middle

# Load Testing with Tsung

__CS291A__

Dr. Bryce Boe

November 9, 2017

---

# Today's Agenda

* TODO

* Load Testing with Tsung

---

# TODO

* Keep making excellent progress on your project.

* Periodically ensure you can successfully deploy your project to AWS.

* Start thinking about what you want to load test and how you will go about it.

## Before the End of Lab

* Write an initial Tsung xml file to load test a simple action on your
  application

---

# Load Testing with Tsung

After today you will know how to evaluate the scalability of a deployed web
application using Tsung.

Today will be interactive, so if you've brought your laptop feel free to
follow-along.

---

# Load Testing Motivation

Assume we are considering making a significant change to our web application.

We suspect that this change will improve our scalability.

> How can we confirm our suspicion?

---

# Load Testing

With load testing we can establish a baseline of expected performance of our
application.

We can then compare the results of previous load tests to those obtained after
making our changes.

> What should we observe?

* Response times

* Error rates

* Synthetic user success rate

---

# Modeling User Behavior

Ideally _real_ users would be used to compare an application's
performance.

Short of that, we want to create load-test flows that resemble
real traffic. There are a few things to consider:

* Ensuring flows contain a mixture of reads and writes

* Ensuring variance within the flows themselves (not all users have the same
  habits)

* The test framework should be able to utilize data returned from prior
  requests (e.g., create a submission and make a comment on it)

> Why are these considerations important?

---

# Load Testing Tools

## High Performance Tools

* [apachebench](http://httpd.apache.org/docs/2.4/programs/ab.html) (ab)
* [httperf](https://github.com/httperf/httperf)
* [Tsung](http://tsung.erlang-projects.org/)

## Feature-rich Tools

* [Funkload](https://funkload.nuxeo.org/)
* [Tsung](http://tsung.erlang-projects.org/)

## Others

* [Bees with Machine Guns](https://github.com/newsapps/beeswithmachineguns)

Tsung is a good choice as it is extremely configurable and delivers high
performance. Today I will show you how to use it.

---

# Set Up

We ideally want one application deployment, and one load testing instance per
team.

* If you can, sit near your team.

* Deploy a version of your application

* Deploy Tsung on a `m3.medium` instance using CloudFormation:

* Visit: https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new

* Use for the "S3 template URL": https://cs291.s3.amazonaws.com/Tsung.json

__Note__: Now that we're directly using CloudFormation, you have one more place
to shutdown instances when you're done working.

---

# Testing EC2 instances from EC2

We are going to launch the test instances in the same data center as our
stack. This decision provides us with:

* Lower cost: AWS charges for bandwidth outside of the data center

* Predictability: fewer moving parts between the test instance and the stack

* Less representative testing: none of our real users will be in the data
  center

The first two points are positive.

The third point could be a concern, but since we are essentially using these
sort of tests for comparison between code changes, we're less concerned with
being completely representative.

---

# Launching a Tsung Stack

* Prefix the stack name with your teamname

* Use the `m3.medium` instance type unless you have a specific reason to use
  one of the larger types

* Select your team name so that your SSH private key works to SSH into the
  instance

---

# Accessing the Tsung Instance

When the stack launch completes, the _Outputs_ tab will contain the SSH string
and a URL.

![Tsung Outputs tab](tsung_outputs.png)

At first accessing the URL will result in an error. It will be useful once
Tsung is running.

Use the SSH string to SSH into the Tsung instance.

---

# The Tsung Instance

The home directory of the Tsung instance contains one file:

```bash
[ec2-user@ip-10-140-200-96 ~]$ ls
simple.xml
```

`simple.xml` is a sample Tsung load-test configuration file that establishes 1
user each second to http://www.google.com.

Run this Tsung test via:

```bash
tsung -f simple.xml -k start
```

One Tsung starts, visit the URL listed in the outputs.

---

# Tsung Server

While Tsung runs (and continues to run after testing with the `-k` option) it
provides a simple web interface.

![Tsung Web Interface](tsung_interface.png)

---

# Tsung Interface

Out of the box, Tsung generates a report `/es/ts_web:report`, and a handful of
graphs `/es/ts_web:graph`.

The server, and web-based access to these files ceases to be accessible once
the Tsung process is stopped. Restarting Tsung only makes the graphs from the
current run available via the web interface.

---

# Fetching Tsung results

Aside from fetching files through the web interface. You may want to obtain all
of the Tsung results.

Assuming your SSH connect string is: `ssh -i demo.pem ec2-user@54.166.5.220`
then the following command will synchronize all the Tsung result files (from
all runs) to your local machine:

```bash
rsync -auvLe 'ssh -i demo.pem' ec2-user@54.166.5.220:tsung_logs .
```

Running that command a second time will only fetch any new files (assuming you
run it from the same path). rsync is awesome!

---

# The Tsung XML File

We will discuss some of the parts of the Tsung XML file. For a more complete
discussion please see Tsung's documentation:
[http://tsung.erlang-projects.org/user_manual/configuration.html](http://tsung.erlang-projects.org/user_manual/configuration.html)

# Tsung File Structure

```xml
<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/local/share/tsung/tsung-1.0.dtd" [] >
<tsung loglevel="notice">

<clients>...</clients>
  <servers>...</servers>
  <load>...</load>
  <options>...</options>
  <sessions>...</sessions>
  ...

</tsung>
```

---

# Tsung Clients Section

Reference: [http://tsung.erlang-projects.org/user_manual/conf-client-server.html](http://tsung.erlang-projects.org/user_manual/conf-client-server.html)

```xml
<clients>
  <client host="localhost" use_controller_vm="true" maxusers='30000'/>
</clients>
```

The clients section specifies the clients to generate load from. While you can
use multiple machines as part of a single test, you probably won't need to.

---

# Tsung Servers Section

Reference: [http://tsung.erlang-projects.org/user_manual/conf-client-server.html](http://tsung.erlang-projects.org/user_manual/conf-client-server.html)

```xml
<servers>
  <server host="www.google.com" port="80" type="tcp"></server>
</servers>
```

The servers section specifies where you intend to connect to. For your tests
you will always want to modify the `host` value to match your deployment's
domain.

---

# Tsung Load Section

Reference: [http://tsung.erlang-projects.org/user_manual/conf-load.html](http://tsung.erlang-projects.org/user_manual/conf-load.html)

```xml
<load>
  <arrivalphase phase="1" duration="60" unit="second">
    <users arrivalrate="2" unit="second"></users>
  </arrivalphase>
  <arrivalphase phase="2" duration="1" unit="minute">
    <users interarrival="2" unit="second"></users>
  </arrivalphase>
</load>
```

Here we describe two phases.

The first phase lasts 60 seconds where 2 new users arrive (`arrivalrate`) each
second.

The second phase also lasts 60 seconds. Here 1 new user arrives
(`interarrival`) every two seconds.

---

# Tsung Options Section

Reference: [http://tsung.erlang-projects.org/user_manual/conf-options.html](http://tsung.erlang-projects.org/user_manual/conf-options.html)

```xml
<options>
  
  <option name="global_ack_timeout" value="2000"></option>

<option type="ts_http" name="user_agent">
    <user_agent probability="100">Mozilla/5.0 ...</user_agent>
  </option>
</options>
```

Unless you do something special with the user-agent in your web application,
you probably will not need to modify these options.

---

# Tsung Sessions Section

Reference: [http://tsung.erlang-projects.org/user_manual/conf-sessions.html](http://tsung.erlang-projects.org/user_manual/conf-sessions.html)

```xml
<sessions>
  <session name="http-example" probability="100" type="ts_http">
    <request>
      <http url='/' version='1.1' method='GET'></http>
    </request>
    <thinktime value="2" random="true"></thinktime>
    <request>
      <http url='/login' version='1.1' method='POST'
            contents="user=foo"></http>
    </request>
  </session>
</sessions>
```

You can define multiple sessions and for each user one will be chosen according
to the probabilities.

---

# Load Testing Your Server

* Modify the `server` tag's `host` section to point to your deployment's
  domain.

* Add a `request` tag for each resource fetched when you load your
  application's home page.

* Group requests to discrete _pages_ in `<transaction>` if you want statistics
  on the individual pages (don't include `<thinktime>` inside a transaction
  section.

---

# Obtaining Dynamic Variables

Reference: [http://tsung.erlang-projects.org/user_manual/conf-advanced-features.html](http://tsung.erlang-projects.org/user_manual/conf-advanced-features.html)

Assume this HTML form:

```html
<form action="/create" method="POST">
  <input name="auth_token" type="hidden" value="gDTI=" />
</form>
```

The value `gDTI=` can be obtained from the response via:

```xml
<request>
  <dyn_variable name="auth_token"></dyn_variable>
  <http url="/new" method="GET" version="1.1"></http>
</request>
```

---

# Using Dynamic Variables

We previously set a variable named `auth_token`. Let's use it.

```xml
<request subst="true">
  <http url="/create" version="1.1"
        contents="user=bboe&amp;auth_token=%%_auth_token%%"
        content_type="application/x-www-form-urlencoded" method="POST">
  </http>
</request>
```

Substitution is done via `%%_VARIABLE_NAME%%`.

In rails CSRF protection values are usually stored in a hidden field named
`authenticity_token`.

__Note__: The `&` character must be encoded like `&amp;`. This causes a problem
with the CSRF token.

---

# Other Dynamic Variables

The above example shows how to extract a variable from a form field. They can
also be extracted from:

* HTML using [XPath](http://tsung.erlang-projects.org/user_manual/conf-advanced-features.html#xpath)

* arbitrary text using [Regexp](http://tsung.erlang-projects.org/user_manual/conf-advanced-features.html#regexp)

* JSON via [JSONPath](http://tsung.erlang-projects.org/user_manual/conf-advanced-features.html#jsonpath)

---

# Generating Random Variables

Variables can be explicitly set via a few functions and occur anywhere in a
`<session>`.
([ref](http://tsung.erlang-projects.org/user_manual/conf-advanced-features.html#set-dynvars))

## Random Number as `rndint`

```xml
<setdynvars sourcetype="random_number" start="3" end="32">
  <var name="rndint" />
</setdynvars>
```

## Random String as `rndstring1`

```xml
<setdynvars sourcetype="random_string" length="13">
  <var name="rndstring1" />
</setdynvars>
```

For a more complete example please see:
[Demo App's load_tests/critical.xml](https://github.com/scalableinternetservices/demo/blob/master/load_tests/critical.xml)

---

class: center, middle, inverse

# Understanding Tsung's Output

---

# Tsung Output

* __connect__: The duration of the connection establishment
* __page__: Response time for each set of requests (a page is a group of
  requests not separated by a thinktime)
* __request__: Response time for each request
* __session__: The duration of a user's session

![Tsung Statistics](tsung_statistics.png)

---

# Response Time Graph

![Tsung Response Time Graph](tsung_response_time.png)

---

# Tsung HTTP Return Codes

Inspect the HTTP return code section:

* 2XX and 3XX status codes are good

* 4XX and 5XX status codes can indicate problems:
    * Too many requests?

* Server-side bugs?

* Bug in testing code?

Ideally you shouldn't see any 4XX and 5XX status codes in your tests unless you
are certain they are due to exceeding the load on your web server's stack.

---

# Error Rate Graph

![Tsung Error Rate Graph](tsung_error_rate.png)

---

# Users Graph

![Tsung Users Graph](tsung_users.png)

---

# Advice

## Reminder: Fetch your Tsung data as soon as it is available

```bash
rsync -auvLe 'ssh -i demo.pem' ec2-user@54.166.5.220:tsung_logs .
```

## Use tsplot to compare separate runs:

```bash
tsplot -d graphs m3_medium tsung_logs/20171108-2133/tsung.log m3_large tsung_logs/20171108-2146/tsung.log
```