Load Testing With JMeter (Part 1)

blog

JMeter is one of the most popular (if not the most popular) tool for automated load generation. It’s a tool that’s easy to install and get productive with but versatile enough that it can hit almost any situation you can think of and you’ll never truly master.

To get it installed, just:

  1. Make sure you have java installed
  2. Download the latest tarball from the website and extract it somewhere easy to get to.
  3. Execute the jmeter.bat script if you’re on Windows otherwise execute jmeter.sh (Mac or Linux)

Contents

Application Overview

The purpose of JMeter is to generate load and collect metrics. Pursuant to that goal, it’s designed with flexibility in mind. It can go from a simple four iteration loop of the same operation to complex test plans involving many client nodes that involves logging into a website and performing a set of specific operations on it.

HTTP is the most popular protocol to use JMeter to load test but it’s not the only one supported. It supports using ODBC connections for generating load directly to the database, FTP transport, generate SMTP email, or LDAP requests.

Terminology

This section is the longest because truth be told, JMeter just isn’t terribly complicated. The only point of confusion I’ve ever had with it is with the terminology they’ve adopted to describe the various features. Maybe it makes sense to you but this took me a few minutes to really get my head around. Some important terms:

  • Test Plan: The root container for everything you’re configuring in JMeter including all thread groups and Config elements. The test plan and all child elements can be saved to an XML-formatted .jmx file.
  • Sampler: Would probably more accurately be called “operation.” This is the actual operation you’re performing as part of the test (GET’ing a page, UPDATE’ing a database row, etc) and collecting the performance metrics/result of.
  • Listener: The object that records the data in some way. For instance, the listener may draw a graph of the sampler results, put them in a table, or simply save it to a CSV file.
  • Thread Group: The basic building block for executing your test. It spawns a configurable number of process threads and executes the test plan elements nested underneath it such that each thread is isolated from the others. Each thread group is supposed to represent a particular scenario. For instance, you might have one thread group for populating a Drupal site with fake articles using 100 concurrently running threads, while another thread group running at the same time has 100 threads pulling the RSS feed for a view listing articles. By default all thread groups run in parallel to one another while the elements inside a thread group are executed sequentially (if applicable). The test plan, though, can be configured to run Thread Groups sequentially, though.
  • Controller: Directs the flow of action within a particular thread group. There’s a wide variety of controllers but the function of each is fairly obvious and something I’m going to go over in detail in Part 2 of this series.
  • Timer: Inserts pauses in a thread group’s execution. Useful for simulating natural user behavior and to help ensure your test doesn’t DoS the server. Another useful feature is the set of “random” timers that come with JMeter. With variable length pauses in each thread (remember each thread is on its own so it has its own random number generated) you can cause each thread to gradually be at a different point in the test plan thus simulating the structured “randomness” of actual user traffic (rather than having a test that moves from one particular operation to another in unnatural unison)
  • Assertion: A high level test for ensuring the response to each sampler operation matches the expected results. For instance you can add assertions for HTML validation as well as generic “Assertion Responses” for executing a series of regex patterns against the server’s response. This is helpful in knowing the test executed successfully and therefore the results can be trusted.

Performing a Simple Test

Believe it or not, if you have it installed and understood the above you’re about 80% the way towards writing useful tests in JMeter.

In this section we’ll create a simple test plan that just requests a single page a bunch of times and graphs the results.

First, let’s start with a blank test plan and add our first thread group to it:

Now that we have our first thread group let’s configure it so that we have 100 separate threads executing at once each loop until we stop it manually:

Now that the thread group is setup, let’s add an actual HTTP request to perform during the test:

And configure it as needed for whatever it is you’re trying to do:

Please change the server name, port, and path to whatever is valid for the page you’re testing. On my system, http://example.dur/dci-status returns a page, but that’s likely not going to be the case with you so please adjust accordingly. You’ll also want to change the HTTP Request element’s name, as that will be the default name for this operation in any reports/graphs that report the results itemized by operation.

You’ll usually want to explore the options they have on the page. In our case we want to simulate the full user experience loading this super important page so let’s switch over to the “Advanced” tab on our HTTP request and ask that it download any images, stylesheets, etc referenced inside the response:

Collecting and Interpreting the Results

OK, great so we have our GET request configured but we need to add some listeners so we can see the results. In my case I added two just to be thorough:

In the above I’ve adding a Response Time Graph which will draw the graph we’re actually planning to show others and View Results Tree which is useful in troubleshooting problems that may crop up in your test creation. The results tree will capture both the HTTP request (including HTTP headers) made to the server as well as the server’s response (including headers). Usually if there’s some issue with your test’s execution it becomes obvious after viewing the results here.

I won’t go through the listener configuration that much since it’s more of a “salt to taste” situation where it works out of the box and the “correct” settings are entirely dependent upon what you want the reporting behavior to actually be.

OK so let’s run this thing and see what we get. Since we’ve configured the threads to run “forever” we’ll need to just wait until it’s issue enough requests to graph for us then manually stop the test execution. On my run of this test, the response graph returned looks like this:

Which is the exact kind of graph you can show your non-technical manager instead of a boring listing of numbers. The above illustrates that (in my case) as the requests began being backlogged the server took longer and longer to respond due to the increased load.

Sometimes you you actually need numbers rather than a pretty graph though. Fret not since you can just add the View Results in Table listener to your test before running. Any samples collected will be itemized individually along with specific numbers for average response time and the standard deviation. If your standard deviation is too high, I would revisit your test as that usually indicates that you haven’t created enough of a controlled experiment for your results to be trustworthy.

Most complicated tests won’t produce results that look that clean. Given that, I have two suggestions for producing graphs that won’t confuse non-technical people:

1) Collect many many samples and adjust the interval on your graph so that each data point merges and the actual mean/median is what the graph represents. That won’t eliminate sudden aberrations though where a single request that takes half an hour drowns out the hundreds or thousands that only took 500ms. To rectify that…

2) Before you start any long test you should configure your listener to save the results to a CSV file you can then edit to remove whatever random aberrations contained within your samples that are throwing the graph off. As long as you keep the samples your manually removing at or below 0.5% the total samples (assuming 10,000+ samples) your results should be considered “cleaned” rather than “doctored.”

If you go much beyond that you should try to identify why you’re getting the sudden aberrations so regularly and either remediate (in the application) whatever larger issue your load testing has uncovered (recollecting entirely new samples) or try to find a way to identify which samples in the CSV represent the aberration and which don’t removing the ones unrelated to larger issues. The latter choice will leave the aberrant pattern on the graph but you’ll be able to explain their presence to management and ultimately represents what the application is actually doing.

Introducing Variables to Your Test Plan

Occasionally, You’ll want to reuse the same test plan for multiple websites. For instance, maybe your full time job is just as a system administrator and you just run a load test against new WordPress websites before they’re made production to ensure a single node can deliver the performance you’re going to need or if you need to deploy the WordPress instance as part of a high availability solution where the load is spread around.

In that example, you could easily see 20-25 new websites spring up over the course of a year and writing a new test plan for each one would be a chore. To add to the trouble, our example is purposefully simple. Real test plans will likely include *many* more HTTP requests than that with hard coded host names and paths.

Luckily, JMeter supports variables for storing often changing or dynamic information. You can either define variables in various places: statically on the root node, statically as part of a “User Defined Variables” configuration element, or dynamically as the result of some post-processor analyzing the results of a previous request.

Simplest is to set the variable on the root node. To do so, just select the root node (the test plan node) and the center of the content pane should be the dialog for adding new user variables:

In that example, I’ve added a new variable called serverName and I’ve modified my HTTP request to utilize this in the “Server Name” field:

Writing your test plans in this way allows you to simply have a “Pre-prod WordPress” test plan that you write once (and occasionally tweak) and then each time you get a new website you need to evaluate, you just change the user variable at the top level and then run your test. No further editing required.

Variables can get more complicated, with post-processors being able to store variables at one point of the Test Plan execution and utilize them at a later step by saving the extracted data to a variable.

Further Reading