Using the Magento Performance Toolkit

The Magento Performance Toolkit will allow you to run JMeter tests on a generated set of data. This is incredibly useful when setting up new environments or testing configuration changes on existing environments. It gives a very nice baseline set of metrics that you can use to judge how changes affect performance, and the overall load capacity of your server or servers.

This article will cover using this toolkit to test against a fresh copy of Magento 1.14, using only the generated data. This methodology is not as limited as it seems at first glance, and I will go over how to use this against existing sites in a future blog post. I ported the 1.14 Test Framework over to CE 1.9.1, and submitted a PR, this is available for download on my github it's tested and working well.

Wait... really, it only works on the generated data, and it requires the base theme!?!??

Yes, I am sorry to tell you it is true, without the generated data and the base theme the JMeter test will fail. Generated data is referred to specifically by the tests, and it does not appear to be trivial to change the tests to work with other data... but it is possible. The base theme is also referred to a lot as its collecting the products, this also would require a bunch of changes to the JMeter tests.

Anyhow, let's dive in...


Get started with a clean dataset

First things first setup a fresh copy of Magento EE 1.14, it will need to be clean and without the sample data from Magento. Next grab a copy of the Magento Performance Toolkit1, and get that in the right spot inside your Magento install.

Take a look through the documentation and copy the 1.14 toolkit into /var/www/html/magento/dev/tools/performance_toolkit. Your magento install path may differ, but install is as simple as just copying the data into the correct folder inside the Magento file structure. The generate script is picky about being in the right spot, or at least the right directory depth, as the script itself refers to ../../../shell/abstract.php and other paths such as that..

Generating the sample data is pretty easy, with one big gotcha, make sure your memory is set to the proper sample datatype your using. The documentation gives a nice chart of how much memory each profile will need. If you do not do this step, it will fail and you will need to dump the whole lot and start again.

The documentation is very clear about how to use it to generate the data, but it does not go into which profiles have how much.

Profiles Breakdown:

Profile Websites Store Groups / Views Customers Simple Products Config Products Categories (nesting) Price Rules Target Rules Cart Rules
Small 1 1/1 20 800 50 30 (3) 10 2 10
Medium 1 2/1 200 16000 1000 300 (3) 20 5 20
Large 3 3/2 2000 400000 25000 1000 (3) 50 10 50
X-Large 5 5/5 5000 800000 50000 3000 (6) 100 50 100


An example of the generate command on the large profile:

php -f generate.php -- --profile=profiles/large.xml tmp_dir=/var/www/html/magento/var 
Generating profile with following params: 
|- Websites: 3 
|- Store Groups: 3 
|- Store Views: 3 
|- Simple Products: 400000 
|- Configurable Products: 25000 
|- Categories: 1000 
|- Catalog Price Rules: 50 
|- Catalog Target Rules: 10 
|- Cart Price Rules: 50 
|- Regitered Customers: 2000 
Generating stores, store views and websites... done in 00:00:00 
Generating categories... done in 00:00:23 
Generating simple products... [snip] 

This generally takes a really long time with the large dataset, go get some coffee , go run a quick 5k, eat lunch, finish another project, grab a beer and then call it a day. When you come back tomorrow it will be done, and we can move on to the next step. Either that, or try it with the small or medium dataset. If you MUST use large or X-large be certain you follow the directions in the documentation having to do with MySQL/PHP settings, it will be critical for it to work.

If the generator fails, your better off removing the database, and reinstalling and trying again. It does not have the capacity to restart.

Once its complete, double check you can checkout with products and that you can login on the admin panel. The site should be working as intended, if it isn't fix the issues your seeing before you do any testing.

The last bit of setup, is to ensure that that the JMeter test will have an admin user to use. I typically setup a second admin user for this, but you can use the one you setup when you installed the site. Typically I add admin users using n98-magerun2. Make note of the admin user, and let's get ready to do some JMeter testing.


JMeter Setup and Preparations

The JMeter tests can run several ways, including in Blazemeter. I will cover 2 methods, first we will cover the JMeter GUI, and then we will go on to using the JMeter profile from a linux box. The preferred method is from the shell, but to test the JMeter file you may want to run it from the GUI.


Using the JMeter GUI with the Magento Performance Toolkit:

I have been using JMeter 2.12 and JMeter 2.133, both work. Out of the box, the benchmark.jmx will load fine, however there is a few listeners that require the JMeter plugins4 that are offered by Apache.

jMeter GUI

The goal of using the JMeter GUI with these tests is to confirm the test can run without errors. I have not had very good luck using it to perform load tests with the benchmark.jmx because it will dump core do to memory issues with the desktop client. Your results may vary, but in my opinion once you have confirmed that its running without issue, move on to the shell.

To run the test from the GUI you're going to want to change a few of the attributes in the top level of the test plan. I have detailed a few of the options below, it is important that you add your data to the following attribute values to make sure your tests will run.

Name Default Value Notes
host ${__P(host)} Missing the default value, add it as follows:
base_path ${__P(base_path,/) Base path to magento site, added to the base URL.
orders ${__P(orders,0) No need to change this, it will still place orders. I think it is calculated if left at 0
users ${__P(users,100)} This is approximately how many threads are started, ranges from 10-200, try with 30 to start.
loops ${__P(loops,1)} Setting this value higher will make it loop through more tests, I typically run this at 2000
admin_user ${__P(admin_user,admin) Set this to an admin user you have created.
admin_password ${__P(admin_password,123123q) Set this to the password you created.


If you take a look at each thread group in the JMeter test you will find: 

This calculates the number of threads per threadgroup as follows: 
users * group_percent / 100 (rounded down)

This can lead to a section not firing if the group_percent is less than 10 and the number of users is less than 20 or so. Additionally this can lead to alot of threads for the catalog browsing when you scale up users. Loops should just affect how long the test runs, but it will run out of customers if it loops too long.

I start with 30 users, 2000 loops. I will typically change only the users attribute to increase/decrease load. One more hint with this, you can set the percentages to 0-100+ on each of the thread groups and they do not need to add up to 100%, this can be handy for testing just the checkouts or something to that effect.

Another nice way to run the test is to set users to 10, and each thread group percentage to 10 with 1 loop. This gives a single thread per thread group, and can be considered the most basic test. Remember these settings because you can use them from the command line once you are able to run JMeter from the shell.

Once you have plugged in the host and you have made your choice as to the users/orders/loops, you are probably ready to trigger a test. Go ahead and hit the green arrow and take a look at the view results tree, you should see the requests start to flow.

The JMeter test will collect a list of categories based on the <a> tag in the top menu. It's rather clunky and this is where one of the requirements for the base theme come in. It's rather picky and a small change to the top menu will break the extract/asset. Take a look at the following Regex in the JMeter test.

jMeter Category Regex Example

After its done with the categories it grabs the simple products from a search. Builds a list from that. Then it repeats this for configurable products.

Next you will find that it logs in on the admin, as to be able to build a list of customers. It does this with a search, however oddly before it does that it will open the admin grid and prepare its search within the admin grid. It's pretty cool functionality, and it ends up building a list of customers to do the checkouts with later.

So this all happens in the space of the first 30-60 seconds of the test. After which it will shift into testing mode. During testing mode it loops through the following sections, and does a percentage of activity by spinning up threads, the defaults stated below.

  • Category Product Browsing(30%): Opens the homepage, goes to a category, opens up 2 simple products and then a configurable.
  • View Product Add To Cart(62%): Same as Category browsing however it adds each of the items to the cart in addition to viewing them.
  • Guest Checkout (4%): Same as View/Add to Cart, in addition it runs through the checkout as a guest user.
  • Customer Checkout (4%): Same as View/Add to Cart, in addition it runs through the checkout as a logged in customer.

To calculate the number of threads it will create use the following: 
users * group_percent / 100 (rounded down)

It will loop through these until it runs out of loops or customers to create orders with and then it will do some clean up and quit.

When running it from the GUI you're want to look for 500 errors or other errors in the responses. If it gets too many errors you will probably have java lock up and dump core. This is VERY typical for the GUI, don't be discouraged just keep testing. When you move to the command line, you will have a lot less of these, although if it is dumping core because of errors its collecting it will have the same issue from the command line.

It's best to work out what's wrong before moving on, as it's very easy to debug whatever issues your running into with the tests from the GUI.


Running The Performance Tests

Using the JMeter tests from the command line

To use the JMeter tests from the command line, you will want to spin up a linux server(preferably in the cloud) and install java. It is best to NOT use the same server that your running Magento on, even the Magento Performance Toolkit documentation refers to not doing this.

You will want to get a copy of JMeter3 in a folder on the server so you can execute the tests. You will also need the benchmark.jmx for your tests. It's ok to use the default one from the Toolkit, and set the attributes you had edited in the GUI at the command line as follows:

./jmeter -n -t ~/benchmark-default.jmx -Jbase_path=/ -Jusers=30 -Jramp_period=30 -Jloops=2000

As you can see you can set any of the attributes by include a -Jattribute=value flag. This includes any of the attributes you saw in the GUI at the top level.

Example of output from a shell run:

[[email protected]]# ./jmeter -n -t ~/benchmark-default.jmx -Jbase_path=/ -Jusers=30 -Jramp_period=28 -Jloops=2000 
Starting the test @ Thu Apr 02 16:14:50 UTC 2015 (1427991290602) 
Waiting for possible shutdown message on port 4445 
summary + 34 in 9.5s = 3.6/s Avg: 237 Min: 2 Max: 863 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0 
summary + 115 in 30s = 3.8/s Avg: 240 Min: 1 Max: 560 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0 
summary = 149 in 40s = 3.8/s Avg: 240 Min: 1 Max: 863 Err: 0 (0.00%) 
You have to increase customers qty for running 'Customer Checkout' thread. 
summary + 3364 in 28s = 122.2/s Avg: 193 Min: 2 Max: 3182 Err: 2 (0.06%) Active: 0 Started: 0 Finished: 0 
summary = 75339 in 574s = 131.1/s Avg: 177 Min: 1 Max: 4188 Err: 2 (0.00%) 
Tidying up ... @ Thu Apr 02 16:24:25 UTC 2015 (1427991865345) ... end of run 


What does the output mean?

Starting the test @ Thu Apr 02 16:14:50 UTC 2015 (1427991290602) 
Tidying up ... @ Thu Apr 02 16:24:25 UTC 2015 (1427991865345) 

These two lines lets you know when the test started and ended. Very useful for calculating things like how many orders per hour it can do. You will need to keep an eye on the orders at the start and finish of the test run and look at the time it took to complete.

summary + 4393 in 32.3s = 136.1/s Avg: 174 Min: 2 Max: 4188 Err: 0 (0.00%) Active: 26 Started: 26 Finished: 0 

Here you have a summary in a time slice during the test. It seems to randomly output these throughout the test and they are handy for watching the last x seconds and to monitor the status of the running threads. The output is a bit jumbled but it has the following info:

  • Requests over time period: 4393 in 32.3s = 136.1/s
  • Average Values for response times: Avg: 174 Min: 2 Max: 4188
  • Errors it encounters: Err: 0 (0.00%)
  • Thread Status: Active: 26 Started: 26 Finished: 0


summary = 75339 in 574s = 131.1/s Avg: 177 Min: 1 Max: 4188 Err: 2 (0.00%) 

These lines are the cumulative totals throughout the test. The last one is always very useful and can really be considered your "test results"

Some hints as to what you should see for values in these:

  • Average Response Time should be one of your main metrics. It should be an acceptable value, less than say 250ms - 1000ms, depending on what you are testing.
  • Max Response Time should be under 10,000ms, maybe a bit more. You will have edge cases where the max is very high. It is best to look through the summary = lines and find peaks in this
  • Min Response Time: I always seem to have 0 or 1 in this, seems useless.

Use Newrelic in conjunction to testing to get a really good feel as to the application health. You can really notch your performance metrics vs JMeter tests when using Newrelic, finding those breaking points and/or bottlenecks in the application.



This article covered a lot of ground but there is more to do. I would like to show how to use the data you are generating and how to tell if a server or set of servers is "healthy". Additionally I would like to figure out and write about how to craft these tests into working with a production website.

Feel free to contact me on twitter @aepod with feedback, and/or help with the Magento Performance Toolkit.

Thanks for reading. Also thanks to Magento for providing this framework, it just about closes a long standing hackathon item5, and I look forward to using it and sharing more insight into the process.


  1. ?

  2. A must have tool for maintaining Magento applications from the command line. ?

  3. ?

  4. You will need Standard and Extras to use charts. ?

  5. ?