Posts Tagged: ‘activemq’

Apache JMeter to test Apache ActiveMQ on CI with Maven/Jenkins

August 27, 2014 Posted by jbonofre

Apache JMeter is a great tool for testing, especially performance testing.
It provides a lot of samplers that you can use to test your web services, web applications, etc.

It also includes a couple of samplers for JMS that we can use with ActiveMQ.

The source code of this blog post is https://github.com/jbonofre/blog-jmeter.

Preparing JMeter for ActiveMQ

For this article, I downloaded JMeter 2.10 from http://jmeter.apache.org.

We uncompress jmeter in a folder:

$ tar zxvf apache-jmeter-2.10.tgz

We are going to create a test plan for ActiveMQ. After downloading ActiveMQ 5.9.0 from http://activemq.apache.org, we install and start an ActiveMQ broker on the machine.

$ tar zxvf apache-activemq-5.9.0-bin.tar.gz
$ cd apache-activemq-5.9.0/bin
$ ./activemq console
...
 INFO | Apache ActiveMQ 5.9.0 (localhost, ID:latitude-45782-1409139630277-0:1) started

In order to use ActiveMQ with JMeter, we have to copy the activemq-all-5.9.0.jar file provided in the ActiveMQ distribution into the JMeter lib folder:

$ cp apache-activemq-5.9.0/activemq-all-5.9.0.jar apache-jmeter-2.11/lib/

We can now start jmeter and start to create our ActiveMQ test plan:

$ cd apache-jmeter-2.10/bin
$ ./jmeter.sh

In the default test plan, we add a thread group to simulate 5 JMS clients that will perform the samplers 10 times:

jmeter1

In this thread group, we add a JMS Publisher sampler that will produce a message in ActiveMQ:

jmeter2

We can note the ActiveMQ configuration:

  • the sampler uses the ActiveMQ JNDI initial context factory (org.apache.activemq.jndi.ActiveMQInitialContextFactory)
  • the Provider URL is the ActiveMQ connection URL (tcp://localhost:61616 in my case). You can use here any kind of ActiveMQ URL, for instance failover:(tcp://host1:61616,tcp://host2:61616)).
  • the connection factory is simply the default one provided by ActiveMQ: ConnectionFactory.
  • the destination is the name of the JMS queue where we want to produce the message, prefixed with dynamicQueues: dynamicQueues/MyQueue.
  • by default, ActiveMQ 5.9.0 uses the authorization plugin. So, the client has to use authentication to be able to produce a message. The default ActiveMQ username is admin, and admin is the default password.
  • finally, we set the body of the message as static using the textarea: JMeter message ...

Now, we save the plan in a file named activemq.jmx.

For a quick test, we can add a Graph Results listener to the thread group and run the plan:

jmeter3

We can check in the ActiveMQ console (pointing a browser on http://localhost:8161/admin) that we can see the queue MyQueue containing the messages sent by JMeter:

activemq1

activemq2

Our test plan is working, we have some metrics about the execution in the graph (it’s really fast on my laptop ;)).

This approach is great to easily implement performance benchmark, and creates some load on ActiveMQ (to test some tuning and configuration for instance).

It can make sense to do it in a continuous integration process. So, let’s see how we can run JMeter with Maven and integrate it in Jenkins.

Using jmeter maven plugin

We have two ways to call JMeter with Maven:

  • we can call the local JMeter instance using the exec-maven-plugin. JMeter can be called in “batch mode” (without the GUI) using the following command:
    $ apache-jmeter-2.10/bin/jmeter.sh -n -t activemq.jmx -l activemq.jtl -j activemq.jmx.log
    

    We use the options:

    • -n to disable the GUI
    • -t to specify the location of the test plan file (.jmx)
    • -l to specify the location of the test plan execution results
    • -j to specify the location of the test plan execution log
  • we have a JMeter Maven plugin. It’s the one that I will use for this blog.

The JMeter Maven plugin allows you to run a JMeter meter plan directly from Maven. It doesn’t require a local JMeter instance: the plugin will download and bootstrap a JMeter instance.

The plugin will look for JMeter JMX files in the src/test/jmeter folder by default.

We create a POM to run JMeter:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>net.nanthrax.blog</groupId>
    <artifactId>jmeter</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <build>
        <plugins>
            <plugin>
                <groupId>com.lazerycode.jmeter</groupId>
                <artifactId>jmeter-maven-plugin</artifactId>
                <version>1.9.1</version>
                <executions>
                    <execution>
                        <id>jmeter-test</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>jmeter</goal>
                        </goals>
                    </execution>
                </executions>
                <dependencies>
                    <dependency>
                        <groupId>org.apache.activemq</groupId>
                        <artifactId>activemq-all</artifactId>
                        <version>5.9.0</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

</project>

We can now run the JMeter test plan:

$ mvn clean verify
...
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building jmeter 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- jmeter-maven-plugin:1.9.1:jmeter (jmeter-test) @ jmeter ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  P E R F O R M A N C E    T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO]
[info]
[debug] JMeter is called with the following command line arguments: -n -t /home/jbonofre/Workspace/jmeter/src/test/jmeter/activemq.jmx -l /home/jbonofre/Workspace/jmeter/target/jmeter/results/20140827-activemq.jtl -d /home/jbonofre/Workspace/jmeter/target/jmeter -j /home/jbonofre/Workspace/jmeter/target/jmeter/logs/activemq.jmx.log
[info] Executing test: activemq.jmx
[info] Completed Test: activemq.jmx
[INFO]
[INFO] Test Results:
[INFO]
[INFO] Tests Run: 1, Failures: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.077s
[INFO] Finished at: Wed Aug 27 14:58:09 CEST 2014
[INFO] Final Memory: 14M/303M
[INFO] ------------------------------------------------------------------------

We can see in the ActiveMQ console that the JMeter messages have been sent.

We are now ready to integrate this build in Jenkins:

jenkins1

jenkins2

We have now included the performance tests in our Jenkins CI.

I would advice to execute the performance tests on a dedicated module or profile, and configure the Jenkins job to execute once per week for instance, or link to a release.

So, we still have our development oriented nightly builds, and we can periodically execute performance tests, and execute the performance tests for a release.

Apache ActiveMQ 5.7, 5.9 and Master-Slave

October 3, 2013 Posted by jbonofre

With my ActiveMQ friends (especially Dejan and Claus), I’m working on ActiveMQ 5.9 next release.

Today, I focus on the HA with ActiveMQ, and especially Master-Slave configuration.

Update of the documentation

The first thing that I noticed is that the documentation is not really up to date.

If you do a search on the ActiveMQ website about Master-Slave, you will probably find these two links:

On the first link (about KahaDB), we can see a note “This is under review – and not currently supported”. It’s confusing for the users as this mechanism is the prefered one !
On the other hand, the second link should be flagged as deprecated as this mechanism is no more maintained.

I sent a message on the dev mailing list to updated these pages.

Lease Database Locker to avoid “dual masters”

In my test cases, I used a JDBC database backend (MySQL) for HA (instead of using KahaDB).

I have two brokers, that use the following configuration:


  <persistenceAdapter>
    <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" />
  </persistenceAdapter>

Broker1 starts, connects to MySQL, and acquires the lock. Broker1 is the master.

Broker2 starts, connects to MySQL, and waits for the lock (as the lock is hold by Broker1). Broker2 is a slave.

Now, I stop MySQL, for instance to do a cold backup. My backup is very fast, and I start MySQL server again, very quickly.

The lock is available in the database, so Broker2 get the lock, whereas Broker1 didn’t yet release it. So I’m in a bad situation where I have two “masters”.

ActiveMQ 5.7.0 introduced the change on locking strategies for shared storage master/slave topologies. Previously storage locking (and thus master election) was hard-coded directly in the particular store. So KahaDB has only the option to use shared file lock, while JDBC was using database lock.

Now, the storage locking is separated from the store, so you can implement your own locking strategies if necessary (or tune existing ones). Of course, every store has its own default locker.

In our previsou use case, to solve the “dual master” issue, we can use a new locker: the lease database locker.

To use it, we update the configuration of each locker like this:


  <persistenceAdapter>
    <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" lockKeepAlivePeriod="5000">
      <locker>
        <lease-database-locker lockAcquireSleepInterval="10000"/>
      </locker>
    </jdbcPersistenceAdapter>
  </persistenceAdapter>

Lease database locker solves master/slave problem of the default database locker. Master acquires a lock only for a certain period and must extend it’s lease from time to time. Slave also checks periodically to see if the lease has expired. The lease can survive a db replica failover.

The lease based lock is acquired by blocking at start and retained by the keepAlivePeriod. To retain, the lease is extended by the lockAcquireSleepInterval, so in theory the master is always (lockAcquireSleepInterval-lockKeepAlivePeriod) ahead of the slave w.r.t the lease. It is imperative that lockAcquireSleepInterval > lockKeepAlivePeriod, to ensure the lease is always current.

In the simplest case, the clocks between master and slave must be in sync for this solution to work properly. If the clocks cannot be in sync, the locker can use the system time from the database CURRENT TIME and adjust the timeouts in accordance with their local variance from the db system time. If maxAllowableDiffFromDBTime is > 0 the local periods will be adjusted by any delta that exceeds maxAllowableDiffFromDBTime.

How to know who is the master ?

The “new” mechanism for Master/Slave is great and very easy to set up. You don’t really define who is the master, and who are the slaves. The first broker which get the lock will be the master.

So, a fair question is: how can I know which broker is the master ?

Actually, you already have the response on the JMX layer.

If you connect a JMX client (for instance jconsole) on the broker, and you take a look on the org.apache.activemq:BrokerName=Broker2,Type=Broker MBean, you can see the Slave attribute.

If Slave is true, it means that this broker is a slave. If Slave is false, it’s the master.

Another way to get this information is to use directly the activemq command with bstat argument (instead of JMX):


bin/activemq bstat
...
Connecting to pid: 563
BrokerVersion = 5.9-SNAPSHOT
TempLimit = 53687091200
Persistent = true
MemoryLimit = 67108864
TempPercentUsage = 0
SslURL =
StorePercentUsage = 0
TransportConnectors = {openwire=tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600}
Type = Broker
StompSslURL =
OpenWireURL = tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600
Uptime = 3 minutes
DataDirectory = /home/jbonofre/broker2/data
StoreLimit = 107374182400
BrokerName = broker2
VMURL = vm://broker2
StompURL =
MemoryPercentUsage = 0
Slave = true

You can see the Slave attribute there.

If you want to “script” this and get only the Slave attribute, you can use the query argument:


bin/activemq query --objname Type=Broker --view Slave
...
Slave = true