Archive for: ‘October 2013’

Talend ESB Continous Integration, part2: Maven and commandline

October 24, 2013 Posted by jbonofre

In the first part of the “Talend ESB Continuous Integration” serie, we saw how to test the Camel routes created by the studio, by leveraging Camel Test Kit. We saw how to have automatic testing using Jenkins.

The Maven POM that we did assumes that the route has been deployed (on the local repository or on a remote repository like Apache Archiva).

But, it’s not so elegant that a Studio directly publish to the Archiva repository, especially from a continuous integration perspective.

In this second article, I will show how to use the Talend commandline with Maven, and do nightly builds using Jenkins.

Talend CommandLine

CommandLine introduction

The Talend commandline is the Talend Studio without the GUI. Thanks to the commandline, you can do a lot of actions, like checkout, export route, publish route, execute route. Actually, you can do all actions except the design itself 😉

You can find commandline*.sh scripts directly in your Talend Studio installation, or you can launch the commandline using:

./Talend-Studio-linux-gtk-x86_64 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace

You can use the commandline in different mode:

  • Shell Mode:

    ./Talend-Studio-linux-gtk-x86 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace shell
    

    Using this mode, the commandline starts a shell. You can execute the action directly in this shell. Type quit to exit from the commandline.

  • Script Mode:

    ./Talend-Studio-linux-gtk-x86 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace scriptFile /path/to/script
    

    Using this mode, the commandline starts and executes the actions (commands) listed in the script file.

  • Server Mode:

    ./Talend-Studio-linux-gtk-x86 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace startServer -p 8002
    

    Using this mode, the commandline starts a server. You can execution actions (commands) on the commandline using telnet (telnet localhost 8002). Type stopServer (eventually –force) to exit from the commandline.

The help command provides a list of all commands that you can execute in the commandline.

The first action to perform in the commandline is to init the Talend repository (containing the metadata). The repository can be local or remote.

To init a local repository, simply execute the following command in the Talend commandline:

talend> initLocal

To init a remote repository, you have to use the initRemote command, providing the location of a Talend Administration Center:

talend> initRemote http://localhost:8080/org.talend.administrator

As the commandline performs the actions asynchronously, you can see all commands (and the status) executed by the commandline, using listCommand:

talend> listCommand -a
0:COMPLETED InitRemoteCommand initRemote

Once the repository initialized, we can list the project in the repository:

talend> listProject
CI (CI) java desc=[Continuous Integration Sample] storage=[Local]


If you don't have existing project, you can create a new project:


talend> createProject -pn "CI" -pd "Continuous Integration Sample" -pl java -pa "jbonofre@talend.com"
talend> listCommand -a
1:COMPLETED CreateProjectCommand createProject -pn 'CI' -pd 'Continuous Integration Sample' -pl 'java' -pa 'jbonofre@talend.com'  name CI description Continuous Integration Sample language java author jbonofre@talend.com

Now, you can logon a project:

talend> logonProject -pn CI -ul "jbonofre@talend.com" [-up "password"]
talend> listCommand -a
2:COMPLETED LogonProjectCommand log on CI

Once logged on a project, you can list routes, jobs, services in this project:

talend> listRoute
talend> listJob
talend> listService

If you use a remote repository, once logged on the project, you will have all jobs, routes, and services checked out from the svn.

If you initialized a local repository, you may want to import items (jobs, routes, services) that you export from a studio.

talend> importItem /home/jbonofre/MyRoute.zip
talend> listCommand -a
3:COMPLETED ImportItemsCommand

Now, you can see the items that you imported:

talend> listRoute
[Samples]
  MyRoute

Now, we can use the command line the create the route kar file:

talend> exportRoute MyRoute -dd /home/jbonofre
talend> listCommand -a
4:COMPLETED ExportRouteServerCommand exportRoute 'MyRoute' -dd '/home/jbonofre'

We have the MyRoute.kar file created:

jbonofre@vostro:~$ ls -lh|grep -i kar
-rw-r--r--  1 jbonofre jbonofre 231K Oct 24 17:29 MyRoute.kar

Using the Talend Enterprise Edition, instead of creating the kar file locally, we can publish the route features (and all dependencies) directly to a Maven repository (Apache Archiva in my case):

talend> publishRoute MyRoute -pv 0.1.0-SNAPSHOT -g net.nanthrax -a MyRoute -r "http://localhost:8082/archiva/repository/repo-snapshot" -u tadmin -p foo 

We gonna use a combination of these commands on a commandline invoked by Maven.

Prepare the commandline

For our build, we use the script mode on the commandline.

To simplify, we create a commandline-script.sh in the Talend Studio installation directory. The commandline-script.sh contains:

./Talend-Studio-linux-gtk-x86_64 -nosplash -application org.talend.commandline.CommandLine -consoleLog -data commandline-workspace scriptFile $1

Publish script

We can now create a publish script called by the commandline-script.sh. This script performs the following action:

  1. Initialize the repository (local or remote, for this example, I use a remote repository)
  2. Logon on the project
  3. Publish a route

This script uses properties that we will filter with Maven using the resource plugin.

We place the script in src/scripts/commandline folder, with the publish name:

initRemote ${tac.location}
logonProject -pn ${talend.project} -ul "${tac.user}" -up ${tac.password}
publishRoute ${project.artifactId} -r "${repo.snapshot}" -u ${repo.user} -p ${repo.password} -pv ${project.version} -g ${project.groupId} -a ${project.artifactId}

We are now ready to call the commandline using Maven.

Maven deploy using commandline

To call the commandline with Maven, we use the exec-maven-plugin from codehaus.

Our Maven POM does:

  1. Disable the “default” deploy plugin.
  2. Use the maven-resource-plugin to filter the commandline scripts.
  3. Execute the commandline-script.sh at the deploy phase, using the filtered script files.

Finally, the Maven POM looks like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>net.nanthrax</groupId>
    <artifactId>MyRoute</artifactId>
    <version>0.1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>MyRoute</name>

    <properties>
        <talend.project>MAIN</talend.project>
        <tac.location>http://localhost:8080/org.talend.administrator</tac.location>
        <tac.user>jbonofre@talend.com</tac.user>
        <tac.password>foobar</tac.password>
        <commandline.location>/opt/talend/commandline</commandline.location>
        <commandline.executable>./commandline-script.sh</commandline.executable>
        <repo.release>http://localhost:8082/archiva/repository/repo-release/</repo.release>
        <repo.snapshot>http://localhost:8082/archiva/repository/repo-snapshot/</repo.snapshot>
        <repo.user>admin</repo.user>
        <repo.password>foobar</repo.password>
    </properties>

    <repositories>
        <repository>
            <id>archiva.repo.release</id>
            <name>Aarchiva Artifact Repository (release)</name>
            <url>${repo.release}</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>archiva.repo.snapshot</id>
            <name>Archiva Artifact Repository (snapshot)</name>
            <url>${repo.snapshot}</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
    </repositories>

    <build>
        <resources>
            <resource>
                <directory>${project.basedir}/src/scripts/commandline</directory>
                <filtering>true</filtering>
                <includes>
                    <include>**/*</include>
                </includes>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-deploy-plugin</artifactId>
                <version>2.7</version>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>1.2.1</version>
                <executions>
                    <execution>
                        <id>export</id>
                        <phase>deploy</phase>
                        <goals>
                            <goal>exec</goal>
                        </goals>
                        <configuration>
                            <executable>${commandline.executable}</executable>
                            <workingDirectory>${commandline.location}</workingDirectory>
                            <arguments>
                                <argument>${project.build.directory}/classes/publish</argument>
                            </arguments>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

By leveraging the commandline, our Maven POM does:

  1. Checkout the Talend metadata repository.
  2. Generate the code using the metadata.
  3. Compile the generated code.
  4. Create the artifact, the Karaf features XML, and deploy on the Archiva repository.

Nightly builds using Jenkins

Now that we have our Maven POM, we can creat the job in Jenkins. Like this, we will have nightly builds including the latest changes performed by the developers.

Of course, we can “couple” this deploy phase with the unit tests that we did in the first article. We can merge both in the same Maven POM.

It’s interesting to note that we can leverage Maven features (like pluginManagement, profiles, etc), and especially reactor (multiple Maven modules) with this, allowing to build a set of jobs, routes, or services in a row.

Talend ESB Continous Integration, part1: Using Camel Test Kit

October 17, 2013 Posted by jbonofre

Introduction

In this serie of articles, I will show how to setup a Continuous Integration solution mixing Talend ESB tools, Maven, and Jenkins.

The purpose is to decouple the design (performed in the studio), the tests (both unit and integration tests), and the deployment of the artifacts.

The developers that use the studio should never directly upload to the Maven repository (Archiva in my case).

I propose to implement the following steps:

  1. the developers use the studio to design their routes: the metadata (used to generate the code) are stored in the subversion. The studio “only” checkouts and commits on subversion: it never directly upload to the artifact repository.
  2. a continuous integration tool (Jenkins in my case) uses Maven. The Maven POM leverages the Talend commandline (a studio without the GUI) to checkout, generate the code, and publish to the artifact repository. The Maven POM is also used to execute unit tests, eventually integration tests, and cleanly cut off the releases.
  3. the Talend runtimes (Karaf) deploy (using JMX or Talend Administration Center) the routes from the artifact repositories.

With this approach, we have a much cleaner isolation of concerns and tasks.

To demonstrate, I used Talend Enterprise edition 5.3.1, but you can do the same using the Open Studio edition.

In this first part, I will show how to use the Camel Test Kit with routes designed by the Talend studio, and how to periodically execute these tests using Jenkins.
To simplify, I will directly publish the routes on Archiva (my artifacts repository) using the studio. As I said before, it should not be done this way: only the Talend commandline (called from Jenkins) should be able to upload to the artifacts repository.

Camel Test Kit benefits

There are multiple reaons to use the Camel Test Kit and to write unit tests “outside” of the Talend studio:

  • it’s a step forward to continuous integration: the unit tests can be periodically executed by Jenkins. Thanks to that, it’s a good way to detect regressions: some changes performed in the studio may break the routes and so the unit tests.
  • it allows you to test components that you can’t run in the studio: for instance, you can’t run routes using vm component directly in the studio (you can but it’s not really useful). Thanks to mock and producer template, we can test the route and the vm endpoints.
  • it allows you to test even if you don’t have the actual dependent systems: in your route, you will probably use endpoints like CXF (for WebServices), file, FTP, JMS/ActiveMQ, etc. It’s not always easy to test route using such components directly in the studio: you may not want to really communicate with a FTP server, or to create local filesystem, etc. The Camel Test Kit allows you to mock some endpoints and mimic the actual endpoint without having really it.
  • Simulate errors: most of the time, in the studio, you test the “happy path”. But, especially when you use “custom” error handling, you may want to see if your error hanlder reacts correctly. The mock component is a good way to generate errors.

Talend Studio for the design

In the Talend Studio, using the Mediation perspective, you can design Camel routes.

The Studio should be used only for the design: not the deployment, the tests, or the releases (even if you can do all in the studio ;)).

Using the Mediation perspective, I created a simple route:

Talend Studio screenshot

We have two routes in this design:

  • from(“vm:start”).to(“log:cLog1”).to(“direct:start”)
  • from(“direct:start”).to(“log:cLog2”).choice().when(simple(“${in.header.type} == ‘region'”)).to(“vm:region”).otherwise().to(“vm:zipcode”)
  • a DeadLetter ErrorHandler which catch any exception and send to vm:errorhandling

The first step is to publish the route on the artifact repository (Apache Archiva or Sonatype Nexus for instance). You configure the location of the artifact repository in the Talend preferences of the studio.

A right click on the route show a menu containing the “Publish” button: it will upload (deploy) the route to the artifact repository. The “Publish” button is available only in Enterprise edition. If you use the OpenStudio edition, you have to export the route as a kar file, explode the kar file and use the Maven deploy plugin to upload to the artifact repository.

The publish window allows you to define the Maven groupId, artifactId, version, etc.

The route jar file (which is an OSGi bundle) contains two “special jar files” that you have to upload to the artifact repository. This step has to be done only one time per Talend studio version. The jar files are located into the lib folder of the route jar, so you can do:

jar xvf ShowUnitTest-0.1.0-SNAPSHOT.jar lib
mvn deploy:deploy-file -DgroupId=org.talend -DartifactId=systemRoutines -Dversion=5.3.1 -Dfile=lib/systemRoutines.jar -Dpackaging=jar -Durl=http://tadmin:tadmin@localhost:8082/archiva/repository/repo-release/
mvn deploy:deploy-file -DgroupId=org.talend -DartifactId=userBeans -Dversion=5.3.1 -Dfile=lib/userBeans.jar -Dpackaging=jar -Durl=http://tadmin:tadmin@localhost:8082/archiva/repository/repo-release/

NB: if systemRoutines artifact doesn’t really change, the userBeans artifact should be uploaded “per route” and updated when you modify or create a new bean that you use in your route.

We have now all the artifacts on our artifact repository to create the unit tests.

Using Camel Test Kit

The Camel Test (provided by the camel-test.jar) provides:

  • JUnit extensions: you can create very easily unit tests by extend the CamelTestSupport and CamelSpringTestSupport abstract classes
  • Producer/Consumer template: you can “inject” exchanges/messages at any point of a route. It allows you to test exactly a route at a given point, and create messages which mimic the actual messages
  • Mock component: you can mock actual endpoints, simulate errors, and set expectations on the mock.

Now, we can create a Maven project that will gather our unit tests. We start by creating the POM:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>utests</artifactId>
    <version>0.1.0-SNAPSHOT</version>

    <properties>
        <camel.version>2.10.4</camel.version>
        <talend.version>5.3.1</talend.version>
        <commandline.path>/home/jbonofre/Talend/Talend-Studio-r104014-V5.3.1</commandline.path>
    </properties>

    <repositories>
        <repository>
            <id>local.archiva.snapshot</id>
            <name>Local Maven Archiva for Snapshots</name>
            <url>http://localhost:8082/archiva/repository/repo-snapshot/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>local.archiva.release</id>
            <name>Local Maven Archiva for Releases</name>
            <url>http://localhost:8082/archiva/repository/repo-release/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.example</groupId>
            <artifactId>ShowUnitTest</artifactId>
            <version>${project.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Talend dependencies -->
        <dependency>
            <groupId>org.talend</groupId>
            <artifactId>systemRoutines</artifactId>
            <version>${talend.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.talend</groupId>
            <artifactId>userBeans</artifactId>
            <version>${talend.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Camel dependencies -->
        <dependency>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-test-spring</artifactId>
            <version>${camel.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-jdk14</artifactId>
            <version>1.6.6</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

</project>

On this Maven POM, we can note:

  • We define the Maven artifact repositories (in my case Apache Archiva) location in the <repositories> element.
  • The first <dependencies> is the route jar file itself.
  • We define the “Talend” dependencies, especially systemRoutines and userBeans.
  • Finally, we define the “Camel” dependencies: the Camel Test Kit itself, and a slf4j provider to have the log messages during the execution of the unit tests.

We are now ready to write the unit test itself. To do so, we create the src/test/java folder. In this folder, we create directly the unit test class. In my case, I create the ShowUnitTestTest class:

package test.showunittest_0_1;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.camel.test.CamelTestSupport;
import org.junit.Test;

import java.io.IOException;

/**
 * Test on the ShowUnitTest routes
 */
public class ShowUnitTestTest extends CamelTestSupport {

    @Override
    public String isMockEndpoints() {
        return "*";
    }

    @Override
    protected RouteBuilder createRouteBuilder() throws Exception {
        ShowUnitTest route = new ShowUnitTest();
        route.initUriMap();
        return route;
    }

    @Test
    public void testRegionRouting() throws Exception {
        MockEndpoint regionMock = getMockEndpoint("mock:vm:region");
        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");

        // we expect to receive one message on the JMS queue:region, and no message on the JMS queue:zipcode
        regionMock.setExpectedMessageCount(1);
        zipcodeMock.setExpectedMessageCount(0);

        // send a message with the region header
        template.sendBodyAndHeader("vm:start", "Foobar", "type", "region");

        // check the assertion
        assertMockEndpointsSatisfied();
    }

    @Test
    public void testZipCodeRouting() throws Exception {
        MockEndpoint regionMock = getMockEndpoint("mock:vm:region");
        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");

        regionMock.setExpectedMessageCount(0);
        zipcodeMock.setExpectedMessageCount(1);

        // send a message with the region header
        template.sendBodyAndHeader("vm:start", "Foobar", "type", "zipcode");

        // check the assertion
        assertMockEndpointsSatisfied();
    }

    @Test
    public void testNoHeaderRouting() throws Exception {
        MockEndpoint regionMock = getMockEndpoint("mock:vm:region");
        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");

        regionMock.setExpectedMessageCount(0);
        zipcodeMock.setExpectedMessageCount(1);

        // send a message with the region header
        template.sendBody("vm:start", "Foobar");

        // check the assertion
        assertMockEndpointsSatisfied();
    }

    @Test
    public void testErrorHandler() throws Exception {
        MockEndpoint zipcodeMock = getMockEndpoint("mock:vm:zipcode");
        MockEndpoint errorhandlingMock = getMockEndpoint("mock:vm:errorhandling");

        // raise an exception at the cLog processor step
        zipcodeMock.whenAnyExchangeReceived(new Processor() {
            @Override
            public void process(Exchange exchange) throws Exception {
                throw new IOException("Test Error Handler");
            }
        });

        // the error handling route should have received a message
        errorhandlingMock.setExpectedMessageCount(1);

        // send a message, it should call the error handler
        template.sendBody("vm:start", "Foobar");

        // check the assertion
        assertMockEndpointsSatisfied();
    }

}

In this class, we said to Camel to be able to mockup any endpoint (overriding the isMockEndpoints() method). To find the Camel URI generated by the studio, you can switch to the source tab in the studio and take a look on the initUriMap() method: this method contains all URI of the route endpoints.

We also override the createRouteBuilder() method to load the route designed in the studio. To do it, we create the route object, call the initUriMap() method, and finally return this object.

Of course, we created four different tests:

  • the testRegionRouting() tests the route, and especially the content base router when setting the header ‘type’ to ‘region’. We mock up the vm:region and vm:zipcode endpoints. We use the producer template to send at the vm:start endpoint step.
  • the testZipCodeRouting() tests the route, and especially the content base router when setting the header ‘type’ to ‘zipcode’.
  • the testNoHeaderRouting() tests the route, and especially the content base router when the header ‘type’ is not set.
  • the testErrorHandler() tests the route, simulate an error to check if the error handler reacts correctly.

Special cases: JMS, context variables, cTalendJob,…

Depending of components that you use, Talend Studio manipulates the CamelContext for you. For instance, when you use the cJMS component, you have to create a cJMSConnectionFactory.

The Talend Studio generates the code to handle the CamelContext and “inject” the JMS connection factory into the Camel JMS component.

Unfortunately, it’s a done in a private method, so not callable directly from the test createRouteBuilder method (as we do with the initUriMap() method).

The workaround is to create the CamelContext in the test and copy the code generated by the studio here. Here’s an example how to use the “custom” JMS component (as the Studio does):

    @Override
    protected CamelContext createCamelContext() throws Exception {
        DefaultCamelContext camelContext = (DefaultCamelContext) super.createCamelContext();

        RouteName_Registry contextRegister = new RouteName_Registry(camelContext.getRegistry());
        camelContext.setRegistry(contextRegister);

        javax.jms.ConnectionFactory jmsConnectionFactory = new org.apache.activemq.ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
        camelContext.addComponent("cJMSConnectionFactory1", org.apache.camel.component.jms.JmsComponent.jmsComponent(jmsConnectionFactory));

        return camelContext;
    }

Another typical use case is about the Talend context variables. Thanks to the Talend Studio, you can define context variables that you can use in any place of your route.

In the route definition (in the studio), you can create multiple contexts.

In the unit test, you can decide which context you want to use for the test. To do so, you can use the readContextValues() method when you instanciate the route:

    @Override
    public RouteBuilder createRouteBuilder() throws Exception {
        RouteToTestName route = new RouteToTestName();
        route.readContextValues("Default");
        route.initUriMap();
        return route;
    }

Another feature provided in Talend ESB is that you can call Data Integration jobs in your Camel routes. To do so, Talend ESB registers a Camel component with “talend:” as URI prefix.
You have to load this component in the test CamelContext:

        TalendComponent talendComponent = new TalendComponent();
        camelContext.addComponent("talend", talendComponent);

Complete test

To summarize, if we take a look on the required resources, we need two things.

The first thing is a Maven POM containing all the resources and artifacts required for the route execution. Here’s a complete example:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
 
  <modelVersion>4.0.0</modelVersion>

  <groupId>org.example</groupId>
  <artifactId>test</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>My Route Test</name>

  <properties>
    <talend.version>5.3.1</talend.version>
    <camel.version>2.10.4</camel.version>
  </properties>

  <dependencies>
    <!-- Route itself -->
    <dependency>
      <groupId>org.example</groupId>
      <artifactId>MyRoute</artifactId>
      <version>1.0-SNAPSHOT</version>
      <scope>test</scope>
    </dependency>
    <!-- Eventually job used in the route (via cTalendJob) -->
    <dependency>
      <groupId>org.example</groupId>
      <artifactId>MyRouteJob</artifactId>
      <version>1.0-SNAPSHOT</version>
      <scope>test</scope>
    </dependency>

    <!-- Eventually Camel components used in the route -->
    <!-- camel-ftp -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-ftp</artifactId>
      <version>${camel.version}</version>
      <scope>test</scope>
    </dependency>
    <!-- camel-http -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-http</artifactId>
      <version>${camel.version}</version>
      <scope>test</scope>
    </dependency>
    <!-- camel-xmljson -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-xmljson</artifactId>
      <version>${camel.version}</version>
      <scope>test</scope>
    </dependency>
    <!-- camel-cxf -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-cxf</artifactId>
      <version>${camel.version}</version>
      <scope>test</scope>
    </dependency>
    <!-- camel-jms and dependencies -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-jms</artifactId>
      <version>${camel.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.geronimo.specs</groupId>
      <artifactId>geronimo-jms_1.1_spec</artifactId>
      <version>1.1.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.activemq</groupId>
      <artifactId>activemq-core</artifactId>
      <version>5.7.0</version>
      <scope>test</scope>
    </dependency>
    <!-- camel-mail and mock-javamail -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-mail</artifactId>
      <version>${camel.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.jvnet.mock-javamail</groupId>
      <artifactId>mock-javamail</artifactId>
      <version>1.7</version>
      <scope>test</scope>
      <exclusions>
        <exclusion>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <!-- Talend dependencies -->
    <dependency>
      <groupId>org.talend</groupId>
      <artifactId>systemRoutines</artifactId>
      <version>${talend.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.talend</groupId>
      <artifactId>userBeans</artifactId>
      <version>${talend.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.talend.camel</groupId>
      <artifactId>camel-talendjob</artifactId>
      <version>${talend.version}</version>
      <scope>test</scope>
    </dependency>

    <!-- Camel dependencies -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-test-spring</artifactId>
      <version>${camel.version}</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-jdk14</artifactId>
      <version>1.6.6</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

</project>

The second resource is the unit test itself (in src/test/java). Here’s a complete example, including registration of “custom” JMS component, Talend component, some custom beans registration:

package main.myroute_1_0;

import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.mock.MockEndpoint;
import org.apache.camel.impl.DefaultCamelContext;
import org.apache.camel.test.junit4.CamelTestSupport;
import org.junit.Test;
import org.talend.camel.TalendComponent;

import java.util.*;

public class MyRoute_Test extends CamelTestSupport {

    @Override
    public String isMockEndpoints() {
        return "cJMSConnectionFactory1:*";
    }

    @Override
    public RouteBuilder createRouteBuilder() throws Exception {
        MyRoute route = new MyRoute();
        route.readContextValues("Default");
        route.initUriMap();
        return route;
    }

    @Override
    public CamelContext createCamelContext() throws Exception {
        DefaultCamelContext camelContext = (DefaultCamelContext) super.createCamelContext();
        MyRoute_Registry contextRegister = new MyRoute_Registry(camelContext.getRegistry());

        // custom MyBean
        beans.MyBean myBean = new beans.MyBean();
        contextRegister.register("myBean", myBean);

        // CXF_PAYLOAD_HEADER_FILTER bean required by cxf endpoint generated by the Studio
        CxfConsumerSoapHeaderFilter cxfConsumerSoapHeaderFilter = new CxfConsumerSoapHeaderFilter();
        registry.register("CXF_PAYLOAD_HEADER_FILTER", cxfConsumerSoapHeaderFilter);

        camelContext.setRegistry(contextRegister);

        // "custom" JMS component as generated by the Studio
        javax.jms.ConnectionFactory jmsConnectionFactory = new  org.apache.activemq.ActiveMQConnectionFactory("vm://localhost?broker.persistent=false");
        camelContext.addComponent("cJMSConnectionFactory1", org.apache.camel.component.jms.JmsComponent.jmsComponent(jmsConnectionFactory));

        // Talend component
        TalendComponent talendComponent = new TalendComponent();
        camelContext.addComponent("talend", talendComponent);        

        return camelContext;
    }

    @Test
    public void testRouteWithMyHeader() throws Exception {
        MockEndpoint queueMock = getMockEndpoint("mock:cJMSConnectionFactory1:queue:OUTPUT_QUEUE");

        queueMock.setMinimumExpectedMessageCount(1);

        String testHeader = "MyHeader";

        // construct the body
        List<String> body = new ArrayList<String>();
        body.add("foo");
        body.add("bar");

        Map<String, Object> camelHeaders = new HashMap<String, Object>();
        camelHeaders.put("MyHeader", testHeader);
        camelHeaders.put("CamelFileName", "/tmp/foobar.csv");
        template.sendBodyAndHeaders("cJMSConnectionFactory1:queue:INPUT_QUEUE", body, camelHeaders);

        assertMockEndpointsSatisfied();

        assertTrue(queueMock.getExchanges().get(0).getIn().getBody() instanceof List<String>);
    }

    class CxfConsumerSoapHeaderFilter extends org.apache.camel.component.cxf.common.header.CxfHeaderFilterStrategy {
        public boolean applyFilterToCamelHeaders(String headerName, Object headerValue, org.apache.camel.Exchange exchange) {
            if (org.apache.cxf.headers.Header.HEADER_LIST.equals(headerName)) {
                return true;
            }
            return super.applyFilterToCamelHeaders(headerName, headerValue,
                    exchange);
        }

        public boolean applyFilterToExternalHeaders(String headerName, Object headerValue, org.apache.camel.Exchange exchange) {
            if (org.apache.cxf.headers.Header.HEADER_LIST.equals(headerName)) {
                return true;
            }
            return super.applyFilterToExternalHeaders(headerName, headerValue,
                    exchange);
        }
    }

}

Integration with Jenkins

Now, we can periodically execute these unit tests.

To do so, I installed Jenkin in a Tomcat, and setup the Maven POM:

screen2

screen3

screen4

Next step

Unit test is the first step to a complete continuous integration process using Talend.

In the next article, I will deal with the usage of the Talend commandline via Maven, and integrate this in Jenkins.

Coming in Karaf 3.0.0: subshell and completion mode

October 10, 2013 Posted by jbonofre

If you are a Karaf user, you probably know that Karaf is very extensible: you can add features in Karaf to provide new functionalities.

For instance, you can install Camel, ActiveMQ, CXF, Cellar, etc in your Karaf runtime.

Most of these features provide new commands:
– Camel provides camel:* commands to manipulate the Camel Context, the routes, etc.
– CXF provides cxf:* commands to manipulate the CXF buses, endpoints, etc.
– ActiveMQ provides activemq:* commands to manipulate brokers.
– Cellar provides cluster:* commands to manipulate cluster nodes, cluster groups, etc.
– and so on

If you install some features like this, the number of commands available in the Karaf shell console is really impressive. And it’s not always easy to find the one that we need.

That’s why subshell support has been introduced.

Subshell

Karaf now uses commands scope to create “on the fly” a subshell: the commands are grouped by subshell. As you will see later, depending of the completion mode that you will use, you will be able to see the commands only in the current subshell, and change from one subshell to another.

Let take an exemple. In Karaf itself, we have commands to manipulate bundle and commands to manipulate feature, for instance:

  • bundle:list list the bundles
  • bundle:start start bundles
  • bundle:stop stop bundles
  • feature:list list the Karaf features
  • feature:repo-list list the Karaf features repositories

In previous Karaf version, to list bundles and features, you did something like this:


karaf@root> osgi:list
...
karaf@root> features:list
...

In Karaf 3.0.0, you can still do the same (just using the new name of the commands):


karaf@root()> bundle:list
...
karaf@root()> feature:list
...

But you can also use subshell:


karaf@root()> bundle
karaf@root(bundle)> list
...
karaf@root(bundle)> feature
karaf@root(feature)> list
...

or


karaf@root()> bundle
karaf@root(bundle)> list
...
karaf@root(bundle)> exit
karaf@root()> feature
karaf@root(feature)> list
...

We can note several things here:

  • You have commands to go into a subshell. These commands are created on the fly by Karaf using the scope of the commands. Here, we use the bundle and feature commands to go into the bundle and feature subshell.
  • You can see your current subshell location directly in the prompt:

    karaf@root(bundle)>

    We can see here that we are in the bundle subshell.
  • We can switch directly from one subhsell to another using the subshell command:

    karaf@root(bundle)> feature
    karaf@root(feature)>
  • You have a new exit command to get out from the current subhsell and return to the root level.

You have the choice between different completion mode, depending the behaviour that you prefer.

Completion Mode

The completion mode defines the behaviour of the TAB key to complete commands.

You have three different modes available:

  • GLOBAL
  • FIRST
  • SUBSHELL

You can define your default completion mode using the completionMode property in etc/org.apache.karaf.shell.cfg file. By default, you have:


completionMode = GLOBAL

But, you can also change the completion mode “on the fly” (while using the Karaf shell console) using a new command: shell:completion:


karaf@root()> shell:completion
GLOBAL
karaf@root()> shell:completion FIRST
karaf@root()> shell:completion
FIRST

shell:completion can inform you about the current completion mode used. You can also provide the new completion mode that you want.

GLOBAL completion mode

GLOBAL completion mode is the default one in Karaf 3.0.0 (mostly for transition purpose).

GLOBAL mode doesn’t really use subshell: it’s the same behavior as in previous Karaf versions.

When you type the TAB key, whatever in which subshell you are, the completion will display all commands and all aliases:


karaf@root()> <TAB>
karaf@root()> Display all 273 possibilities? (y or n)
...
karaf@root()> feature
karaf@root(feature)> <TAB>
karaf@root(feature)> Display all 273 possibilities? (y or n)
...

FIRST completion mode

FIRST completion mode is an alternative to the GLOBAL completion mode.

If you type the TAB key on the root level subshell, the completion will display the commands and the aliases from all subshells (as in GLOBAL mode). However, if you type the TAB key when you are in a subshell, the completion will display only the commands of the current subshell:


karaf@root()> shell:completion FIRST
karaf@root()> <TAB>
karaf@root()> Display all 273 possibilities? (y or n)
...
karaf@root()> feature
karaf@root(feature)> <TAB>
karaf@root(feature)>
info install list repo-add repo-list repo-remove uninstall version-list
karaf@root(feature)> exit
karaf@root()> log
karaf@root(log)> <TAB>
karaf@root(log)>
clear display exception-display get log set tail

SUBSHELL completion mode

SUBSHELL completion mode is the real subshell mode (to be honest, it’s my prefered one ;)).

If you type the TAB key on the root level, the completion displays the subshell commands (to go into a subshell), and the global aliases. Once you are in a subshell, if you type the TAB key, the completion displays the commands of the current subshell:


karaf@root()> shell:completion SUBSHELL
karaf@root()> <TAB>
karaf@root()>
* bundle cl config dev feature help instance jaas kar la ld lde log log:list man package region service shell ssh system
karaf@root()> bundle
karaf@root(bundle)> <TAB>
karaf@root(bundle)>
capabilities classes diag dynamic-import find-class headers info install list refresh requirements resolve restart services start start-level stop
uninstall update watch
karaf@root(bundle)> exit
karaf@root()> camel
karaf@root(camel)> <TAB>
karaf@root(camel)>
backlog-tracer-dump backlog-tracer-info backlog-tracer-start backlog-tracer-stop context-info context-list context-start context-stop endpoint-list route-info route-list route-profile route-reset-stats
route-resume route-show route-start route-stop route-suspend

Tips

The “old” full qualified command names are still valid. So, you don’t have to change anything in your scripts, you can use:


karaf@root()> feature:install
karaf@root()> ssh:ssh
...

You have the choice: use the completion mode that you prefer, you can always change the mode when you want using the shell:completion command.

My preference is for the SUBSHELL completion mode. Using this mode, you don’t see a bunch of commands on the root level, just the subshell switch commands. I think it’s clear and straight forward. When you “extend” your Karaf runtime with a lot of additional features, it’s interesting to have commands grouped by subshell.

Coming in Karaf 3.0.0: JAAS users, groups, roles, and ACLs

October 4, 2013 Posted by jbonofre

This week I worked with David Booschaert. David proposed a patch for Karaf 3.0.0 to add the notion of groups and use ACL for JMX.

He posted a blog entry about that: http://coderthoughts.blogspot.fr/2013/10/jmx-role-based-access-control-for-karaf.html.

David’s blog is very detailed, mostly in term of implementation, the usage of the interceptor, etc. This blog is more about the pure end-user usage: how to configure group, JMX ACL, etc.

JAAS users, groups, and roles

Karaf uses JAAS for user authentication and authorisation. By default, it uses the PropertiesLoginModule, which use the etc/users.properties file to store the users.

The etc/users.properties file has the following format:


user=password,role

For instance:


karaf=karaf,admin

that means we have an user karaf, with password karaf, and admin for role.

Actually, the roles are not really used in Karaf: for instance, when you use ssh or JMX, Karaf checks the principal and credentials (basically the username and password) but it doesn’t really use the roles. All users have exactly the same permissions (basically all permissions): they can execute any shell commands, access to any MBeans and call any operation on these MBeans.

More over, the roles are “only” assigned by users. So, it means that we had to define the same roles list for two different users: it was the only way to assign the same roles list to different users.

So, in addition of users and roles, we introduced JAAS groups.

An user can be a member of a group or have roles assigned directly (as previously).

A groups has typically one or more roles assigned. An user that is part of that group will get these roles associated too.
Finally, an user has the union of the roles associated with his groups, togeher with his own roles.

Basically, the etc/users.properties file doesn’t change in terms of format. We just introduced a prefix to identify a group: _g_. An “user” with the _g_: prefix is actually a group.
So a group is defined as an user, and it’s possible to use a group in the list of roles of an user:


# users
karaf = karaf,_g_:admingroup
manager = manager,_g_:managergroup
other = other,_g_:managergroup,otherrole

#groups
_g_\:admingroup = admin,viewer,manager
_g_\:managergroup = viewer,manager

We updated the jaas:* shell commands to be able to manage groups, roles, and users:


karaf@root> jaas:realm-manage --realm karaf
karaf@root> jaas:group-add managergroup
karaf@root> jaas:group-add --help
karaf@root> jaas:user-add joe joe
karaf@root> jaas:group-add joe managergroup
karaf@root> jaas:group-role-add managergroup manager
karaf@root> jaas:group-role-add managergroup viewer
karaf@root> jaas:update
karaf@root> jaas:realm-manage --realm karaf
karaf@root> jaas:user-list
User Name | Group | Role
----------------------------------
karaf | admingroup | admin
karaf | admingroup | manager
karaf | admingroup | viewer
joe | managergroup | manager
joe | managergroup | viewer

Thanks to the groups, it’s possible to factorise the roles, and easily share different roles between the different users.

Define JMX ACLs based on roles

As explained before, the roles were not really used by Karaf. Especially, on the JMX layer, for instance, using jconsole with karaf user, you were able to see all MBeans and perform all operations.

So, we introduced the support of ACL (AccessLists) on JMX.

Now, whenever a JMX operation is invoked, the roles of the current user are checked against the required roles for this operation.

The ACL are defined using configuration files in the Karaf etc folder.

The ACL configuration file is prefixed with jmx.acl and completed with the MBean ObjectName that it applies to.

For example, to define the ACL on the MBean foo.bar:type=Test, you will create a configuration file named etc/jmx.acl.foo.bar.Test.cfg.
It’s possible to define more generic configuration file: on the domain (using jmx.acl.foo.bar.cfg) applied to all MBeans in this domain , or the most generic (jmx.acl.cfg) applied to all MBeans.

A very simple configuration file looks like:


# operation = roles
test = admin
getVal = manager,viewer

The configuration file supports different syntax to provide fine-grained operation ACL:

  • Specific match for the invocation, including arguments value:

    test(int)["17"] = role1

    It means that only users with role1 assigned will be able to invoke the test operation with 17 as argument value.
  • Regex match for the invocation:

    test(int)[/[0-9]/] = role2

    It means that only users with role2 assigned will be able to invoke the test operation with argument between 0 and 9.
  • Signature match for the invocation:

    test(int) = role3

    It means that only users with role3 assigned will be able to invoke test operation.
  • Method name match for the invocation:

    test = role4

    It means that only the users with role4 assigned will be able to invoke any test operations (whatever the list of arguments is).
  • A method name wildcard match:

    te* = role5

    It means that only the users with role5 assigned will be able to invoke any operations matching te* expression.

Karaf looks for required roles using the following process:

  1. The most specific configuration file is tried first (etc/jmx.acl.foo.bar.Test.cfg).
  2. If no matching definition is found in the specific configuration file, a more generic configuration file is inspected. In our case, Karaf will use etc/jmx.acl.foo.bar.cfg.
  3. If no matching definition is found in the domain specific configuration file, the most generic configuration file is inspected, etc/jmx.acl.cfg.

The ACLs work for any kind of MBeans including the one from the JVM itself. For instance, it’s possible to create etc/jmx.acl.java.lang.Memory.cfg configuration file containing:


gc = manager

It means that only the users with manager role assigned will be able to invoke the gc operation of the JVM Memory MBean.

It’s also possible to define more advanced configuration. For instance, we want that bundles with ID between 0 and 49 can be stopped only by an admin, the other bundles can be stopped by a manager. To do so, we create etc/jmx.acl.org.apache.karaf.bundle.cfg configuration file containing:


stop(java.lang.String)[/([1-4])?[0-9]/] = admin
stop = manager

etc/jmx.acl.cfg configuration file is a global configuration for the invocations of any MBean that doesn’t have a more specific ACL.
By default, we define this configuration:


list* = viewer
get* = viewer
is* = viewer
set* = admin
* = admin

We introduced a new MBean: org.apache.karaf:type=security,area=jmx.
The purpose of this MBean is to check whether the current user can access a certain MBean or invoke a specific operation on it.
This MBean can be used by management clients to decide whether to show certain MBeans or operations to the end user.

What’s next ?

Now, David and I are working on ACL/RBAC for:

  • shell commands: as we have ACL for MBeans, it makes sense to apply the same for shell commands.
  • OSGi services: the same can be applied to any OSGi service.

I would like to thank David for this great job. It’s a great addition to Karaf and a new very strong reason to promote Karaf 3 😉

Apache ActiveMQ 5.7, 5.9 and Master-Slave

October 3, 2013 Posted by jbonofre

With my ActiveMQ friends (especially Dejan and Claus), I’m working on ActiveMQ 5.9 next release.

Today, I focus on the HA with ActiveMQ, and especially Master-Slave configuration.

Update of the documentation

The first thing that I noticed is that the documentation is not really up to date.

If you do a search on the ActiveMQ website about Master-Slave, you will probably find these two links:

On the first link (about KahaDB), we can see a note “This is under review – and not currently supported”. It’s confusing for the users as this mechanism is the prefered one !
On the other hand, the second link should be flagged as deprecated as this mechanism is no more maintained.

I sent a message on the dev mailing list to updated these pages.

Lease Database Locker to avoid “dual masters”

In my test cases, I used a JDBC database backend (MySQL) for HA (instead of using KahaDB).

I have two brokers, that use the following configuration:


  <persistenceAdapter>
    <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" />
  </persistenceAdapter>

Broker1 starts, connects to MySQL, and acquires the lock. Broker1 is the master.

Broker2 starts, connects to MySQL, and waits for the lock (as the lock is hold by Broker1). Broker2 is a slave.

Now, I stop MySQL, for instance to do a cold backup. My backup is very fast, and I start MySQL server again, very quickly.

The lock is available in the database, so Broker2 get the lock, whereas Broker1 didn’t yet release it. So I’m in a bad situation where I have two “masters”.

ActiveMQ 5.7.0 introduced the change on locking strategies for shared storage master/slave topologies. Previously storage locking (and thus master election) was hard-coded directly in the particular store. So KahaDB has only the option to use shared file lock, while JDBC was using database lock.

Now, the storage locking is separated from the store, so you can implement your own locking strategies if necessary (or tune existing ones). Of course, every store has its own default locker.

In our previsou use case, to solve the “dual master” issue, we can use a new locker: the lease database locker.

To use it, we update the configuration of each locker like this:


  <persistenceAdapter>
    <jdbcPersistenceAdapter dataDirectory="${activemq.data}" dataSource="#mysql-ds" lockKeepAlivePeriod="5000">
      <locker>
        <lease-database-locker lockAcquireSleepInterval="10000"/>
      </locker>
    </jdbcPersistenceAdapter>
  </persistenceAdapter>

Lease database locker solves master/slave problem of the default database locker. Master acquires a lock only for a certain period and must extend it’s lease from time to time. Slave also checks periodically to see if the lease has expired. The lease can survive a db replica failover.

The lease based lock is acquired by blocking at start and retained by the keepAlivePeriod. To retain, the lease is extended by the lockAcquireSleepInterval, so in theory the master is always (lockAcquireSleepInterval-lockKeepAlivePeriod) ahead of the slave w.r.t the lease. It is imperative that lockAcquireSleepInterval > lockKeepAlivePeriod, to ensure the lease is always current.

In the simplest case, the clocks between master and slave must be in sync for this solution to work properly. If the clocks cannot be in sync, the locker can use the system time from the database CURRENT TIME and adjust the timeouts in accordance with their local variance from the db system time. If maxAllowableDiffFromDBTime is > 0 the local periods will be adjusted by any delta that exceeds maxAllowableDiffFromDBTime.

How to know who is the master ?

The “new” mechanism for Master/Slave is great and very easy to set up. You don’t really define who is the master, and who are the slaves. The first broker which get the lock will be the master.

So, a fair question is: how can I know which broker is the master ?

Actually, you already have the response on the JMX layer.

If you connect a JMX client (for instance jconsole) on the broker, and you take a look on the org.apache.activemq:BrokerName=Broker2,Type=Broker MBean, you can see the Slave attribute.

If Slave is true, it means that this broker is a slave. If Slave is false, it’s the master.

Another way to get this information is to use directly the activemq command with bstat argument (instead of JMX):


bin/activemq bstat
...
Connecting to pid: 563
BrokerVersion = 5.9-SNAPSHOT
TempLimit = 53687091200
Persistent = true
MemoryLimit = 67108864
TempPercentUsage = 0
SslURL =
StorePercentUsage = 0
TransportConnectors = {openwire=tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600}
Type = Broker
StompSslURL =
OpenWireURL = tcp://0.0.0.0:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600
Uptime = 3 minutes
DataDirectory = /home/jbonofre/broker2/data
StoreLimit = 107374182400
BrokerName = broker2
VMURL = vm://broker2
StompURL =
MemoryPercentUsage = 0
Slave = true

You can see the Slave attribute there.

If you want to “script” this and get only the Slave attribute, you can use the query argument:


bin/activemq query --objname Type=Broker --view Slave
...
Slave = true