Posts Tagged: ‘apache’

Apache Karaf 2.3.0 released !

October 16, 2012 Posted by jbonofre

Waiting for Karaf 3.0.0, we worked hard in the Karaf team to provide Apache Karaf 2.3.0.

The Karaf 2.2.x branch is now only in maintenance mode: it means that no new features will be implemented in this branch, only major bug fixes.

The new “stable” branch is now Karaf 2.3.x which is a perfect transition branch between Karaf 2.2.x (heavely used) and the future Karaf 3.x (which should arrive very soon).

What’s new in this 2.3.0 release:

* OSGi r4.3: Karaf 2.2.x branch was powered by OSGi frameworks implementing OSGi r4.2 norm. Karaf 2.3.0 is now powered by the new OSGi r4.3 framework (Apache Felix 4.0.3 and Equinox 3.8.x), for both OSGi core and compendium. It provides new features like weaving, etc.
* Aries Blueprint 1.0.x: Karaf 2.3.0 uses the new Aries Blueprint version at different level (core, JMX, etc).
* Update to ASM 4.0: in order to work with Aries proxies, we updated to new ASM bundle. We also provided configuration that allows you to enable/disable weaving.
* OSGi Regions and SCR support: Karaf 2.3.0 provides both Regions and SCR support.
* JMX improvement: the previous MBeanRegistrer from Karaf 2.2.x has been removed to be replaced by Aries JMX. It allows an easier way to integrate MBeans by registering OSGi services. The MBeans have been improved to provide new operations and options corresponding with what you can do using the shell commands.
* Complete itest framework: Karaf 2.3.0 provides a new tool: Karaf exam. This tool provides a framework to very easily implements integration tests. It’s able to download and bootstrap a Karaf version on which you can run your commands, deploy your features and bundles, etc. It allows you to run a complete integration tests chain from Maven.
* Dependencies upgrade: a lot of dependencies have been updated. Karaf 2.3.0 uses Pax Logging 1.7.0 including bug fixes and SLF4J 1.7.1 support, new Pax Web and Jetty version for the web container, new JLine, SSHD and Mina versions which especially fix weird behavior on Windows for certain keys, etc.
* KAR improvements: if Karaf 3.x will provide a lot of enhancements around the KAR files, Karaf 2.3.0 already provides fixes in the KAR lifecycle.
* JAAS commands improvements: the jaas:* commands have been enhanced to allow you a fine-grained management of the realms and login modules.

You can find the Karaf 2.3.0 content details on the Release Notes.

The Karaf team is proud to provide this release to you. We hope you will enjoy it !

What’s new in Karaf 2.2.4 ?

October 17, 2011 Posted by jbonofre

Apache Karaf 2.2.4 has been released this week end.

You can take a look on the release notes.

More than a bug fixes release, this version includes several new features and enhancements.

Command aliases

Previously, the osgi:* commands gathered different kind of usage: osgi:start, osgi:stop, etc are related to bundles, osgi:shutdown is related to container itself, and osgi:ls is OSGi service related.

To be clearer for the users, new aliases have been introduced.

NB: on Karaf trunk (future 3.0.0 release), the commands have been fully renamed (not aliased).

system:*

The system:* commands are related to the Karaf container itself:

  •  system:shutdown is equivalent to osgi:shutdown and shutdown the Karaf container
  •  system:start-level is equivalent to osgi:start-level and defines the default start-level to consider bundles as system bundles

services:*

The services:* commands are related to OSGi services:

  • services:list is equivalent to ls and lists the OSGi services available

bundles:*

The bundles:* commands are related to OSGi bundles:

  • bundles:list is equivalent to osgi:list and lists the OSGi bundles
  • bundles:install is equivalent to osgi:install and installs OSGi bundles
  • bundles:uninstall is equivalent to osgi:uninstall and uninstalls OSGi bundles
  • bundles:start is equivalent to osgi:start and starts OSGi bundles
  • bundles:stop is equivalent to osgi:stop and stops OSGi bundles
  • bundles:header is equivalent to osgi:headers and displays OSGi bundles headers
  • bundles:info is equivalent to osgi:info and displays bundle.info file
  • bundles:refresh is equivalent to osgi:refresh and refresh OSGi bundles
  • bundles:resolve is equivalent to osgi:resolve and resolves OSGi bundles
  • bundles:restart is equivalent to osgi:restart and restarts OSGi bundles
  • bundles:start-level is equivalent to osgi:start-level and defines the OSGi bundles start level
  • bundles:update is equivalent to osgi:update and updates OSGi bundles

New MBeans

To be able to monitor and administrate with your own tool (Nagios, Apache Kalumet, your own JMX client), Karaf now provides a set of new MBeans.

Bundles MBean

The bundles MBean (org.apache.karaf:type=bundles) allows you to view and change the bundles. As with bundles:* shell commands, you can select bundles using bundle ID (for instance 20), bundle ID range (for instance 21-25 means bundle ID 21, 22, 23, 24, 25), bundle name and version including support of regex (for instance mybunde, mybundle/version, myb*).

  • list() operation gives your a tabular view about all bundles deployed, including the ID, the location, and the current status of the bundle
  • getStartLevel() and setStartLevel() allows you to get/set the start level of bundles
  • refresh() allows to refresh bundles
  • update() allows to update bundles
  • resolve() allows to resolve bundles
  • restart() allows to restart bundles
  • install() installs bundles from an URL. The URL could be a “classic” one (file: or http:), a Maven URL (mvn:), an OBR URL (obr:)
  • start() starts bundles
  • stop() stops bundles
  • uninstall() uninstalls bundles

Config MBean

The config MBean (org.apache.karaf:type=config) allows you to view and change the OSGi configuration (ConfigAdmin layer). As with config:* shell commands, the changes are flushed into the /etc configuration file.

  • list() gives you all configuration PID present in the Karaf container
  • create() creates a new configuration PID
  • delete() deletes a configuration PID
  • proplist() list properties for a given configuration PID
  • propdel() deletes a property from a given configuration PID
  • propappend() appends content to a property in the given configuration PID
  • propset() set/add a property in the given configuration PID

Dev MBean

The dev MBean (org.apache.karaf:type=dev) allows you view the OSGi framework in use and set the framework options:

  • framework() gives the current OSGi framework in use (Felix or Equinox)
  • frameworkOptions() allows you to define OSGi framework options (debug mode, framework to use)
  • restart() allows you to restart the Karaf container

Log MBean

The log MBean (org.apache.karaf:type=log) allows you to view/change the log level currently used (globally or for a given logger):

  • set() sets the log level to use (for the root logger, or for a given logger)
  • get() gives the log level in use (for the root logger, or for a given logger)

OBR MBean

The OBR MBean (org.apache.karaf:type=obr) allows you to manipulate the OBR client:

  • listUrls() list the OBR server URL (containing the repository.xml file) register in the Karaf OBR
  • addUrl() adds a new OBR server in the OBR client
  • removeUrl() removes an OBR server from the OBR client
  • refreshUrl() updates the OBR information in the OBR client
  • list() lists the bundles available via the OBR client
  • deploy() deploys a bundle (and its dependency bundles) from the OBR

Packages MBean

The packages MBean (org.apache.karaf:type=packages) allows you to get information about exported and imported packages (and the related bundles):

  • exportedPackages lists the exported package (globally or for a given bundle)
  • importedPackages lists the imported package (globally or for a given bundle)

Services MBean

The services MBean (org.apache.karaf:type=services) allows you to view the OSGi services available:

  • list() lists the OSGi services available (globally, or only one in use, or provided by a given bundle)

System MBean

The system MBean (org.apache.karaf:type=system) allows you to manipulate the Karaf container itself:

  • shutdown() shutdown the Karaf container

Web MBean

The web MBean (org.apache.karaf:type=web) allows you to see the web bundles (web application) available:

  • list() gives you a tabular view of all web application deployed in Karaf (including the bundle and the context path)

Better KAR support

A KAR (Karaf ARchive) artifact is a zip file containing a feature XML descriptor and related bundles jar.

In previous Karaf version, we had issue with the Kar support. In Karaf 2.2.4:

  • the features Maven plugin provides a goal to create a KAR. Simply use a the create-kar goal with a feature.xml and you will have a ready to deploy KAR in the target directory
  • the KAR deployer was not able to correctly uncompress the KAR zip entries, it’s now fixed
  • the KAR deployer now automatically starts all features containing in a KAR file. It means that simply by dropping the KAR file in the deploy folder, Karaf will uncompress the KAR file, correctly creating a Karaf local repository for the KAR, and start all features containing in the KAR shipped feature XML

Spring features

To avoid Spring versions mismatch in others projects (such as Apache Camel, CXF, or ServiceMix), Karaf now provides a large set of Spring features:

  • spring
  • spring-web
  • spring-aspects
  • spring-dm
  • spring-dm-web
  • spring-instrument
  • spring-jdbc
  • spring-jms
  • spring-struts
  • spring-test
  • spring-orm
  • spring-oxm
  • spring-tx
  • spring-web
  • spring-web-portlet

It allows others projects to directly use these features (with a version range), and so, be less coupled to a Spring version.

Others

Some others minors things have been added:

  • the dev:watch command now supports multiple bundle as argument
  • the wrapper service provides a lib for Linux 64 bits platform
  • the J2SE-1.6 and J2SE-1.7 execution environment have been added
  • obr:deploy shell command now supports a -s option to automatically start bundles
  • Karaf JMX service could be configured to use SSL now
  • system:shutdown shell command displays the instance name (for confirmation)

Use Camel, CXF and Karaf to implement batches

August 23, 2011 Posted by jbonofre

Introduction

Apache Camel has not be designed to be used for implementing batch tasks.

For instance, if your Camel route has a consumer endpoint polling files in a directory, Camel will periodically and indefinitely monitor the folder and poll any new incoming files.
It’s not a batch behavior: in batch mode, we want to run the file polling on demand, at a certain time, launched by a batch scheduler like ControlM, $Universe or Tivoli Worksheet Scheduler.

However, there are several interesting points to use Camel for batch implementation. First, Camel provides a large set of components. A lot of batches read/write files, read from a JMS queues, write into JMS queues, etc. Usage of Camel components in a batch way is really valuable.
Second, Camel uses a DSL to describe the process executed by the routes. Especially, it supports “human readable” DSL like Spring XML or Blueprint XML. It means that it’s easy to review what the batch is doing, eventually change an endpoint definition, etc. Most of the time, batches are “black box”: you run it, and you only get a status code to know if it’s OK or not. With Camel, you have a look on the batch process.
Third, Camel is a highly plug and play framework. It means that it’s easy to replace an endpoint by another one. For instance, if your batch polls files in a folder currently, it’s very easy to change this to poll messages from a JMS queue. You don’t really have to re-implement the whole batch.

More over, tools like Talend ESB Studio provide an IDE to create and design your Camel routes.

In this article, we are going to see how to use Camel in a “batch way”.

Design

In fact, we are going to have two Camel routes:
– the first one is called “control”. This route will “expose” a REST service to start the batch. A bean in this route will be responsible to start the “batch” route.
– the second one is called “batch”. It’s the core implementation of our batch. It’s a “standard” route, but at the end, we have a processor that “stop” the route (to avoid to have the route up indefinitely). This route is not auto started as it will be controller by the first one.

It means that a simple HTTP client (like a browser or REST client) will start the batch, on-demand. Most of enterprise batch schedulers ship a component to make HTTP requests.

POM

Our batch will be packaged as an OSGi bundle. It will allow us to deploy the batch in an Apache Karaf OSGi container:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>net.nanthrax.examples</groupId>
  <artifactId>camel-batch</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>bundle</packaging>

  <properties>
    <camel.version>2.8.0</camel.version>
    <cxf.version>2.4.1</cxf.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-core</artifactId>
      <version>${camel.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-spring</artifactId>
      <version>${camel.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-cxf</artifactId>
      <version>${camel.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.cxf</groupId>
      <artifactId>cxf-rt-frontend-jaxrs</artifactId>
      <version>${cxf.version}</version>
    </dependency>
    <,dependency>
      <groupId>org.apache.cxf</groupId>
      <artifactId>cxf-rt-transports-http</artifactId>
      <version>${cxf.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.cxf</groupId>
      <artifactId>cxf-rt-transports-http-jetty</artifactId>
      <version>${cxf.version}</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.felix</groupId>
        <artifactId>maven-bundle-plugin</artifactId>
        <version>2.3.4</version>
        <extensions>true</extensions>
        <configuration>
          <instructions>
            <Bundle-SymbolicName>${project.artifactId}</Bundle-SymbolicName>
            <Require-Bundle>org.apache.cxf.bundle,org.apache.camel.camel-cxf,org.springframework.beans</Require-Bundle>
          </instructions>
        </configuration>
      </plugin>
    </plugins>
  </build>

</project>

In this POM, we can see:
– the packaging is an OSGi bundle. That’s why we use the Apache Felix maven-bundle-plugin. We name the bundle with the project artifactId, and we define Camel and CXF bundles as dependencies (Require-Bundle).
– in the dependency sets, we define the Camel components that we use (camel-core, camel-spring to use the Camel Spring XML DSL, and camel-cxf to use the CXF JAX-RS implementation) and the CXF dependency to be able to create a JAX-RS server.

Control route

The first Camel route is the “control” one. This route will bind a JAX-RS server, listening HTTP requests (consumer) and will start the “batch” route on-demand.

The route definition will be located in the META-INF/spring/routes.xml folder of our bundle. We use the Camel Spring XML DSL in this file:


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:cxf="http://camel.apache.org/schema/cxf"
  xsi:schemaLocation="
    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
    http://camel.apache.org/schema/cxf http://camel.apache.org/schema/cxf/camel-cxf.xsd
  ">

  <cxf:rsServer id="rsServer" address="http://localhost:9090/batch"
serviceClass="net.nanthrax.examples.camel.batch.impl.ControllerService"/&t;

  <camelContext xmlns="http://camel.apache.org/schema/spring">
    <route id="control">
      <from uri="cxfrs:bean:rsServer"/>
      <to uri="log:net.nanthrax.examples.camel.batch"/>
      <to uri="controllerBean"/>
    </route>
  </camelContext>

  <bean id="controllerBean" class="net.nanthrax.examples.camel.batch.impl.ControllerBean">
    <property name="routeId" value="batch"/>
  </bean>

</beans>

We use the Camel CXF to create the JAX-RS server (using <cxf:rsServer/gt;). This JAX-RS server will listen on the local machine on the 9090 port, and the context path is /batch.
To “describe” the REST service behavior, we define the ControllerService class in the serviceClass attribute.
The ControllerService class is just an “empty container”. The purpose is just to “describe” the REST service, not to process it:


package net.nanthrax.examples.camel.batch.impl;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;

/**
* REST service implementation of the Camel batch service.
*/
@Path("/")
public class ControllerService {

  @GET
  @Path("/start")
  @Produces("text/plain")
  public String startRoute() throws Exception {
    // nothing to do, it's just a wrapper
    return null;
  }

}

We can see the JAX-RS annotations:
– the ControllerService REST Path is /, it means directly bound to the JAX-RS server context path.
– the startRoute() method will accept GET HTTP method, on the context path /start and it will produce pure text (text/plain).

The process itself will be performed in the controllerBean:


package net.nanthrax.examples.camel.batch.impl;

import org.apache.camel.CamelContext;
import org.apache.camel.Handler;

/**
* Camel controller bean involved in the starting routed
*/
public class ControllerBean {

  private String routeId;

  public String getRouteId() {
    return this.routeId;
  }

  public void setRouteId(String routeId) {
    this.routeId = routeId;
  }

  @Handler
  public String startRoute(CamelContext camelContext) throws Exception {
    camelContext.startRoute(routeId);
    return "Batch " + routeId + " started.";
  }

}

We inject the Camel route ID of the batch route: “batch”. The CamelContext is automatically injected by Camel. This bean is quite simple, as it only starts the “batch” route.

Batch route

This route contains the “batch logic”. You can use any kind of routes, components, Enterprise Integration Patterns, etc provided by Camel. The only specific parts are:
– the autoStartup attribute set to false, to avoid to start the route automatically at context bootstrap
– the final processor which stop the route after processing.

We gather the two routes in the same META-INF/spring/routes.xml file:


<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:cxf="http://camel.apache.org/schema/cxf"
  xsi:schemaLocation="
    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
    http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
    http://camel.apache.org/schema/cxf http://camel.apache.org/schema/cxf/camel-cxf.xsd
  ">

  <cxf:rsServer id="rsServer" address="http://localhost:9090/batch"
  serviceClass="net.nanthrax.examples.camel.batch.impl.ControllerService"/>

  <camelContext xmlns="http://camel.apache.org/schema/spring">
    <route id="control">
      <from uri="cxfrs:bean:rsServer"/>
      <to uri="log:net.nanthrax.examples.camel.batch"/>
      <to uri="controllerBean"/>
    </route>
    <route id="batch" autoStartup="false">
      <from uri="file:/tmp"/>
      <to uri="file:output"/>
      <process ref="stopProcessor"/>
    </route>
  </camelContext>

  <bean id="controllerBean" class="net.nanthrax.examples.camel.batch.impl.ControllerBean">
    <property name="routeId" value="batch"/>
  </bean>

  <bean id="stopProcessor" class="net.nanthrax.examples.camel.batch.impl.StopProcessor">
    <property name="routeId" value="batch"/>
  </bean>

</beans>

In this example, the batch polls files in the /tmp folder, and copies into the output folder.

The StopProcessor is Camel processor (aka, it implements the Camel Processor interface). It stops the route after processing the incoming message (we inject the “batch” route ID using Spring):


package net.nanthrax.examples.camel.batch.impl;

import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;

/**
* A Camel processor which stop routes.
*/
public class StopProcessor implements Processor {

  private String routeId;

  public String getRouteId() {
    return this.routeId;
  }

  public void setRouteId(String routeId) {
    this.routeId = routeId;
  }

  public void process(Exchange exchange) throws Exception {
    CamelContext camelContext = exchange.getContext();
    // remove myself from the in flight registry so we can stop this route without trouble
    camelContext.getInflightRepository().remove(exchange);
    // stop the route
    camelContext.stopRoute(routeId);
  }

}

Deployment and execution

Now, we can build our OSGi bundle, simply using:


mvn clean install

In a fresh Apache Karaf instance, we have first to install the CXF and Camel features:


karaf@root> features:addurl mvn:org.apache.cxf.karaf/apache-cxf/2.4.1/xml/features
karaf@root> features:install cxf
karaf@root> features:addurl mvn:org.apache.camel.karaf/apache-camel/2.8.0/xml/features
karaf@root> features:install camel-spring
karaf@root> features:install camel-cxf

Now, we can install our bundle:


karaf@root> osgi:install -s mvn:net.nanthrax.examples/camel-batch/1.0-SNAPSHOT

Our bundle appears as “created”:


karaf@root> la|grep -i batch
[ 134] [Active ] [ ] [Started] [ 60] camel-batch (1.0.0.SNAPSHOT)

Using a simple browser, we can access to http://localhost:9090/batch/start. The route is started (as a batch) and we can see in the browser:


Batch batch started.

Conclusion

Even if the first Camel route purpose is to be up and running all the time, we can use it in a more “batch” way. It allows developers to use the large set of Camel components, and be able to use all Enterprise Integration Patterns. For instance, the batch needs to copy a file, and after send an e-mail and a message into a JMS queue, it’s very easy using a recipient list. You have to send to a target endpoint depending of the content of the message, no problem using a Content Based Router.

You can run such kind of batches in Talend ESB. It’s an interesting addition to the Talend Data Integration products (ETL jobs, MDM, DQ, etc).