What’s new in Apache Karaf Cellar 4.0.0 ?

September 22, 2015 Posted by jbonofre

Apache Karaf Cellar 4.0.0 release is now on vote, and hopefully, it should be available very soon.

This release is a major new release. More than just bug fixes, this release brings several refactoring and new features.

It’s time to take a tour in the new Cellar 4.0.0.

HTTP features

Cellar 4.0.0 brings new HTTP features.

HTTP load balancer

Cellar 4.0.0 provides a new feature: cellar-http-balancer.

The purpose is to use any nodes in a cluster group to access a web application, even if the web application is not actually deployed on the local node.

For instance, you have a cluster group containing four nodes. You deploy a web application on two nodes. So basically, to access your web application, you have to:

  • specify the address of one of two nodes where the web application is deployed in your browser
  • use a load balancer (mod_proxy_balancer, Cisco, Juniper, F5, whatever) to load balance on the two nodes. The drawback of this is that the load balancer is a single point of failure, and adding a new node providing the web application needs to update the load balancer configuration.

The cellar-http-balancer feature install a proxy in the nodes where the web application is not deployed. It means that you can use any node in the cluster group to access your web application, even if the application is not deployed there.

To illustrate this, let’s take a cluster with two nodes: node1 and node2.

On node1, we install http, http-whiteboard, and cellar feature:

karaf@node1()> feature:install http
karaf@node1()> feature:install http-whiteboard
karaf@node1()> feature:repo-add cellar 4.0.0
karaf@node1()> feature:install cellar

We now install the cellar-http-balancer feature on the cluster:

karaf@node1()> cluster:feature-install default cellar-http-balancer

Now, we install the webconsole only on node1:

karaf@node1()> feature:install webconsole

We can see the webconsole locally deployed using the http:list command:

karaf@node1()> http:list 
ID  | Servlet          | Servlet-Name    | State       | Alias               | Url
101 | KarafOsgiManager | ServletModel-2  | Undeployed  | /system/console     | [/system/console/*]
105 | InstancePlugin   | ServletModel-7  | Deployed    | /instance           | [/instance/*]
101 | ResourceServlet  | /res            | Deployed    | /system/console/res | [/system/console/res/*]
103 | GogoPlugin       | ServletModel-5  | Deployed    | /gogo               | [/gogo/*]
101 | KarafOsgiManager | ServletModel-11 | Deployed    | /system/console     | [/system/console/*]
102 | FeaturesPlugin   | ServletModel-9  | Deployed    | /features           | [/features/*]

Using a browser, we can access the webconsole using the http://localhost:8181/system/console URL.

But we can also see that the webconsole is also available on the cluster group:

karaf@node1()> cluster:http-list default
Alias               | Locations
/system/console/res |
/gogo               |
/instance           |
/system/console     |
/features           |

It means that I can use any node member of this cluster group to access the webconsole from node1 (I agree it’s not really interesting, but it’s just for the example, imagine that the webconsole is your own web application).

On node2, as I’m using the same machine, I have to use another port than 8181 for the HTTP service, so I’m adding etc/org.ops4j.pax.web.cfg file containing:


It means that the HTTP service on node2 will listen on port 8041.

Now, on node2, I install the http, http-whiteboard, and cellar features:

karaf@node2()> feature:install http
karaf@node2()> feature:install http-whiteboard
karaf@node2()> feature:repo-add cellar 4.0.0
karaf@node2()> feature:install cellar

As we installed the cellar-http-balancer feature on the default cluster group, it’s automatically installed on node2 when we enable Cellar.

Of course, on node2, we can see the HTTP applications available on the cluster, with node1 as location:

karaf@node2()> cluster:http-list default 
Alias               | Locations
/system/console/res |
/gogo               |
/instance           |
/system/console     |
/features           |

Now, if we take a look on the “local” HTTP applications on node2 (using http:list), we can see:

karaf@node2()> http:list 
ID  | Servlet                    | Servlet-Name   | State       | Alias               | Url
100 | CellarBalancerProxyServlet | ServletModel-3 | Deployed    | /gogo               | [/gogo/*]
100 | CellarBalancerProxyServlet | ServletModel-2 | Deployed    | /system/console/res | [/system/console/res/*]
100 | CellarBalancerProxyServlet | ServletModel-6 | Deployed    | /features           | [/features/*]
100 | CellarBalancerProxyServlet | ServletModel-5 | Deployed    | /system/console     | [/system/console/*]
100 | CellarBalancerProxyServlet | ServletModel-4 | Deployed    | /instance           | [/instance/*]

We can see the same URLs available on node2, providing by CellarBalancerProxyServlet. In your browser, if you access to http://localhost:8041/system/console you will access to the webconsole deployed on node1 whereas you use node2.

It means that the CellarBalancerProxyServlet act as a proxy. It does:

  1. Cellar HTTP Balancer is listening for HTTP servlets on local node. When a servlet is deployed locally to the node, it updates the servlets set on the cluster, and send a cluster event to the other nodes in the same cluster group.
  2. When a node receives a cluster event from the HTTP balancer, if a servlet with the same alias is not already deployed locally, Cellar HTTP balancer creates a CellarBalancerProxyServlet with the same alias.
  3. When the CellarBalancerProxyServlet receives a HTTP request, it retrieves the locations where the servlet is actually deployed from the cluster set, and randomly choose one, where the request is proxied.

HTTP sessions replication

Cellar 4.0.0 also brings support of HTTP session replication.

You don’t need any specific Cellar feature: just install http, http-whiteboard, and cellar features (in this order):

karaf@node1()> feature:install http
karaf@node1()> feature:install http-whiteboard
karaf@node1()> feature:repo-add cellar 4.0.0
karaf@node1()> feature:install cellar

To be able to use HTTP sessions replication, you have to implement serializable HTTP sessions in your web application.

Now, the only change in your application, is to add a specific filter. For that, you have to update the WEB-INF/web.xml like this:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"

                Name of the distributed map storing
                your web session objects
            <!-- How is your load-balancer configured? stick-session means all requests of
                a session is routed to the node where the session is first created. This is
                excellent for performance. If sticky-session is set to false, when a session
                 is updated on a node, entry for this session on all other nodes is invalidated.
                 You have to know how your load-balancer is configured before setting this
                 parameter. Default is true. -->
                Are you debugging? Default is false.


That’s all: if you deploy your web application on several nodes, then the sessions will be replicated and available on all node. It means that your clients will be able to transparently switch from a node to another.

Refactoring of the synchronizers

Cellar 4.0.0 also brings refactoring of the different synchronizers.

Now the synchronizers:

  • support new sync policies
  • send cluster events to the other nodes allowing a complete sync when a node joins a cluster group

If you take a look in the etc/org.apache.karaf.cellar.groups.cfg file, you will see:

default.bundle.sync = cluster
default.config.sync = cluster
default.feature.sync = cluster
default.obr.urls.sync = cluster
default.balanced.servlet.sync = cluster

Now, the synchronizers support the following policies:

  • disabled means that the synchronizer doesn’t do anything
  • cluster means that, first the synchronizer retrieves the state from the cluster, and update the node state if needed (pull first), and then, push the node state to cluster and send cluster events if needed (push after)
  • node means that, first the synchronizer pushed the state of the node to the cluster and send cluster events if needed (push first), and then, retrieves the state from the cluster and update the local node if needed (pull after)
  • clusterOnly means that the synchronizer only retrieve the state from the cluster and the local node if needed, nothing is pushed to the cluster. With this policy, the cluster acts as a “master”.
  • nodeOnly means that the synchronizer only pushed the local node state to the cluster and send cluster events if required. With this policy, the node acts as a “master”.

Karaf 4 powered and complete refactoring

Cellar 4.0.0 is a complete refactoring compared to previous versions, as it’s designed for Karaf 4.0.0:

  • blueprint is not used anymore, Cellar modules use their own activator extending Karaf BaseActivator, and leveraging the Karaf annotations (@Services, @RequireService, @ProvideService, etc) and the karaf-services-maven-plugin.
  • the Cellar commands use the new Karaf 4 API, and annotations

It allows Cellar to install faster than before, and ready to support the new Karaf Features Resolver, including requirements/capabilities definitions.

What’s next

But, Cellar 4.0.0 is the first release on the new 4.x serie. I’m already planning a 4.1.0 bringing new features and enhancements (and of course bug fixes).

DOSGi refactoring and load balancing policies

I would like to refactore the DOSGi layer:

  • right now, Cellar DOSGi uses two ServiceListeners. It would like to replace the ServiceListeners with pure ServiceTrackers, and use the same design used for the Cellar HTTP Balancer (tracking services, sending cluster events to the other nodes, where the handler creates proxies). It will gives more flexibility and easier lifecycle/tracking of DOSGi.
  • Cellar DOSGi doesn’t support cluster group. A remote OSGi service is available on all cluster nodes, whatever in which cluster group the node is. The refactoring will leverage the cluster group, as we will have the OSGi services available per cluster group, so the proxies on cluster group members.
  • Cellar DOSGi will also support balancing policy. Assuming that several nodes provide the same service, the client nodes will be able to use random, round-robin, weight based balancing selection of the remote node. After this refactoring, it could make sense to include the local service as part of the balancing selection (I have to think about that ;)).

New HTTP balancer policies

Right now, the Cellar HTTP balancer (in the CellarBalancerProxyServlet) only supports random balancing. For instance, if two nodes provides the same service, the balancer randomly choses one of the two.

I will introduce new balancing policies, configurable using the etc/org.apache.karaf.cellar.groups.cfg file:

  • random: as we have right now, it will still be there
  • round-robin: in the cluster, I will keep the index of the last node used in the proxy. The next call will use the next node in the list.
  • weight-based: the user will be able to give a weight on each node (based on the node ID). It’s a ratio of the number of requests that each node should deal with. The proxies will proxy the requests according to these ratios.

New Cellar HTTP sessions replication

Right now, the Cellar HTTP replications directly leverage the Hazelcast WebFilter and sessions replication.

The only drawback is that we don’t leverage the Cellar cluster groups.

In 4.1.0, Cellar will provide its own WebFilter (extending the Hazelcast one) in order to support cluster groups: it means that the sessions replication can be narrowed to only nodes member of the same cluster group.

It will give more flexibility to the users and advanced sessions replications.


Of course, Cellar 4.0.0 also brings lot of bug fixes. I think it’s a good start in the new Cellar 4 series, leveraging Karaf 4.

I hope you will enjoy it and let me know if you have any new ideas !

About jbonofre

ASF Member, PMC for Apache Karaf, PMC for Apache ServiceMix, PMC for Apache ACE, PMC for Apache Syncope, Committer for Apache ActiveMQ, Committer for Apache Archiva, Committer for Apache Camel, Contributor for Apache Falcon Twitter: jbonofre IRC: jbonofre on #servicemix,#karaf,#camel,#cxf on Freenode

Comments are closed.