Jean-Baptiste Onofré's Blog

What’s new

Quite one month ago, we released Karaf Cellar 2.2.1, the first “official” release of the Karaf clustering sub-project.

This new Karaf Cellar 2.2.2 release includes bug fixes, especially one bug was a blocker as it was not possible to install Cellar on a Karaf instance running on Equinox OSGi framework.

But, it’s not just a bug fix release, we merge two features from the Cellar trunk.

Bundle synchronization

In Karaf Cellar 2.2.1, we were able to synchronize features (including features repositories) and configuration between Karaf Cellar instances. It means that you can install a feature on one node (cluster:features-install group feature), the feature will be install on each Karaf note.

Karaf Cellar 2.2.2 includes the same behavior for pure OSGi bundle. You can install a bundle on one node, the bundle will be installed on each other nodes on the same cluster group.


karaf@root> osgi:install mybundle


mybundle will be installed on all nodes in the same cluster group.

It’s a first step, as we have for features and config, we will add specific command to manipulate bundle, something like:


karaf@root> cluster:install-bundle group mybundle


Cloud support

Cellar relies on Hazelcast in order to discover cluster nodes. This can happen either by using multicast or by unicast (specifying the ip address of each node).
Unfortunately multicast is not allowed in most IaaS providers and specifying the all the ip addresses is not very flexible, since in most cases they are not known in advance.

Cellar solves this problem using a cloud discovery service powered by jclouds.

Cloud discovery service

Most cloud providers among other provide cloud storage. Cellar uses the cloud storage via jclouds, in order to put there the ip addresses of each node so that Hazelcast can found them.
This approach is also called blackboard and in other words is the process where each nodes registers itself in a common storage, so that other nodes know its existence.

Installing Cellar cloud discovery service

To install the cloud discovery service simply the appropriate jclouds provider and then install cellar-cloud feature. For the rest of this manual I will use amazon s3 as an example, but it applies to any provider supported by jclouds.


karaf@root> features:install jclouds-aws-s3
karaf@root> features:install cellar-cloud


Once the feature is installed, it requires you to create a configuration that contains credentials and type of the cloud storage (aka blobstore).
To do that add a configuration file under etc with the name org.apache.karaf.cellar.cloud- .cfg and put there the following information:

provider=aws-s3 (this varries according to the blobstore provider)
identity=”the identity of the blobstore account”
credential=”the credential/password of the blobstore account)”
container=”the name of the bucket”
validity=”the amount of time an entry is considered valid, after that time the entry is removed”

After creating the file the service will check for new nodes. If new nodes are found the Hazelcast instance configuration is updated and the instance is restarted.

Apache Karaf Cellar 2.2.1 has been released today.

Cellar is a Karaf sub-project which aim to provide a clustering solution for Karaf.

Quick start

To enable Cellar into a Karaf, you just have to install the Cellar feature.

First, register the Cellar features descriptor in your running Karaf instance:

karaf@root> features:addurl mvn:org.apache.karaf.cellar/apache-karaf-cellar/2.2.1/xml/features


Now, you can see the Cellar features available:

karaf@root> features:list|grep -i cellar
[uninstalled] [2.2.1 ] cellar Karaf clustering
[uninstalled] [2.2.1 ] cellar-webconsole Karaf Cellar Webconsole Plugin


To start Cellar, install the cellar feature:

karaf@root> features:install cellar


It’s done: your Karaf instance is Cellar cluster ready.

You can see your cluster node ID and, eventually, others cluster nodes:

karaf@root> cluster:list-nodes
No. Host Name Port ID
* 1 node1.local 5701 node1.local:5701
2 node2.local 5702 node2.local:5702


The * indicates your local node (on which you are connected).

You can ping a given node to see how the network behaves:

karaf@root> cluster:ping node2.local:5702
Pinging node :node2.local:5702
PING 1 node2.local:5702 82ms
PING 2 node2.local:5702 11ms
PING 3 node2.local:5702 14ms


Now, if you install a feature, it will be installed on all nodes. For instance, if you install eventadmin feature on node1:

karaf@node1> features:install eventadmin
karaf@node1> features:list|grep -i eventadmin
[installed ] [2.2.1 ] eventadmin karaf-2.2.1


you can see it installed on node2:

karaf@node2> features:list|grep -i eventadmin
[installed ] [2.2.1 ] eventadmin karaf-2.2.1


Cellar groups

In Karaf Cellar, you can define cluster groups. It allows you to select nodes and resources involved in one group.

By default, you have the default group:

karaf@root> cluster:group-list
Node Group
* node1.local:5701 default
node2.local:5702 default


You can create a route, for instance test:

karaf@root> cluster:group-create test
Name test
Members []

For now, the group doesn’t have any member:

kaaf@root> cluster:group-list
Node Group
node1.local:5701 default
* node2.local:5702 default
test


We can add a node as a member of a cluster group:

karaf@root> cluster:group-join test node1.local:5701
Node Group
node1:5701 default
* node2:5702 default
node1:5701 test


The node could be local or remote.

Cluster units

Currently, Apache Karaf Cellar is able to manage features and configuration. It’s based on an event-driven model.

It means that Cellar is listening for some events (such as install/start/remove features, or set configuration) and fire the actions on all nodes.

Features

You can list the features of a given group:

karaf@root> cluster:features-list test


The feature events (install, uninstall) are sync between all members of the same cluster group.

Configuration

Cellar is able to manage configuration event, as it does for features.

You can see the configuration PID associated to a given cluster group:

karaf@root> cluster:config-list test
No PIDs found for group:test


You can create a configuration (with a PID) directly into a given group:

karaf@root> cluster:config-propset test my mykey myvalue


Now, we can see this configuration in the cluster memory:

karaf@root> cluster:config-list test
PIDs for group:test
PID
my


and the value in this configuration PID:

karaf@root> cluster:config-proplist test my
Property list for PID:my for group:test
Key Value
mykey myvalue


Now, you can see that the cluster group test doesn’t have any member for now:

karaf@root> cluster:group-list
Node Group
* node1.local:5701 default
test


If I list the configuration on my node1 local instance, my is not known:

karaf@root> config:list "(pid=my)"


Now, node1 join the test cluster group:

karaf@root> config:group-join test node1.local:5701


And, now, the node1.local inherits of the configuration defined in the cluster:

karaf@root> config:edit my
karaf@root> proplist
service.pid = my
mykey = myvalue


Now, time to work

OK, we have a first release of Karaf Cellar, allowing mainly discovery, group and state replication between Karaf instances. The bundle events support is also quite implemented (it just needs to be finalized).

We are going to add JMX Cellar MBeans and logging message with Mike Van. As Cellar is fully based on Blueprint, adding MBeans should be very easy and quick

But it’s not enough

Now, I’m sure you have a question: is it possible to have a kind of shared OSGi services registry, with proxy, and be able (transparently or not) to use a remote service ? Do we have DOSGi support ?

The short answer is: not in Cellar 2.2.1.

But, we are working on these two topics (I name it “distributed service registry” and “transport”) and it should be included the next Cellar releases.

I hope you enjoy this new Karaf sub-project.