When you use Cellar with web applications, or CXF/HTTP endpoints, a “classic” need is to load balance the HTTP requests on the Karaf nodes.

You have different ways to do that:
- using Camel Load Balancer EIP: it’s an interesting EIP, working with any kind of endpoints. However, it requires to have a Karaf running the Load Balancer routes, so it’s not always possible depending of the user security policy (for instance, putting it in DMZ or so)
- using hardware appliances like F5, Juniper, Cisco: it’s a very good solution, “classic” solution in network teams. However, it requires expensive hardwares, not easy to buy and setup for test or “small” solution.
- using Apache httpd with mod_proxy_balancer: it’s the solution that I’m going to detail. It’s a very stable solution, powerful and easy to setup. And it costs nothing

For instance, you have three Karaf nodes, exposing the following services and the hostname:
- http://192.168.134.3:8040/services
- http://192.168.134.4:8040/services
- http://192.168.134.5:8040/services

We want to load balance those three nodes.

On a dedicated server (it could be installed on one hosting Karaf), we just install Apache httpd:


# on Debian/Ubuntu system
aptitude install apache2



# on RHEL/CentOS/Fedora system
yum install httpd
# enable network connect on httpd
/usr/sbin/setsebool -P httpd_can_network_connect 1


Apache httpd comes with mod_proxy, mod_proxy_http, and mod_proxy_balancer modules. Just check if those modules are loaded in the main httpd.conf.

You can now create a new configuration for your load balancer (directly in the main httpd.conf or by creating a conf file in etc/httpd/conf.d):


<Proxy balancer://mycluster>
  BalancerMember http://192.168.134.3:8040
  BalancerMember http://192.168.134.4:8040
  BalancerMember http://192.168.134.5:8040
</Proxy>
ProxyPass /services balancer://mycluster


The load balancer will proxy the /services requests to the different Karaf nodes.

By default, the mod_proxy_balancer module uses a byrequests algorithm: all nodes will receive the same number of requests.
You can switch to bytraffic (using the lbmethod=bytraffic in the proxy configuration): in that case, all nodes will receive the same amount of traffic (by KB).

The mod_proxy_balancer module is able to support session “affinity” if your application needs it.
When a request is proxied to some back-end, then all following requests from the same user should be proxied to the same back-end.
For instance, you can use the cookie in header to define the session affinity:


Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://mycluster>
  BalancerMember http://192.168.134.3:8040 route=1
  BalancerMember http://192.168.134.4:8040 route=2
ProxySet stickysession=ROUTEID
</Proxy>
ProxyPass /myapp balancer://mycluster


The mod_proxy_balancer module also provide a web manager allowing you to see if your Karaf nodes are up or not, the number of requests received by each node, and the current lbmethod in use.

To enable this balancer manager, you just have to add a dedicated handler:


<Location /balancer-manager>
  SetHandler balancer-manager
  Order allow,deny
  Allow from all
</Location>


Point your browser to http://host/balancer-manager and you will see the manager page.

You can find more information about mod_proxy_balancer here: http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html.

Apache httpd with mod_proxy_balancer is an easy and good HTTP load balancer solution in front of Karaf and Cellar.

By default, Karaf create a Jetty connector on the 8181 http port (and 8443 for https). You can change this port number by providing etc/org.ops4j.pax.web.cfg file.

But, you can also create new connector in the embedded Jetty.

You may see several advantages for multiple connectors:

• you can isolate a set of applications, CXF services, Camel routes on a dedicated port number
• you can setup a different configuration for each connector. For instance, you can create two SSL connectors, each with a different keystore, truststore, …

You can find etc/jetty.xml configuration file where you can create custom Jetty configuration.

NB: if you want to have both etc/org.ops4j.pax.web.cfg and etc/jetty.xmll, don’t forget to reference jetty.xml in org.ops4j.pax.web.cfg using the org.ops4j.pax.web.config.file property pointing to the jetty.xml, for instance:


# in etc/org.ops4j.pax.web.cfg
org.ops4j.pax.web.config.file=${karaf.home}/etc/jetty.xml


To configure a new connector, you can add a addConnector call in this configuration. For instance, we can create a new connector on 9191 http port number (and 9443 https port number):


  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
        <Set name="host">0.0.0.0</Set>
        <Set name="port">9191</Set>
        <Set name="maxIdleTime">300000</Set>
        <Set name="Acceptors">1</Set>
        <Set name="statsOn">false</Set>
        <Set name="confidentialPort">9443</Set>
        <Set name="name">myConnector</Set>
      </New>
    </Arg>
  </Call>


Now, Karaf will listen on 8181 and 9191 (for http), 8443 and 9443 (for https).

You can also define a connector dedicated to https with dedicated configuration for this connection, especially keystore, truststore, and client authentication:


  <Call name="addConnector">
    <Arg>
      <New class="org.eclipse.jetty.server.ssl.SslSelectChannelConnector">
        <Set name="port">9443</Set>
        <Set name="maxIdleTime">30000</Set>
        <Set name="keystore">./etc/keystore</Set>
        <Set name="password">password</Set>
        <Set name="keyPassword">password</Set>
      </New>
    </Arg>
  </Call>


By default, the web application will be bind on all connectors. If you want that your web application use a specific connector, you have to define it in the MANIFEST using the following properties:


Web-Connectors: myConnector
Web-VirtualHosts: localhost


If you use CXF services or Camel routes, if you use a connetor hostname and port number in the endpoint, it will use the corresponding connector.

For instance, the following CXF endpoint of a Camel route will use myConnector:


...
  <cxf:cxfEndpoint id="cxfEndpoint" address="http://localhost:9191/services/myservice" wsdlUrl="..."/>
...


Karaf allows you a fine grained Jetty configuration. Karaf becomes a real complete WebContainer, with custom configuration on several connectors. It’s especially interesting for SSL connector where each connector can have a dedicated keystore and truststore, and client authentication configuration.

Karaf provides a default logging configuration in etc/org.ops4j.pax.logging.cfg file.

By default, all INFO log messages (rootLogger) are send into a file appender (in data/log/karaf.log). The file appender “maintains” one file of 1MB, and store up to 10 backup files.

Adding a new appender configuration, example with Syslog appender

We can add new appender configuration in the Karaf logging module.

For instance, we can add a syslog appender in etc/org.ops4j.pax.logging.cfg:


log4j.rootLogger = INFO, out, syslog, osgi:*
...
# Syslog appender
log4j.appender.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.syslog.layout=org.apache.log4j.PatternLayout
log4j.appender.syslog.layout.ConversionPattern=[%p] %c:%L - %m%n
log4j.appender.syslog.syslogHost=localhost
log4J.appender.syslog.facility=KARAF
log4j.appender.syslog.facilityPrinting=false
...


We create the syslog appender configuration, and we use this appender for the rootLogger.

Pax Logging provides all default Log4j appenders.

Creating a custom appender

It’s also possible to create your own appender.

For instance, you want to create MyJDBCAppender, extending the standard Log4J JDBCAppender. MyJDBCAppender has a better management of the quote in the SQL query for a DB2 backend for instance:


package org.apache.karaf.blog.logging.appender;

}


We put the MyJDBCAppender java file in a src/main/java/org/apache/karaf/blog/logging folder.

We package this appender as an OSGi bundle. This bundle is a fragment to the Pax Logging service bundle:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

</project>


We can use our appender in etc/org.ops4j.pax.logging.cfg file, for instance:


log4j.rootLogger = INFO, out, myappender, osgi:*
...
log4j.appender.myappender=org.apache.karaf.blog.logging.appender.MyJDBCAppender
log4j.appender.myappender.url=jdbc:db2:....
log4j.appender.myappender.driver=com.ibm.db2.jcc.DB2Driver
log4j.appender.myappender.user=username
log4j.appender.myappender.password=password
log4j.appender.myappender.sql=insert into logs values({{sql_apos}%x{sql_apos}, {sql_apos}%d{sql_apos}, {sql_apos}%C{sql_apos}, {sql_apos}%p{sql_apos}, {sql_apos}%m{sql_apos})
log4j.appender.myappender.layout=org.apache.log4j.PatternLayout


In order to be loading very early in the Karaf bootstrap, our appender bundle should be present in the system folder and defined in etc/startup.properties.

The system folder has a “Maven repo like” structure. So you have to copy with:


system/groupId/artifactId/version/artifactId-version.jar


In our example, it means:


mkdir -p $KARAF_HOME/system/org/apache/karaf/blog/logging/appender
cp target/org.apache.karaf.blog.logging.appender.jdbc-1.0-SNAPSHOT.jar $KARAF_HOME/system/org/apache/karaf/blog/logging/appender/org.apache.karaf.blog.logging.appender.jdbc/1.0-SNAPSHOT/org.apache.karaf.blog.logging.appender.jdbc-1.0-SNAPSHOT.jar


and in etc/startup.properties, we define the appender bundle just after the pax-logging-service bundle:


...
org/ops4j/pax/logging/pax-logging-api/1.6.9/pax-logging-api-1.6.9.jar=8
org/ops4j/pax/logging/pax-logging-service/1.6.9/pax-logging-service-1.6.9.jar=8
org/apache/karaf/blog/logging/appender/org.apache.karaf.blog.logging.appender.jdbc/1.0-SNAPSHOT/org.apache.karaf.blog.logging.appender.jdbc-1.0-SNAPSHOT.jar=8
...


You can now start Karaf, it will use our new custom appender.

The purpose is to exchange certificates and allow only “trusted” clients to use the Karaf HTTP service.

Enable HTTP client auth

First of all, we have to enable the HTTP client auth support in Karaf.

When you install the HTTP feature, Karaf leverages Pax-Web to provide HTTP OSGi service:


karaf@root> features:install http


Now, we have to add a custom etc/org.ops4j.pax.web.cfg file:


org.osgi.service.http.port=8181

org.osgi.service.http.port.secure=8443
org.osgi.service.http.secure.enabled=true
org.ops4j.pax.web.ssl.keystore=./etc/keystores/keystore.jks
org.ops4j.pax.web.ssl.password=password
org.ops4j.pax.web.ssl.keypassword=password
#org.ops4j.pax.web.ssl.clientauthwanted=false
org.ops4j.pax.web.ssl.clientauthneeded=true


NB: clientauthwanted and clientauthneeded properties are valid for Karaf 2.2.x which use Pax Web 1.0.x.

Thanks to the clientauthneeded property, we “force” the client to be trusted.

Create the trusted client certificate

We are going to use keytool (provided with the JDK) to manipulate the keys and certificates.

The first step is to create two key pairs:

• one for the server side (use for SSL)
• one as a example of client side (use for “trust”, should be performed for each client, on the client side)


mkdir -p etc/keystores
cd etc/keystores
keytool -genkey -keyalg RSA -validity 365 -alias serverkey -keypass password -storepass password -keystore keystore.jks
keytool -genkey -keyalg RSA -validity 365 -alias clientkey -keypass password -storepass password -keystore client.jks


NB: these key are self-signed. In a production system, you should use a Certificate Authority (CA).

Now, we can export the client certificate to be imported in the server keystore:


keytool -export -rfc -keystore clientKeystore.jks -storepass password -alias clientkey -file client.cer
keytool -import -trustcacerts -keystore keystore.jdk -storepass password -alias clientkey -file client.cer


We can now check that the client certificate is trusted in our keystore:


keytool -list -v -keystore keystore.jks
...
Alias name: clientkey
Creation date: Dec 12, 2012
Entry type: trustedCertEntry
...


and we can now remove the client.cer certificate.

Start Karaf and test with WebConsole

Now we can start Karaf:


bin/karaf


and install the WebConsole feature:


karaf@root> features:install webconsole


If we try to access to the WebConsole (using a simple browser) using https://localhost:8443/system/console, we have:


An error occurred during a connection to localhost:8443.

(Error code: ssl_error_bad_cert_alert)


which is normal as the browser doesn’t have any trusted certificate.

Now, we can add the client certificate in the browser.

Firefox supports the import of PKCS12 keystore. So, we are going to “transform” the JKS keystore into a PKCS12 keystore:


keytool -importkeystore -srckeystore clientKeystore.jks -srcstoretype JKS -destkeystore client.pfx -deststoretype PKCS12
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
Entry for alias clientkey successfully imported.
Import command completed: 1 entries successfully imported, 0 entries failed or cancelled


Now, we can import the client certificate in Firefox. To do so, open the Preferences window (in Edit menu), and click on the Advanced tab.
You can go in Encryption tab and click on “View Certificates” button.

In “Your Certificates” tab, you can click on the Import button and choose the client.pfx keystore file.

If you try to access to https://localhost:8443/system/console again, you will have access as a trusted client and use it.

Conclusion

It’s the same with any kind of HTTP client that try to use the HTTPs layer of Karaf.

Now, we can disable the HTTP support in Karaf (to force the usage of HTTPs), and we can allow only “trusted” clients to use the HTTPs layer of Karaf.

It’s a simple mechanism if you want to limit access to HTTP resources only for trusted clients.

Now, Cellar 2.2.5 is released ! But, what’s new in this version ?

Groups are now persistent

In Cellar 2.2.4, the empty groups disappear after a restart.

You created a new cluster group without any member (empty group) with:


karaf@root> cluster:group-create foobar
karaf@root> cluster:group-list|grep -i foobar
foobar []


If you restart Cellar (or Karaf), the empty groups were lost:


karaf@root> cluster:group-list|grep -i foobar


To avoid this, in Cellar 2.2.5, the cluster groups are now persistent on each node. We introduced a new groups property in etc/org.apache.karaf.cellar.groups.cfg to store the list of groups. Cellar now reads this property as startup to populate the cluster groups not present on the cluster.

On the other hand, the groups property in etc/org.apache.karaf.cellar.node.cfg defines the group membership of the local node.

If you restart Karaf (or Cellar), this group disappeared from

Cluster producers, consumers, and handlers persistency

Like for groups, with Cellar 2.2.4, the status of cluster event producers, consumers and handlers was not persistent. It means that if you stop the cluster event producer, for instance, after a restart, the producer was start again. So we loosed the status before the restart.

In Cellar 2.2.5, to avoid that, the status of cluster event producers, consumers and handlers is now persistent in etc/org.apache.karaf.cellar.node.cfg (it’s the current status on the local node). Cellar now reads the properties from this file at startup to set the previous status (before the restart).

Bundles blacklist and whitelist

In Cellar 2.2.4, the bundles blacklist and whitelist were not correct by default in etc/org.apache.karaf.cellar.groups.cfg: all bundles were blocked (inbound and outbound). If you tried to install a bundle on the cluster, you saw a “Bundle xxxx is BLOCKED …” in the log.

We changed the default setup to allow all bundle cluster events.

Config sync enhancement

In Cellar 2.2.4, to avoid infinite loop, we introduced a karaf.cellar.sync property appended to all synchronized configuration PID. This property contained the timestamp of the last Cellar configuration synchronization. This mechanism has two issues:

• it pollutes the configuration PID (it can be confusing for the users to see a “not usable” property)
• if a configuration change occurs between the timestamp and the Cellar configuration timeout, it’s not synchronized on the cluster

We changed the configuration synchronization mechanism in Cellar 2.2.5. The karaf.cellar.sync property has been removed. Now we compare the dictionary of configuration PID in the cluster (distributed map) and the local.

Bundle state, name, and symbolic name

The bundle distributed map stored only the bundle name. It was a little bit restrictive.

In Cellar 2.2.5, both bundle name and symbolic name are stored in the cluster distributed map.

It allows the users to select a bundle (on the cluster) using both name and symbolic name.

Improvements on the cluster:* commands and MBeans

In order to mimic the Karaf core commands and MBeans, the Cellar commands and MBeans have been improved.

The cluster:feature-install (and the corresponding MBeans) now supports norefresh and noclean options, as supported by features:install Karaf command.

The cluster:bundle-list supports the -l option (to display the bundle location) and -s option (to display the bundle symbolic name), as the bundles:list/osgi:list Karaf command.

The cluster:config-list command now allows to display directly a configuration PID dictionary.

A new command has been introduced in Cellar 2.2.5: cluster:sync. This command forces a synchronization on the local node. It’s particulary interesting when the node loosed the communication with the other nodes (for instance, due to a network issue), the cluster:sync forces the resynchronization of the node and the cluster (in both direction).

Restart issues

Cellar uses a LocalBundleListener to listen for changes on the local bundles, and broadcast these changes as a bundle cluster event.

In Cellar 2.2.4, this listener was a simple BundleListener. The problem was that this listener get the “bundle stop” local event when stopping the framework and broadcast it to the cluster (including to the local node). It means that the “latest” state on the bundle was “stopped”. At restart, the OSGi framework reset the bundle in “stopped” status (instead of “started”).

In Cellar 2.2.5, the listener has been changed to a SynchronousBundleListener. Thanks to this listener, we are able to get the stopping event from the OSGi framework. When the framework stops, Cellar disable the bundle listener in order to avoid to change the bundle states.

Like this, we restart the bundle in the correct state.

We hope that you will like this new Cellar 2.2.5 release. We mostly focused on the bug fixes to provide a more stable cluster solution for Karaf.

Now, we are preparing Cellar 2.2.6 with new bug fixes, new feature, etc. In the mean time, Cellar 2.3.0 is in preparation, supporting Karaf 2.3.x and new Hazelcast version.

The Karaf 2.2.x branch is now only in maintenance mode: it means that no new features will be implemented in this branch, only major bug fixes.

The new “stable” branch is now Karaf 2.3.x which is a perfect transition branch between Karaf 2.2.x (heavely used) and the future Karaf 3.x (which should arrive very soon).

What’s new in this 2.3.0 release:

* OSGi r4.3: Karaf 2.2.x branch was powered by OSGi frameworks implementing OSGi r4.2 norm. Karaf 2.3.0 is now powered by the new OSGi r4.3 framework (Apache Felix 4.0.3 and Equinox 3.8.x), for both OSGi core and compendium. It provides new features like weaving, etc.
* Aries Blueprint 1.0.x: Karaf 2.3.0 uses the new Aries Blueprint version at different level (core, JMX, etc).
* Update to ASM 4.0: in order to work with Aries proxies, we updated to new ASM bundle. We also provided configuration that allows you to enable/disable weaving.
* OSGi Regions and SCR support: Karaf 2.3.0 provides both Regions and SCR support.
* JMX improvement: the previous MBeanRegistrer from Karaf 2.2.x has been removed to be replaced by Aries JMX. It allows an easier way to integrate MBeans by registering OSGi services. The MBeans have been improved to provide new operations and options corresponding with what you can do using the shell commands.
* Complete itest framework: Karaf 2.3.0 provides a new tool: Karaf exam. This tool provides a framework to very easily implements integration tests. It’s able to download and bootstrap a Karaf version on which you can run your commands, deploy your features and bundles, etc. It allows you to run a complete integration tests chain from Maven.
* Dependencies upgrade: a lot of dependencies have been updated. Karaf 2.3.0 uses Pax Logging 1.7.0 including bug fixes and SLF4J 1.7.1 support, new Pax Web and Jetty version for the web container, new JLine, SSHD and Mina versions which especially fix weird behavior on Windows for certain keys, etc.
* KAR improvements: if Karaf 3.x will provide a lot of enhancements around the KAR files, Karaf 2.3.0 already provides fixes in the KAR lifecycle.
* JAAS commands improvements: the jaas:* commands have been enhanced to allow you a fine-grained management of the realms and login modules.

You can find the Karaf 2.3.0 content details on the Release Notes.

The Karaf team is proud to provide this release to you. We hope you will enjoy it !

The 2.2.9 version is a major bug fixes release.

We fixed 52 issues in this release, and we are inline with our release cycle (around every 2 months).

Especially, it includes:

• Work with the latest JRE/JDK 1.6.0 and 1.7.0 updates: some changes in the Java Virtual Machine resulted to issues in previous Karaf releases. It’s now fixed in Karaf 2.2.9, allowing Karaf to work with Java 1.6.0_33 or 34 for instance, but fixed in Karaf itself and an upgrade to Aries Proxy 0.3.1.
• New OSGi frameworks: this release upgrade to the latest minor versions of the OSGi frameworks, especially Apache Felix Framework 3.2.2.
• Dependencies minor version updates: in order to fix several minor issues, some dependencies have been upgraded (javax.mail 1.4.5, ServiceMix bundles, etc).
• Stopping the shell console doesn’t stop the OSGi framework: previously, when you stopped the Karaf shell console bundle, it also stopped the OSGi framework. No, the bundles are no more coupled, allowing you to stop the shell console but let your application bundles running in the OSGi framework.
• MBeans and shell commands are consistent: some options present in the shell commands (for instance, -c and -r options in features:install command) were not present for the same action using MBeans. Now, you can find the same options in both commands and MBeans.
• Features installation optimisation: Karaf now avoid to resolve the same bundles many times when installing features in cascade. It improves the features installation time.

You can have more information and download this release here:

http://karaf.apache.org/index/community/download/karaf-2.2.9-release.html

Now, we are working on Karaf 2.3.0 and 3.0.0-RC1. Karaf 2.3.0 should arrive very soon.

The laptop worked fine, it was pretty fast, building a lot of projects in the same time, etc.

I complained with:

• the temperature was really hot (sensors said between 80°C and 95°C all of the time). Sometime, the temperature was critical and the system shutted down.
• due to the previous point, the fan was very noisy
• the battery autonomy was average

This laptop use dual graphic cards: Intel SGI graphic card (to reduce energy consumption), and AMD Radeon HD 6600M (for 3D enhanced graphics and HD videos).

When I got this laptop, I tried to install the fglrx open source drivers a couple of times. Without success: it seems that the Radeon card was not fully supported.

As the laptop ran really fine (unity was fast, vlc was able to read HD videos without problem, etc), I stayed with the Intel Xorg driver.

Yesterday evening, watching a movie, the laptop was so hot that I have to use a pillow to avoid to burn my legs

So today, I decided to find a solution.

As the fglrx open source drivers didn’t work, I try to proprietary AMD Catalyst drivers.

And now, ALL IS DIFFERENT

The temperature stays around 70/80°C, so the fan are pretty quite, and the system is even faster than before.

More over, the admcccle (Catalyst Console) allows you to tweak the graphic card configuration. You can also choose and switch between the Intel SGI card or the RADEON one (it required a reboot).

To summarize, for all Dell Vostro users on Ubuntu (and more generally on Linux), install the ADM Catalyst graphics drivers

Here’s the list of key things included in this release.

Consistent behavior

Cellar is composed by two parts:

• the distributed resources which is a datagrid maintained by each cluster nodes and containing the current cluster status (for instance of the bundles, features, etc)
• the cluster event which is broadcasted from a node to the others

Cluster shell commands, cluster MBeans, synchronizers (called at startup) and listeners (called when a local event is fired, such as feature installed) update the distributed resource and broadcast cluster events.

To broadcast cluster events, we use an event producer. The cluster event is consommed by a consumer which delegates the handling of the cluster event to a handler. We have a handler for feature, bundle, etc.

Now, all Cellar “producers” do:

1. check if the cluster event producer is ON
2. check if the resource is allowed, checking in the blacklist/whitelist configuration
3. update the distributed resources
4. broadcast the cluster event

Only use hazelcast.xml

The org.apache.karaf.cellar.instance.cfg file has disappear. It’s now fully replaced by the hazelcast.xml.

It fixes issue around the network configuration and allows new configuration, especially around the encryption.

OSGi event support

cellar-event feature now provides OSGi event support in Cellar. It uses eventadmin layer. All local event generates a cluster event which is broadcasted to the cluster. It allows to sync remote nodes.

Better shell commands

Now, all cluster:* shell commands mimic the core Karaf commands. It means that we will find quite the same arguments and options and similar output.

The cluster:group-* shell commands have been improved and fixed.

A new shell command has been introduced: cluster:config-propappend to append a String to a config property.

Check everywhere

We added a bunch of check to be sure to have a consistent situation on the cluster and predictable behavior.

It means that the MBeans and shell commands check if a cluster group exists, if a cluster event producer is on, if a resource is allowed on the cluster (for the given cluster group), etc.

You have clean messages informing you about the current status of your commands.

Improvement on the config layer

The Cellar config layer has been improved. It now uses a karaf.cellar.sync property to avoid infinite loop. The config delete operation support has been added, including the cluster:config-delete commands.

Feature repositories

Previously, the feature repositories handling was hidden for the users.

Now, you have a full access to the distributed features repositories set. It means that you can see the distributed repositories for a cluster group, add a new features repository to a cluster group, and remove a features repository from a cluster group.

To do that, you have the cluster:feature-url-* shell commands.

CellarOBRMBean

Cellar provides a MBean for all parts of the cluster resources (bundles, features, config, etc).

However, if an user installed cellar-obr feature, he got the cluster:obr-* shell commands but no corresponding MBean.

The CellarOBRMBean has been introduced and is installed with the cellar-obr feature.

Summary

Karaf Cellar 2.2.4 is really a major release, and I think it should have been named 2.3.0 due to the bunch of the bug fixes and new features: we fixed 77 Jiras in this release and performed lot of manual tests.

The quality has been heavily improved in this release comparing to the previous one.

I encourage all Cellar users to update to Karaf Cellar 2.2.4 and I hope you will be pleased with this new release

One of the first purpose of Apache Karaf Cellar is to synchronize the state of each Karaf instances in the Cellar cluster.

It means that any change performed on one node (install a feature, start a bundle, add a config, etc) generates a “cluster event” which is broadcasted by Cellar to all others nodes. The target node handles the “cluster event” and performed the corresponding action (install a feature, start a bundle, add a config, etc).

By default, the nodes have the same role. It means that you can perform actions on any node.

But, you may prefer to have one node dedicated to the management. It’s what we name “central management”.

Central management

With central management, one node is identified by the manager. It means that cluster actions will be performed only on this node. The manager is the only one which is able to produce cluster event. The managed nodes are only able to receive and handle events, not to produce.

With this approach, you can give access (for instance grant ssh or JMX access) to the manager node to an operator and manage the cluster (and all cluster groups) from this manager node.

We will see now how to implement central management.

We assume that we have three nodes:

• manager
• node1
• node2

First, we install Cellar on the three nodes:


karaf@manager> features:addurl mvn:org.apache.karaf.cellar/apache-karaf-cellar/2.2.4-SNAPSHOT/xml/features
karaf@manager> features:install cellar



karaf@node1> features:addurl mvn:org.apache.karaf.cellar/apache-karaf-cellar/2.2.4-SNAPSHOT/xml/features
karaf@node1> features:install cellar



karaf@node2> features:addurl mvn:org.apache.karaf.cellar/apache-karaf-cellar/2.2.3/xml/features
karaf@node2> features:install cellar


It’s a default installation, nothing special here.

Now, we disable the event producing on node1 and node2.

To do that, from the manager:


karaf@manager> cluster:producer-stop node1:5702
Node Status
node1:5702 false
karaf@manager> cluster:producer-stop node2:5703
Node Status
node2:5703 false


We can check that the producer are really stopped:


karaf@manager> cluster:producer-status node1:5702
Node Status
node1:5702 false
karaf@manager> cluster:producer-status node2:5703
Node Status
node2:5703 false


Whereas the manager is always able to produce event:


karaf@manager> cluster:producer-status
Node Status
manager:5701 true


Now, for instance, we can install eventadmin feature from the manager:


karaf@manager> cluster:features-install default eventadmin


We can see that this feature has been installed on node1:


karaf@node1> features:list|grep -i eventadmin
[installed ] [2.2.5 ] eventadmin karaf-2.2.5


and node2:


karaf@node2> features:list|grep -i eventadmin
[installed ] [2.2.5 ] eventadmin karaf-2.2.5


Now, we uninstall this feature from the manager:


karaf@manager> cluster:features-uninstall default eventadmin


We can see that the feature has been uninstall from node1:


karaf@node1> features:list|grep -i eventadmin
[uninstalled] [2.2.5 ] eventadmin karaf-2.2.5


And now, we try to install the eventadmin feature on the cluster from the node1:


karaf@node1> cluster:features-install default eventadmin


The event is not generated and the others nodes are not updated.

However, two kind of events are always produced (even if the producer is stopped):

• the events with the “force” flag. When you create an event (programmatically), you can use setForce(true) to force the event producing.
• the “admin” events, like the events produced to stop/start a producer, consumer, handler in the cluster.

NB: the event produce handling has been largely improved for Cellar 2.2.4.

Coming in next Cellar release

Currently, the cellar feature install the cellar core bundles, but also the cellar shell commands, and the MBeans.

If it makes sense for the manager, the managed nodes should not “display” the cluster:* commands as they are managed by the manager.

I will make a little refactoring of the Cellar features to split in two parts:

• the cellar-core feature will install hazelcast, and Cellar core bundles
• the cellar-manager feature will install the cluster:* shell commands and the MBeans