So I’ve just finished a stint of coding (a rarity nowadays) where I’ve been adding some capabilities to our platform based on OSGi (we’re running on Apache Servicemix, running Apache Felix, under the covers). The drivers behind the coding were to address requirements we had to support a more “tiered” configuration structure to the one provided by OSGi Configuration Admin. I’ll attempt to summarise here, in the hope that others have faced same/similar challenges, and can possibly share any experiences they had — or alternative approaches I might want to consider.

The crux of the requirement is this:

For an OSGi platform, with a given bundle, I might want three different set of configuration available to me:

• Global Configuration – available to ALL bundles
• Bundle Configuration – configuration specific to a named bundle (e.g. “MyApplicationBundle”)
• Version-specific Configuration – configuration that is specific to a particular version of a bundle

Global and Bundle should be self-explanatory, the Version-specific might not be so clear. Version-specific config comes mainly into play when we’re deploying multiple versions of the same bundle side-by-side. Useful for upgrades, backwards-compatibility, etc. If I write a new version of a bundle (say “MyApplicationBundle”, version 2.0), it’s not necessarily true that the configuration properties I used for version 1.0 are going to be applicable to version 2.0.

But ultimately what I want is to be able to have ONE set of Configuration which is:

GLOBAL + BUNDLE + VERSION = [My Bundle Configuration]

Where if the same property is specified at the BUNDLE or VERSION level, it overrides the more general property. This allows an administrator to set a particular property once and only override it if necessary, rather than copying it into each configuration for a bundle that requires it.

Config Admin today allows me to load a particular named Configuration (e.g. “MyApplication”). I guess you could argue that I could achieve a version of what I want by simply having three property files (“GLOBAL”,”MyApplication”, and “MyApplication_2.0″) stored in ConfigAdmin, and write some re-usable code that merges these together. True, but what that would give me (today, using Felix/Servicemix), is a Config Admin directory full of files all at different levels and versions. If I have 100 bundles, that could sure be a lot of files :) A more manageable structure is required.

So I naturally looked to Maven as a source of inspiration. It already has a good syntax for a hierarchical structure of artefacts, namely:


[COMPONENT]
---> [VERSION]
---------> [FILES]


So this allows me to store the following:


[ROOT]
---- GLOBAL_PROPERTIES.CFG
---- [MYAPPLICATION]
----------- PROPERTIES.CFG
------------[2.0]
---------------PROPERTIES.CFG


This allows me to keep my three different sets of configuration stored in a logical way.

But sadly, this is only half the problem. When we move from a single server to a multi-node cluster, we likely don’t want to store our configuration under each node – we want to store and manage it in a single place (possibly even protected by Version-Management). One such solution might be to store all our configuration in the structure above on a shared drive, and then “mount” this drive as the “configuration directory” for each node instance. That gets us some way to a clustered configuration, but we still have issues.

In an ideal world (yeah, right) every node in a cluster is equal, and all configured the same. Back in the real world, that isn’t the case. Certainly for the cases I’ve seen so far, a number of things might be configured differently, even for a cluster of nodes:

• Host and port information for callbacks, server URL publishing, etc
• Tuning parameters for each node in the cluster, based on the different loads they are expecting (if the cluster is partitioned that way)
• Configuration for the applications, if they are partitioned to service different subsets of data (geography, users, volume, etc)

Worse still, we almost certainly want the same useful “merging” approach here as we do for our single-node problem; that is, I might have a component for which I typically want to have the same configuration across all nodes in the cluster (CLUSTER_COMPONENT), but with overrides for just a few properties on a single node (NODE_COMPONENT). Same goes for GLOBAL and VERSION-level configuration too:

CLUSTER_GLOBAL + NODE_GLOBAL + CLUSTER_COMPONENT + NODE_COMPONENT + CLUSTER_VERSION + NODE_VERSION = [My Bundle Configuration in a Cluster]

Now of course, this is the WORST CASE SCENARIO for combining configuration, but it’s still very likely. So now, our structure looks like:


[ALL-NODES]
--->[COMPONENT]
-------> [VERSION]
----------> [FILES]
[NODE-01]
--->[COMPONENT]
[NODE-02]
--->[COMPONENT]
[...]


Where “all-nodes” is the configuration that applies across the cluster, and “Node-01″, “Node-02″, etc. is the configuration overrides for a particular node in the cluster. Provided this was stored on a shareable filesystem, this could be the centralised configuration for a cluster. Maybe something like GIT could be then used to easily apply Version-Management to it and deploy configuration on-site?

Finally, a large portion of my design was to support multiple sources of the configuration data. Filesystems, LDAP, ConfigAdmin. Certainly for sensitive or easier-to-manage data, LDAP (or a database) is a natural choice, yet I don’t see a way (that is documented, anyway) in Felix to configure an LDAP source (for example). Certainly nothing that meets my somewhat broad requirements to load multiple sources at once.

So my recent coding-stint addresses all of this into a custom component that we use today. I had a design choice as to whether to position this code underneath Configuration Admin (e.g. our components use CA, CA invokes our code to gather configuration, and returns), or on top (we use a custom bean to request configuration, and that bean uses CA if-and-only-if the requested Configuration is managed by CA). Right now, I’ve settled on the latter, mainly for portability point-of-view. The component today can be used both inside AND outside of the container – it has no hard dependencies on OSGi or CA, and so we can use it for Configuring non-OSGi applications too.

But I’m wondering, has this been discussed/tackled before? Certainly my initial browsing and investigations suggested to me that it hadn’t. I’ve taken a look at ConfMan from OPS4J PAX which seems to at least be looking at a similar problem, but not as broad a scope with the solution.

Probably the most promising option is Fuse Fabric, a product based on Apache Zookeeper, which is starting to work its way into the Servicemix and Karaf products. But I think that, right now, it’s focus is in a slightly different place to mine – albeit an area that I’m likely to be spending more and more time looking at in the future. Fabric is more about the co-ordination of the cluster, which certainly includes configuration, but covers much, much more (upgrading, provisioning, etc). Compared to my approach above (for configuration), it’s much more of a “push” model that an “pull”. In my approach above, each node “pulls” configuration from the central configuration source, whereas Fabric would “push” it. Also, today, I don’t see anything to suggest the level of complexity in the configuration to support some of the use cases I describe above. The interaction seems to be directly with OSGi Configuration Admin, and so it’s deploying single configuration files into CA on each node. That said, I’d love to reach a point where maybe I can have Fabric interacting with my own Configuration component to do the “tiered config” approach, and let Fabric manage the rest – that would be the absolute “sweet spot”.

Lastly, I’ve been toying with the idea of turning this recent work into a separately-named component and opening it up for general use, with the hope that a platform provider like Servcemix might adopt it. Would anyone use it? Would be interested to know what people think.

Dan