Big Bubbles (no troubles)

What sucks, who sucks and you suck

Running WebSphere in a Solaris Zone

“My new rule of thumb is that absolutely no Internet facing service be run in anything but a non-global zone. Anything less is being reckless.”
- Jarod Jenson, Aeysis

OK, right, that’s the new paradigm, is it? Let’s see if we can make IBM WebSphere Application Server run inside a non-global zone then.

There ought to be a number of advantages to this architecture: * The base WebSphere installation should only be installed once in the global zone and then shared between the other zones. * Following from that, if the local zones are compromised, perhaps via a rogue web application, then the WebSphere executables and libraries can’t be modified, since they will be read-only. This is a major security win. * Patches and WebSphere fixes can be applied once to the global zone and will immediately update all the local zones (but see note below). * Theoretically, new zones/WebSphere instances can be quickly provisioned, duplicated or migrated (although some of this functionality isn’t easily realised with the current Zones functionality). Some of this may help the flow of a new application release through development, acceptance testing, staging and production environments. * Developers can have full access to particular WebSphere zones without impacting other instances. * Because zones are independent, it will be easier (and more robust) to run multiple instances of a particular web application. For example, the application I look after uses a standard log4j configuration, logging to a local file. Having two application instances in the same environment would cause both to write to the same log file, which results in corruption and missed entries. However, in separate zones each could have the exact same configuration but the output files would actually be different (zone-specific). * A multi-node WebSphere topology (separate web/application/database servers) can be simulated on a single host. At present, it’s expensive to create a test site that mirrors the distributed topology of a production site, yet without this an application cannot be properly qualified.

By default, all dynamic WebSphere data, such as configurations and logs, is held within the WebSphere installation tree. This wouldn’t work if that tree was only available read-only (i.e. as an inherited package in a zone). Fortunately, WebSphere 5.x contains a feature called configuration instances, which are multiple executable WebSphere instances that share the base installation but hold their dynamic data in separate, user-defined directory trees. The wsinstance command is used to create and delete configuration instances (available from WAS 5.0.2 onwards). (Note: If you’re using WAS 6.x, configuration instances have been replaced by profiles, which offer more functionality and are probably a better fit with Zones.)

Hence, the strategy will be: 1. Install WebSphere 5.1 within the global zone but do not activate it. 2. Create a local zone per required WebSphere instance, each one inheriting the base WebSphere installation. 3. Within each zone, run wsinstance to create a new configuration instance. (NB. For ease of management and other reasons, it is probably better to stick with one instance per zone.)

There’s a gotcha here: normally, wsinstance “registers” new instances within the base installation so that fixes are propagated to the configuration instances when they are applied to the base instance. This won’t work if the base installation is mounted read-only. Commendably, wsinstance has a -readOnly option to skip this step, but beware IBM’s caveat:

“Creating configuration instances from an original installation in a read-only partition has service implications. There is no mechanism in the original installation to identify and update a configuration instance that you create from a read-only partition.
…
When you install a fix pack or interim fix to the original installation, you must use the wsinstance tool to delete the configuration instance and recreate it. A configuration instance must be at the same fix level as the product from which you create it.”

Recreating all the dependent instances after installing a new fix would be a major pain. I haven’t tackled this one yet, but my initial thoughts on possible workarounds are: * Use the backupConfig and restoreConfig commands either side of recreating the instance; this seems like the most promising solution. * Write a custom JACL/Jython script to configure a new instance (including JDBC resources and app servers). This isn’t a bad idea anyway, but it means facing the living nightmare of MBeans. * Manually register the instances in the base install (e.g. by running a non-readOnly test in the global zone and discovering which files are updated). I think this will be a non-starter, as the configuration instances will still have installation paths that are zone-dependent and won’t be valid in the global zone. (…Unless we actually create all the instances in the global zone, but mount (lofs,rw) and run each in its own zone. Hmmm… But we’d lose path consistency because each instance must have the same path in both global and non-global zone; symlinks?) * Manually copy updated files from the base install to the configuration instances. At first glance, this would mainly seem to involve reinstalling the adminconsole application. However, there could be other changes, such as within the properties files, that would have to be merged in by hand. Yuk - nasty and error-prone.

Enough theorising, here’s the procedure so far: 1. Install a recent Solaris Express release. 2. Create a new local zone in the global zone, ensuring that it inherits the base WebSphere installation path and any dependencies (e.g. our application uses the DB2 V7 client to talk to a backend database): add inherit-pkg-dir set dir=/opt/WebSphere end add inherit-pkg-dir set dir=/opt/IBMdb2 end Install and boot the zone. 3. Login to the new zone. Create a configuration instance: # cd /opt/WebSphere/AppServer/bin/wsinstance # ./wsinstance.sh -name `zonename` -path /opt/WebSphere-i/AppServer
-host `uname -n` -readOnly -create I’m not sure of the arguments here yet. The path is a consistent variant of the base installation path, for ease of management; that way, a WebSphere startup script can check whether that path exists and, if so, source the parameter setup script for the instance first. The instance takes the name of the local zone. I’ve used the hostname of the local zone for the -host parameter here; maybe it should be the hostname of the global zone? (IBM expected this command to be run within a single consistent environment, not some bastard hybrid.) Certainly, the new instance takes its cell name from the base install in the global zone. 4. Source config_inst_path/bin/setupCmdLine.sh to initialise the environment for the new instance. (This must be done before running any WebSphere commands for the instance during a session.) 5. Edit ${USER_INSTALL_ROOT}/config/cells/*/nodes/`uname -n`_`zonename`/serverindex.xml and change all occurrences of the global zone hostname to the local zone hostname. This is required for the WebSphere administration commands to communicate with running servers. 6. Test the instance by starting and stopping server1 (the default server/administration app). When I started the server, I got the following error: WSVR0613E: WebSphere process class preloading could not write to the preload file. However, the process ran correctly. If this happens, delete any files in ${WAS_INSTALL_ROOT}/logs/preload/ (the base install), as they appear to confuse the class preloader for the startServer command. 7. If using Network Deployment, add the new instance as a managed node of the deployment manager (in another zone?). I haven’t tested this yet. 8. Using the administration console in the local zone, create JDBC resources and servers and install your applications as normal.

During my setup, I also created a new DB2 client instance in the zone. db2icrt spat out an error:

cannot open /dev/ksyms
DBI1069E Unexpected error. Function = chk\_kernel\_conf, Return code = 20.

The script is running sysdef(1m) to verify kernel parameters, which doesn’t work in a non-global zone (at least until Sun fix it). Fortunately, it ignores the return value of chk_kernel_conf() anyway, so the rest of the instance creation works! Presumably, this will be fixed when IBM qualifies DB2 for Solaris 10 (or am I expecting a lot…?).

2004-07-16

I’ve created a new application server and installed our main application. It appears to run successfully, although I don’t have sufficient resources in the local environment to exhaustively test it.

Caveat: This procedure is almost certainly unsupported by IBM at present.

2006-06-15

If you’re using WAS V6.0.2 or later, which is officially supported on Solaris 10, you can do something similar using WAS profiles in zones. There is a Sun whitepaper by Dileep Kumar and Albert Leigh that describes the procedure (link below), although it doesn’t discuss subsequent product maintenance (since the central installation registry doesn’t have a record of the zoned profiles).

There still seems to be some confusion between IBM and Sun regarding support for WAS 5.1.x on Solaris 10. However, it definitely isn’t supported in local zones. Regardless of that, I’ve had some success using zoned installs to temporarily relocate test WAS servers while performing prolonged maintenance on the original hardware. (I use the backupConfig/restoreConfig utilities to transfer the WAS configuration to the zone, then modify WAS_NODE and WAS_CELL in the setupCmdLine.sh script to match the original system.) That alone may justify running an unsupported configuration for a brief time in non-critical situations.

This entry is still open for updates.

Other bubbles