Tomcat Cluster Mod Jk

tomcat-cluster

On this page, we focus on setting up the cluster to act as a load balancer, without session replication. For session replication, refer to tomcat-cluster-session-replication

What is mod_jk?

mod_jk is an Apache HTTPD module that is used to provide load balancing and proxy capabilities. It uses the AJP protocol to facilitate communication between Tomcat servers and the Apache Web Server.

What are the basic steps to get Apache HTTPD, mod_jk, and Tomcat to implement in-memory session replication?

  1. Install Apache
  2. Install Tomcat
  3. Download and install mod_jk
  4. Configure mod_jk inside Apache
  5. Configure Tomcat workers - enabling session replication (distributable element in web.xml)
  6. Configure Tomcat Cluster (server.xml)
  7. Make our web application distributable via the <distributable/> tag in web.xml. Add the <distributable/> element at the end of the web.xml file (just before the </web-app>)
  8. Add more load balancers

How to install Apache?

This is not covered here.

How to install Tomcat?

This is not covered here.

How to download and install mod_jk?

mod_jk is distributed separately from Apache HTTPD as part of the Tomcat project. Binary distributions of the module are available for most major platform. If the version and platform that you are looking for is not yet available, you can build the appropriate version from source. Once you have either downloaded and unzipped or built the module, place it in the 'modules' directory of your Apache HTTPD server.

How to configure mod_jk inside Apache?

Add the following lines to httpd.conf:

# Load module
LoadModule jk_module path/to/apache2/mod_jk.so

# Specify path to worker configuration file
JkWorkersFile /path/to/apache2/conf/workers.properties

# Configure logging and memory
JkShmFile /path/to/desired/log/location/mod_jk.shm
JkLogFile /path/to/desired/log/location/mod_jk.log
JkLogLevel info

# Configure monitoring
JkMount /jkmanager/* jkstatus

<Location /jkmanager>
    Order deny, allow
    Deny from all
    Allow from localhost
</Location>

# Configure applications
JkMount /webapp-directory/* LoadBalancer
  • JkWorkersFile - sets the path to the worker configuration file, which we will create in the next step
  • JkShmFile - sets the path to the shared memory files for the module. Generally, you'll want to keep this with the logs.
  • JkLogFile - sets the path to the module log file.
  • JkLogLevel - sets the level of logging for the module. The valid values for this attribute, in descending order by verbosity, are "debug", "error" or "info".
  • JkMount - this is used to map a certain URL pattern to a specific worker configured in the worker configuration file. Here, we use it twice - once to enable /jkmanager as the access URL for jkstatus, a virtual monitoring worker, and once to map all requests we want to be handled by the cluster to the "lb" worker, a virtual worker that contains the load balancing capability
  • Location - this is a security constraint. The settings we have included allow access to the jkmanager only from the localhost (this is a Good Idea).

How to configure the cluster worker?

Now that we've configured the main settings, we will configure the worker. "Workers" is a blanket term used within mod_jk to refer to both real Tomcat servers that will process requests, and virtual servers included in the module to handle load balancing and monitoring.

Add the following to /path/to/apache2/conf/workers.properties:

# Define worker names
worker.list=jkstatus, loadbalancer, stat

# Create virtual workers
# The status worker allows us to get statistical data
worker.jkstatus.type=status
worker.loadbalancer.type=lb

# Declare Tomcat server workers 1 through n
worker.worker1.type=ajp13
worker.worker1.host=hostname
worker.worker1.port=8009

# ...
worker.worker[n].type=ajp13
worker.worker[n].port=8010
worker.worker[n].host=hostname

# Associate real workers with virtual LoadBalancer worker
worker.loadbalancer.balance_workers=worker1,worker2,…worker[n]

How to configure Tomcat worker (enabling session replication)?

In this example, we'll use simple all-to-all in-memory session replication. This means every worker in our cluster will replicate their session across every other worker. This is not always the most efficient method of session replication in higher load environments, but you can easily build on the concept. Tomcat provides in-memory session replication through a combination of serializable session attributes, "sticky sessions", which are provided by the load balancer, and specialized components configured in Tomcat's XML configuration files.

Serializable Session Attributes:

In order to use Tomcat's built-in replication, any session attribute or class that will need to be available in the event of failover must implement java.io.Serializable. This interface allows the JVM to convert session objects into bytecode that can be stored in memory and copied to other instances.

How to configure Tomcat Cluster (server.xml)?

The server.xml file is located in $CATALINA_HOME/conf directory. Because not every server configuration requires clustering, the Cluster element is commented out by default. Open server.xml and un-comment the Cluster element.

<Engine name="Catalina" defaultHost="www.mysite.com" jvmRoute="[worker name]">
    <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
        <Manager 
            className="org.apache.catalina.ha.session.DeltaManager" 
            expireSessionsOnShutdown="false" 
            notifyListenersOnReplication="true"
        />
        <Channel className="org.apache.catalina.tribes.group.GroupChannel">
            <Membership 
                className="org.apache.catalina.tribes.membershipt.McastService" 
                address="228.0.0.4" 
                port="45564" 
                frequency="500" 
                dropTime="3000"
            />
            <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
                <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
            </Sender>
            <Receiver 
                className="org.apache.catalina.tribes.transport.nio.NioReceiver" 
                address="auto" 
                port="4000" 
                autoBind="100" 
                selectorTimeout="5000" 
                maxThreads="6"
            />
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
        </Channel>
        <!-- <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/> -->
        <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;" />

        <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
        <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionDBinderListener"/>
        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
    </Cluster>
</Engine>

Here is another example (see http://www.ramkitech.com/2012/11/tomcat-clustering-series-part-3-session.html):

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
    <Manager className="org.apache.catalina.ha.session.DeltaManager"
            expireSessionsOnShutdown="false"
            notifyListenersOnReplication="true"/>
    <Channel className="org.apache.catalina.tribes.group.GroupChannel">
        <Membership className="org.apache.catalina.tribes.membership.McastService"
            address="228.0.0.4"
            port="45564"
            frequency="500"
            dropTime="3000"
        />
        <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
        </Sender>
        <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
            address="auto"
            port="4000"
            autoBind="100"
            selectorTimeout="5000"
            maxThreads="6"
        />
        <Interceptor  
            className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"
        />
        <Interceptor 
            className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"
        />
    </Channel>
    <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""/>
    <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
    <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
    <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>

In the Receiver element, we define which port the receiver can bind and use for receiving the session replication information. The address is our system IP address and port is any unused port. The address="auto" automatically pick the system IP address.

The TcpFailureDetector interceptor detects dead instances. In some case, multicast messages are delayed, so all tomcat instances think that a particular tomcat instance is dead, but this interceptor will make a TCP unicast to the failed tomcat instance and verify that this instance is actually dead or not.

The JvmRouteSessionIDBinderListener appends the appropriate tomcat instance to the end of the session ID.

Does mod_jk support the use of variables in its configuration file?

Yes. The mod_jk worker configuration file supports the use of variables to make adding new clusters or migrating configuration to a new server an easier process. Variables can be used in place of any directive-defined value. It's very simple to define a variable. See below.

How to define a variable?

[variable_name]=[value]

Make sure that you do not use a word that is already associated with a specific function.

What are the restrictions on variable names?

Variable names can contain alphanumeric characters, dashes, underscores, and periods, and are not case sensitive.

How to use a variable?

mynetwork=193.228.43
worker.worker1.host=$(variable_name).12

When defining a worker, can we use variable?

Yes. You can define and use variables in the worker.properties file.

When defining a worker, how can we define a variable?

To define a variable, use the following syntax:

<variable_name>=<value>

When defining a worker, can we use dots as part of the variable name to denote hierarchy?

Yes. Dots are allowed in the variable name, but you have to be careful not to use variable names that clash with the standard directives. Therefore, variable names should never start with "worker".

When defining a worker, how can we use a variable?

To use a variable, you can insert "$(variable_name)" at any place on the value side of a property line.

When defining a worker, if a variable is used before it is defined, what happens?

If a variable has not been defined before it is used, Tomcat will search the process environment for a variable with the same name and use its value.

What is property inheritance?

To reduce duplication of configuration lines and to ease maintenance of the worker.properties file, you can inherit properties from one worker to another worker, or even from a template to real workers.

How can we define one worker so that it inherit properties from another worker?

The directive "reference" allows us to copy configuration between workers in a hierarchical way. For example, if worker castor set worker.caster.reference=worker.pollux then it inherits all properties of pollux, except for the ones that are explicitly set for castor.

How to define a template worker?

To define a template worker, simply define it like a real worker, but do not add it to the worker.list or as a member of any load balancer. Such a template worker does not have to contain mandatory directives.

Configuration Directives:

See Tomcat Cluster Directives

References:

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License