Configuring JGroups and Alfresco Clusters

From AlfrescoWiki

Jump to: navigation, search

Contents

[edit] Introduction

*** JGroups functionality was introduced in Enterprise V3.1 ***

Alfresco requires servers to discover each other on a network in order to set up cluster communications. Before V3.1, this discovery process was done using a UDP multicast message (provided by EHCache); servers in the cluster picked the message up and used the information to set up inter-server communication for inter-cache communication.

Alfresco limited some installations by not providing a more flexible cluster discovery process, which is why JGroups was integrated into the repository. JGroups is a toolkit for multicast communication between servers. It allows inter-server communication using a highly configurable transport stack, which includes UDP and TCP protocols. Additionally, JGroups manages the underlying communication channels and cluster entry and exit.

This page covers the options available for configuring JGroups and other Alfresco-specific cluster options for V3.1.

[edit] Core JGroups Support

JGroups is supported in both the Open and Enterprise code lines, but is only used for cache communication in Enterprise. The core setup of JGroups is therefore common to both code streams.

[edit] JGroups Configuration

NB: The JGroups cluster will only initialize if the following property is defined: 

 alfresco.cluster.name=<CLUSTERNAME>

Once the cluster name is specified, JGroups uses that name to uniquely distinguish inter-server communication. This allows machines to use the same protocol stacks, but to ignore broadcasts from different clusters.

The default JGroups configuration files are located at <configRoot>/alfresco/jgroups/alfresco-jgroups-XYZ.xml, where XYZ is the name of the protocol being used. There is a separate configuration file for each stack. Switch between the default TCP and UDP stacks using: 

 alfresco.jgroups.defaultProtocol=<STACKNAME>

You can also point to a completely new configuration file of your own making using: 

 alfresco.jgroups.configLocation=classpath:some-classpath.xml
 alfresco.jgroups.configLocation=file:some-file-path.xml

Within the JGroups configuration files, there are parameters that can be passed, for example, see <configRoot>/alfresco/jgroups/alfresco-jgroups-TCP.xml: 

<config>
    <TCP bind_port="${alfresco.tcp.start_port:7800}"
...
    <TCPPING timeout="3000"
             initial_hosts="${alfresco.tcp.initial_hosts:localhost[7800]}"
             port_range="${alfresco.tcp.port_range:3}"
             num_initial_members="2"/>
    <MERGE2 max_interval="30000"
...
</config>

The variable substitution if provided natively by JGroups; properties are taken from the JVM's system properties. In order to support setting JGroups properties in either the system properties or directly on the command line - and not only the latter - Alfresco has a bean that pushes the JGroups properties from the standard properties into the JVM properties:

<configRoot>/alfresco/core-services-context.xml: 

    <bean id="jgroupsPropertySetter" class="org.alfresco.config.SystemPropertiesSetterBean" init-method="init">
        <property name="propertyMap">
            <map>
                <entry key="jgroups.bind_addr">
                    <value>${alfresco.jgroups.bind_address}</value>
                </entry>
                <entry key="jgroups.bind_interface">
                    <value>${alfresco.jgroups.bind_interface}</value>
                </entry>
                <entry key="alfresco.tcp.start_port">
                    <value>${alfresco.tcp.start_port}</value>
                </entry>
                <entry key="alfresco.tcp.initial_hosts">
                    <value>${alfresco.tcp.initial_hosts}</value>
                </entry>
                <entry key="alfresco.tcp.port_range">
                    <value>${alfresco.tcp.port_range}</value>
                </entry>
                <entry key="alfresco.udp.mcast_addr">
                    <value>${alfresco.udp.mcast_addr}</value>
                </entry>
                <entry key="alfresco.udp.mcast_port">
                    <value>${alfresco.udp.mcast_port}</value>
                </entry>
            </map>
        </property>
    </bean>

If the JGroups config is changed or extended and the preferred means of setting properties is NOT via the system (-D options) properties, then override the bean or add a new bean in your extension location. Values set directly on the VM using the -D options are always taken in preference in this case.

[edit] JGroups Properties

  • General
    • alfresco.jgroups.bind_address (default: ): The address to bind local sockets to
    • alfresco.jgroups.bind_interface (default: ): The interface to bind local sockets to
  • UDP Stack
    • alfresco.udp.mcast_addr (default: 230.0.0.1): The multicast address to broadcast on
    • alfresco.udp.mcast_port (default: 4446): The port to use for UDP multicast broadcasts
    • alfresco.udp.ip_ttl (default: 2): The multicast "time to live" to control the packet scope
  • TCP Stack
    • alfresco.tcp.start_port
      • The port that the server will start listening on
      • default: 7800
      • e.g.: 7800
    • alfresco.tcp.initial_hosts
      • A list of hosts and start ports that must be pinged. This can all potential members of the cluster, including the current server and servers that might not be available. The port listed in square brackets is the port to start pinging.
      • default: localhost[7800]
      • e.g.: HOST_A[7800],HOST_B[7800]
    • alfresco.tcp.port_range
      • The number of increments to make to each host's port number during scanning. Each host has a port number in square brackets e.g. 7800. If the host does not respond to the ping message, the port number will be increased and another attempt made.

[edit] Clustering the Repository

[edit] What Has Changed?

  • JGroups is now used to send the initial broadcast messages announcing a server's availability.
  • After initial setup, it is possible to have a server enter a cluster by setting a single property: alfresco.cluster.name.

[edit] Steps to Initiate Clustering

  1. Activate the ehcache-custom.xml.sample.cluster
    • This has been modified and you should take the latest version that ships with V3.1.
  2. Set the required properties using overrides (e.g. custom-repository.properties) or system properties (Java -D options)
    • alfresco.cluster.name: The name of the cluster. This is new and critical. Without it, neither JGroups nor index tracking will be enabled.
    • Any JGroups properties, especially if the TCP stack is required
    • index.recovery.mode: Set this to AUTO to ensure indexes are refreshed properly on startup
    • index.tracking.cronExpression: This does not need to be set and is 0/5 * * * * ? by default. The index tracking code will not activate unless the cluster name has been set!
  3. Configure the content stores
    • This has not changed from previous versions

[edit] If You Do Not Want JGroups

It is still possible to use the EHCache multicast discovery. Replace the cacheManagerPeerProviderFactory in your EHCache custom config file as follows:

<extensionRoot>/alfresco/extension/ehcache-custom.xml: 

    <cacheManagerPeerProviderFactory
            class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
            properties="peerDiscovery=automatic,
                        multicastGroupAddress=230.0.0.1,
                        multicastGroupPort=4446"/>

You will still need to set the alfresco.cluster.name property in order to activate index tracking.

[edit] Logging

  • org.alfresco.enterprise.repo.cache.jgroups
    • INFO: Watch entry and exit of cluster members
    • DEBUG: Verbose output on heartbeat messages sent and received by machines in the cluster as well as the above
  • More...

[edit] Is it Working?

The cluster checks have not changed: Testing the Cluster

Retrieved from "http://wiki.alfresco.com/wiki/Configuring_JGroups_and_Alfresco_Clusters"