Jusene's Blog

Tomcat Cluster集群

web
字数统计: 2.7k阅读时长: 15 min
2017/06/07 Share

tomcat cluster

Tomcat很重要，当然也少不了它的集群构造，当然我们首先需要关注的是负载均衡集群，而在一个负载集群中会话保持是一个必须考虑的问题，而会话的保持我们也有许多种方法来实现。

cluster session

  1. session sticky
    source ip，cookie，hash Header
    实现的方法：
  • nginx：hash ip_hash
  • haproxy: source cookie hdr
  • lvs: SH
  • httpd: ROUTID
  1. session cluster
  • tomcat delta manager
  • tomcat session cluster
  1. session server
    memcached redis
    实现方法：
  • memcached-session-manager

httpd+tomcat负载均衡集群

(1)第一种方式
apache

  • mod_proxy
  • mod_proxy_http
  • mod_proxy_balancer
    tomcat
  • http connector

实现负载均衡：

1
<proxy balancer://tserver>
2
        BalancerMember http://10.211.55.43:8080 loadfactor=1 
3
        BalancerMember http://10.211.55.39:8080 loadfactor=1 
4
        ProxySet lbmethod=bybusyness
5
</proxy>
6
7
<VirtualHost *:80>
8
        ServerName www.jusene.com
9
        ProxyVia On
10
        ProxyRequests Off
11
        ProxyPreserveHost On
12
        <Proxy *>
13
                Require all granted
14
        </Proxy>
15
        ProxyPass / balancer://tserver/
16
        ProxyPassReverse / balancer://tserver/
17
        <Location />
18
                Require all granted
19
        </Location>
20
</VirtualHost>

实现会话保持：

1
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
2
<proxy balancer://tserver>
3
        BalancerMember http://10.211.55.43:8080  route=TomcatA  loadfactor=1
4
        BalancerMember http://10.211.55.39:8080  route=TomcatB  loadfactor=2
5
        ProxySet lbmethod=bybusyness
6
        ProxySet stickysession=ROUTEID
7
</proxy>
8
9
<VirtualHost *:80>
10
        ServerName www.jusene.com
11
        ProxyVia On
12
        ProxyRequests Off
13
        ProxyPreserveHost On
14
        <Proxy *>
15
                Require all granted
16
        </Proxy>
17
        ProxyPass / balancer://tserver/
18
        ProxyPassReverse / balancer://tserver/
19
        <Location />
20
                Require all granted
21
        </Location>
22
</VirtualHost>

启用管理接口:

1
<Location /balancer>
2
                SetHandler balancer-manager
3
                ProxyPass !
4
                Require ip 10.211.55.24
5
                Require all deny
6
</Location>

(2)第二种方式
apache

  • mod_proxy
  • mod_proxy_ajp
  • mod_proxy_balancer
    tomcat
  • ajp connector

实现负载均衡:

1
<proxy balancer://tserver>
2
        BalancerMember ajp://10.211.55.43:8009 loadfactor=1 
3
        BalancerMember ajp://10.211.55.39:8009 loadfactor=1 
4
        ProxySet lbmethod=bybusyness
5
</proxy>
6
7
<VirtualHost *:80>
8
        ServerName www.jusene.com
9
        ProxyVia On
10
        ProxyRequests Off
11
        ProxyPreserveHost On
12
        <Proxy *>
13
                Require all granted
14
        </Proxy>
15
        ProxyPass / balancer://tserver/
16
        ProxyPassReverse / balancer://tserver/
17
        <Location />
18
                Require all granted
19
        </Location>
20
</VirtualHost>

实现会话保持:

1
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
2
<proxy balancer://tserver>
3
        BalancerMember ajp://10.211.55.43:8009  route=TomcatA  loadfactor=1
4
        BalancerMember ajp://10.211.55.39:8009  route=TomcatB  loadfactor=2
5
        ProxySet lbmethod=bybusyness
6
        ProxySet stickysession=ROUTEID
7
</proxy>
8
9
<VirtualHost *:80>
10
        ServerName www.jusene.com
11
        ProxyVia On
12
        ProxyRequests Off
13
        ProxyPreserveHost On
14
        <Proxy *>
15
                Require all granted
16
        </Proxy>
17
        ProxyPass / balancer://tserver/
18
        ProxyPassReverse / balancer://tserver/
19
        <Location />
20
                Require all granted
21
        </Location>
22
</VirtualHost>

(3)第三种方式
apache

  • mod_jk
    tomcat
  • ajp connector

mod_jk的下载地址：http://mirror.bit.edu.cn/apache/tomcat/tomcat-connectors/jk/tomcat-connectors-1.2.42-src.tar.gz

1
~]# tar xf tomcat-connectors-1.2.42-src.tar.gz
2
~]# cd tomcat-connectors-1.2.42-src
3
~]# cd native
4
~]# ./configure --with-apxs=/usr/bin/apxs
5
~]# make && make install

反向代理实现:

1
LoadModule jk_module modules/mod_jk.so
2
3
JkWorkersFile /etc/httpd/conf.d/workers.properties
4
JkLogFile logs/mod_jk.log
5
JkLogLevel debug
6
JkMount /* TomcatA
7
JkMount /jk_status Stat
1
~]# cat /etc/httpd/conf.d/workers.properties
2
worker.list=TomcatA,Stat
3
worker.TomcatA.host=10.211.55.39
4
worker.TomcatA.port=8009
5
worker.TomcatA.type=ajp13
6
worker.Stat.type=status

负载均衡的实现:

1
LoadModule jk_module modules/mod_jk.so
2
3
JkWorkersFile /etc/httpd/conf.d/workers.properties
4
JkLogFile logs/mod_jk.log
5
JkLogLevel debug
6
JkMount /* tcserver
7
JkMount /jk_status Stat
1
worker.list=tcserver,Stat
2
worker.TomcatA.host=10.211.55.39
3
worker.TomcatA.port=8009
4
worker.TomcatA.type=ajp13
5
worker.TomcatA.lbfactor=1
6
worker.TomcatB.host=10.211.55.43
7
worker.TomcatB.port=8009
8
worker.TomcatB.type=ajp13
9
worker.TomcatB.lbfactor=2
10
worker.tcserver.type=lb
11
worker.tcserver.balance_workers=TomcatA,TomcatB
12
worker.Stat.type=status

会话保持：

1
LoadModule jk_module modules/mod_jk.so
2
3
JkWorkersFile /etc/httpd/conf.d/workers.properties
4
JkLogFile logs/mod_jk.log
5
JkLogLevel debug
6
JkMount /* tcserver
7
JkMount /jk_status Stat
1
worker.list=tcserver,Stat
2
worker.TomcatA.host=10.211.55.39
3
worker.TomcatA.port=8009
4
worker.TomcatA.type=ajp13
5
worker.TomcatA.lbfactor=1
6
worker.TomcatB.host=10.211.55.43
7
worker.TomcatB.port=8009
8
worker.TomcatB.type=ajp13
9
worker.TomcatB.lbfactor=2
10
worker.tcserver.type=lb
11
worker.tcserver.balance_workers=TomcatA,TomcatB
12
worker.tcserver.sticky_session=1
13
worker.Stat.type=status

这个还需要在每个tomcat的配置文件server.xml中engine中jvmRoute与worker的名称相同

Tomcat Session Replication Cluster

session测试页：

1
<%@ page language="java" %>
2
<html>
3
 <head><title>TomcatB</title></head>
4
 <body>
5
 <h1><font color="blue">TomcatB </h1>
6
 <table align="centre" border="1">
7
  <tr>
8
  <td>Session ID</td>
9
 <% session.setAttribute("abc","abc"); %>
10
  <td><%= session.getId() %></td>
11
  </tr>
12
  <tr>
13
  <td>Created on</td>
14
  <td><%= session.getCreationTime() %></td>
15
  </tr>
16
 </table>
17
 </body>
18
</html>

根据需要自行修改成不同

负载均衡配置:

1
<proxy balancer://tserver>
2
        BalancerMember http://10.211.55.43:8080 loadfactor=1 
3
        BalancerMember http://10.211.55.39:8080 loadfactor=1 
4
        ProxySet lbmethod=byrequests
5
</proxy>
6
7
<VirtualHost *:80>
8
        ServerName www.jusene.com
9
        ProxyVia On
10
        ProxyRequests Off
11
        ProxyPreserveHost On
12
        <Proxy *>
13
                Require all granted
14
        </Proxy>
15
        ProxyPass / balancer://tserver/
16
        ProxyPassReverse / balancer://tserver/
17
        <Location />
18
                Require all granted
19
        </Location>
20
</VirtualHost>

(1)配置启用集群
TomcatA

1
~]# vim /etc/tomcat/server.xml
2
...
3
<Host name="www.jusene.com"  appBase="webapps"
4
            unpackWARs="true" autoDeploy="true">
5
            <Context path="/" docBase="/usr/share/tomcat/webapps/myapp" reloadable="true">
6
            <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
7
                 channelSendOptions="8">
8
9
          <Manager className="org.apache.catalina.ha.session.DeltaManager"
10
                   expireSessionsOnShutdown="false"
11
                   notifyListenersOnReplication="true"/>
12
13
          <Channel className="org.apache.catalina.tribes.group.GroupChannel">
14
            <Membership className="org.apache.catalina.tribes.membership.McastService"
15
                        address="228.0.0.4"      #集群成员通信组播地址
16
                        port="45564"			 #心跳端口
17
                        frequency="500"          #每隔500ms检查一次
18
                        dropTime="3000"/>        #3000ms检测未成功剔除集群
19
            <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
20
                      address="auto"             #session通信地址，为外网可通信端口
21
                      port="4000"                #session通信端口
22
                      autoBind="100"
23
                      selectorTimeout="5000"
24
                      maxThreads="6"/>
25
26
            <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
27
              <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
28
            </Sender>
29
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
30
            <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
31
          </Channel>
32
33
          <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
34
                 filter=""/>
35
          <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
36
37
          <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
38
                    tempDir="/tmp/war-temp/"
39
                    deployDir="/tmp/war-deploy/"
40
                    watchDir="/tmp/war-listen/"
41
                    watchEnabled="false"/>
42
43
          <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
44
          <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
45
        </Cluster>
46
~]# scp server.xml node2:/etc/tomcat/server.xml
47
~]# cp web.xml /usr/share/tomcat/webapps/myapp/WEB-INF
48
~]# cd /usr/share/tomcat/webapps/myapp/WEB-INF
49
~]# cat web.xml
50
...
51
52
<distributable/>
53
~]# scp web.xml node2:/usr/share/tomcat/webapps/myapp/WEB-INF
54
~]# systemctl start tomcat;ssh node2 'systemctl start tomcat'

(2)配置webapps
编辑WEB-INF/web.xml,添加<distributable/>
(3)启动tomcat

session server

memcached-session-manager项目地址：http://code.google.com/p/memcached-session-manager/

下载如下jar文件至各tomcat节点的tomcat安装目录下的lib目录中，其中的${version}要换成你所需要的版本号，tc${6,7,8}要换成与tomcat版本相同的版本号。
memcached-session-manager-${version}.jar
memcached-session-manager-tc${6,7,8}-${version}.jar
spymemcached-${version}.jar
msm-javolution-serializer-${version}.jar
javolution-${version}.jar

负载均衡配置:

1
<proxy banalcer://tcserver>
2
	BalancerMember http://10.211.55.43:8080 loadfactor=1
3
	BalancerMember http://10.211.55.39:8080 loadfactor=1
4
	ProxySet lbmethod=byrequests
5
</proxy>
6
7
<VirtualHost *:80>
8
	ServerName www.jusene.com
9
	ProxyVia On
10
	ProxyRequests Off
11
	ProxyPreserveHost On
12
	<Proxy *>
13
		Require all granted
14
	</Proxy>
15
	ProxyPass / balancer://tserver/
16
    ProxyPassReverse / balancer://tserver/
17
    <Location />
18
        Require all granted
19
    </Location>
20
</VirtualHost>

TomcatA:

1
~]# yum install -y memcached
2
~]# systemctl start memcached
3
~]# vim /etc/tomcat/server.xml
4
....
5
<Host name="localhost"  appBase="webapps"
6
            unpackWARs="true" autoDeploy="true">
7
        <Context path="/" docBase="/usr/share/tomcat/webapps/myapp" reloadable="true">
8
              <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
9
                memcachedNodes="n1:10.211.55.43:11211,n2:10.211.55.39:11211"
10
                failoverNodes="n1"
11
                requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
12
                transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
13
              />
14
             </Context>
15
...
16
~]# systemctl start tomcat
17
~]# cat /usr/share/tomcat/webapps/myapp/index.jsp
18
<%@ page language="java" %>
19
<html>
20
 <head><title>TomcatA</title></head>
21
 <body>
22
 <h1><font color="blue">TomcatA </h1>
23
 <table align="centre" border="1">
24
  <tr>
25
  <td>Session ID</td>
26
 <% session.setAttribute("abc","abc"); %>
27
  <td><%= session.getId() %></td>
28
  </tr>
29
  <tr>
30
  <td>Created on</td>
31
  <td><%= session.getCreationTime() %></td>
32
  </tr>
33
 </table>
34
 </body>
35
</html>

TomcatB:

1
~]# yum install -y memcached
2
~]# systemctl start memcached
3
~]# vim /etc/tomcat/server.xml
4
<Host name="localhost"  appBase="webapps"
5
            unpackWARs="true" autoDeploy="true">
6
        <Context path="/" docBase="/usr/share/tomcat/webapps/myapp" reloadable="true">
7
              <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
8
                memcachedNodes="n1:10.211.55.43:11211,n2:10.211.55.39:11211"
9
                failoverNodes="n1"
10
                requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
11
                transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
12
              />
13
             </Context>
14
~]# systemctl start tomcat
15
~]# cat /usr/share/tomcat/webapps/myapp/index.jsp
16
<%@ page language="java" %>
17
<html>
18
 <head><title>TomcatB</title></head>
19
 <body>
20
 <h1><font color="blue">TomcatB </h1>
21
 <table align="centre" border="1">
22
  <tr>
23
  <td>Session ID</td>
24
 <% session.setAttribute("abc","abc"); %>
25
  <td><%= session.getId() %></td>
26
  </tr>
27
  <tr>
28
  <td>Created on</td>
29
  <td><%= session.getCreationTime() %></td>
30
  </tr>
31
 </table>
32
 </body>
33
</html>

结果：

前端负载均衡无论分发到那台机器上，会话id我们都是一样的保持，即实现了负载均衡有实现了会话保持。

原文作者：Zhang Jusene

原文链接：http://jusene.github.io/2017/06/07/tomcat-cluster/

发表日期：June 7th 2017, 8:59:01 pm

更新日期：November 30th 2019, 1:54:24 am

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. tomcat cluster
    1. 1.1. cluster session
  2. 2. httpd+tomcat负载均衡集群
  3. 3. Tomcat Session Replication Cluster
  4. 4. session server
Total : 190
2020
2019
2018
2017
python boot troubshooting docker drbd ssh web ELK kubernetes Kubernetes git ansible tools dns corosync consul Django etcd security ftp Go bash bigdata grub heartbeat Java kickstart kvm lvs mongodb mysql mesh nc nfs mogilefs openresty upgrade puppet rabbitmq saltstack rsync screen socat svn tcpdump fastdfs haproxy iscsi keepalived cache openssl vpn optimization prometheus redis RPM cncf supervisord kernel JAVA
python Linux troubshooting Docker docker storage nginx Nosql kubernetes Flask Kubernetes ansible Shell cluster Django Go bigdata boot Java hypervisor mysql mesh tools openresty puppet rabbitmq saltstack optimization cncf varnish tomcat JAVA