[toc]

nginx+keepalived实现负载均衡高可用

1.keepalived简介

Keepalived是用C语言编写的路由软件。该项目的主要目标是为Linux系统和基于Linux的基础架构提供负载均衡和高可用性的简单而强大的功能。负载平衡框架依赖于提供第4层负载平衡的著名且广泛使用的Linux虚拟服务器（IPVS）内核模块。Keepalived实现了一组检查器，以根据其运行状况动态，自适应地维护和管理负载平衡的服务器池。另一方面，VRRP实现了高可用性协议。VRRP是路由器故障转移的基础。此外，Keepalived还实现了一组VRRP有限状态机的钩，从而提供了低级和高速协议交互。为了提供最快的网络故障检测，Keepalived实施BFD协议。VRRP状态转换可以考虑BFD提示来驱动快速状态转换。Keepalived框架可以独立使用，也可以一起使用以提供弹性基础架构。

2.keepalived负载均衡高可用配置过程

实验环境

角色	IP	主机名
keepalived-master	10.0.0.10	keepalived01
keepalived-slave	10.0.0.11	keepalived02
web01	10.0.0.51	nginx01
web02	10.0.0.52	nginx02

2.1 负载均衡端配置

2.1.1 安装keepalived

yum -y install keepalived

centos7.7中默认的keepalive版本是1.3.5，如果需要高版本请到keepalived github地址或者keepalived官网下载，这里仅做实验直接yum安装

$ keepalived -version
Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

2.1.2 lb01(keepalived master)编辑配置文件

lb01编辑keepalived配置文件/etc/keepalived/keepalived.conf

#备份原有文件
mv /etc/keepalived/keepalived.conf{,.bak}

#重新编辑文件
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {     
    router_id lb01
}
      
vrrp_instance VI_1 {
    state MASTER			        
    interface eth0			    
    virtual_router_id 50		   
    priority 150				
    advert_int 1				 
    authentication {			
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100		
    }
}
EOF

配置文件参数含义

#配置含义
global_defs {     
    router_id lb01   			#表示id身份，名称随意
}
      
vrrp_instance VI_1 {
    state MASTER			    #状态，角色为master
    interface eth0			    #VIP绑定的网卡
    virtual_router_id 50		#主和备的虚拟id号要相同，表明在一个组当中，名称随意，不要超过255
    priority 150				#优先级，越大优先级越高
    advert_int 1				#心跳检测，单位秒，备机每隔1秒检测一次主的存活状态 
    authentication {			#认证信息
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100				#虚拟IP
    }
}

2.1.3 lb02(keepalived backup)编辑配置文件

#备份原有文件
mv /etc/keepalived/keepalived.conf{,.bak}

#重新编辑文件
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {     
    router_id lb02
}
      
vrrp_instance VI_1 {
    state BACKUP			
    interface eth0			
    virtual_router_id 50		                                                 
    priority 100				
    advert_int 1				
    authentication {			
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100
    }
}
EOF

配置文件参数含义

global_defs {     
    router_id lb02  			#表示id身份，名称随意
}
      
vrrp_instance VI_1 {
    state BACKUP			    #状态，因为是slave，所以是BACKUP
    interface eth0			    #VIP绑定的网卡
    virtual_router_id 50		#主和备的虚拟id号要相同，表明在一个组当中，名称随意，不要                                                     超过255
    priority 100				#优先级，越大优先级越高
    advert_int 1				#心跳检测，单位秒，备机每隔1秒检测一次主的存活状态 
    authentication {			#认证信息
        auth_type PASS
        auth_pass 123456
}
    virtual_ipaddress {
        10.0.0.100				#虚拟IP
    }
}

对比keepalived的master与backup配置的区别

Keepalived配置区别	Master配置	Backup节配置
route_id(唯一标识)	route_id lb01	route_id lb02
state(角色状态)	state Master	state Backup
priority(竞选优先级)	priority 150	priority 100

2.1.4 启动keepalived

lb01和lb02相同操作

systemctl start keepalived && systemctl enable keepalived

2.1.5 检测lb01是否有vip

可以看到，启动keepalived后，lb01 eth0网卡就有了VIP 10.0.0.100

$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:f3:3c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fef3:3c78/64 scope link
       valid_lft forever preferred_lft forever

2.1.6 验证VIP是否能够正常漂移

提示

VIP首先在master上，当master宕机后，VIP会漂移至backup，当master恢复时VIP会自动漂移回来！！！

先停止lb01上的keepalived

systemctl stop keepalived

查看lb01上是否有VIP

可以看到，VIP此时已经没有了

$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:f3:3c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fef3:3c78/64 scope link
       valid_lft forever preferred_lft forever

查看lb02上是否有VIP

当keepalived master宕机后，VIP就会漂移到lb02

$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:c1:0b:16 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.11/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fec1:b16/64 scope link
       valid_lft forever preferred_lft forever

重新启动lb01上的keepalived

systemctl start keepalived

#查看VIP
$ ip a s eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:f3:3c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::21c:42ff:fef3:3c78/64 scope link
       valid_lft forever preferred_lft forever

2.1.7 编辑nginx配置文件

lb01和lb02编辑nginx配置文件

cat > /etc/nginx/conf.d/kd.test.conf <<'EOF'
upstream kd {
  server 10.0.0.51;
  server 10.0.0.52;
}

server {
  listen 80;
  server_name _;

  location / {
    proxy_pass http://kd;
  }
}
EOF

检测nginx语法并重载nginx

$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ nginx -s reload

2.2 后端真实服务web配置

web01操作

2.2.1 编辑nginx配置文件

cat >> /etc/nginx/conf.d/kd.test.conf <<EOF
server {
 listen 80;
 server_name _;
 root /code;
 index index.html;
}
EOF

2.2.2 创建网站根目录

mkdir /code && echo "web01 web01 web01" > /code/index.html

2.2.3 检测nginx语法并重载nginx

$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ nginx -s reload

web02操作

2.2.4 编辑nginx配置文件

cat >> /etc/nginx/conf.d/kd.test.conf <<EOF 
server {
	listen 80;
	server_name _;
	root /code;
	index index.html;
}
EOF

2.2.5 创建网站根目录

mkdir /code && echo "web02 web02 web02" > /code/index.html

2.2.6 检测nginx语法并重载nginx

$ nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

$ nginx -s reload

2.3 浏览器验证负载均衡是否正常工作

浏览器访问VIP 10.0.0.100

3.nginx宕机问题

如果Nginx宕机，会导致用户请求失败，但Keepalived并不会进行切换，所以编写nginx检测脚本，检测nginx存活状态，不存活则kill掉nginx和keepalived，然后VIP进行漂移

3.1 编辑脚本

lb01操作

#创建存放脚本的目录
mkdir -p /server/scripts

#编辑脚本内容
cat >> /server/scripts/check_web.sh <<\EOF
#!/bin/sh
#使用while死循环
while true;do
nginxpid=$(ps -C nginx --no-header|wc -l)
#判断Nginx是否存活,如果不存活则尝试启动Nginx
if [ $nginxpid -eq 0 ];then
    systemctl start nginx
    sleep 5
    #5秒后再次获取一次Nginx状态
    nginxpid=$(ps -C nginx --no-header|wc -l) 
    #再次进行判断, 如Nginx还不存活则停止Keepalived,让地址进行漂移,并退出脚本  
    if [ $nginxpid -eq 0 ];then
        systemctl stop keepalived
        exit 1
   fi
fi
     sleep 5
done
EOF

#给脚本赋予执行权限
chmod +x /server/scripts/check_web.sh

#后台运行脚本
nohup sh /server/scripts/check_web.sh &

3.2 验证脚本是否生效

故意把nginx配置文件修改错误(比如删除一个括号或者分号)，然后停止lb01上的nginx，因为上边的脚本中会判断nginx如果不存活就启动nginx，把nginx配置文件改错了nginx就无法正常启动过了

systemctl stop nginx

后台脚本检测到nginx不存活会kill掉keepalived，此时浏览器访问会报错，隔几秒就恢复了，因为VIP已经漂移至lb02上

kdnginx

3.3 在keepalived中调用检测脚本

提示

nohup sh /server/scripts/check_web.sh & 执行脚本属于手动执行，还可以在keepalived配置文件中配置调用监控脚本

lb01、lb02都需要操作

编辑keepalived配置文件/etc/keepalived/keepalived.conf

#在global_defs标签下增加如下内容
vrrp_script check_web {
    script "/server/scripts/check_web.sh"
    interval 10
    weight 50
}

#track_script标签要写在vrrp_instance VI_1{}中
track_script {
    check_web
}

警告

vrrp_script中的interval时间需大于脚本中的sleep时间

否则会报错Keepalived_vrrp[2612]: /server/scripts/check_web.sh exited due to signal 15

这样就能够使keepalived自动调用监控脚本了，keepalived会根据配置文件中vrrp_script {}中的interval参数来决定每隔几秒执行脚本，interval后面的数字就表明执行脚本的间隔时间

在脚本中我们定义了，keepalived master上检测nginx是否存活，不存活尝试启动nginx，当无法成功启动nginx的时候停止master上的keepalived，然后让VIP漂移，这样的话就实现了当nginx服务挂掉时keepalived VIP能够漂移从而不影响客户端访问

4.keepalived高可用脑裂

概念

脑裂就是由于某些原因，导致两台keepalived高可用服务器在指定时间内，无法检测到对方的心跳消息，各自取得资源及服务的所有权，而此时的两台高可用服务器又都还活着，产生裂脑的情况下，master和backup上都会有VIP

产生脑裂的原因

1.服务器网络故障

2.服务器硬件故障发生损坏现象而崩溃

3.主备都开启firewalld防火墙

4.1 模拟脑裂

lb01和lb02开启firewalld防火墙

systemctl start firewalld

开启防火墙后产生裂脑原因

信息

lb01和lb02都开启防火墙后，因为没有允许vrrp，当backup去检测master的时候无法检测到，此时backup认为master已经宕机，所以将VIP抢占；而master实际上并没有宕机，所以VIP不会漂移，这样就造成了master和backup都在抢占VIP。此时就出现了裂脑的情况

4.2 测试脑裂问题

在keepalived备上编写检测脚本, 测试如果能ping通主keepalived并且备节点还有VIP的话则认为产生了脑裂

lb02操作

#创建存放脚本的目录
mkdir -p /server/scripts

#编辑脚本内容
cat >/server/scripts/check_split_brain.sh <<'EOF'
#!/bin/sh
lb01_vip=10.0.0.10
lb01_ip=10.0.0.11
while true;do
    ping -c 2 -W 3 $lb01_ip &>/dev/null
    if [ $? -eq 0 -a `ip add|grep "$lb01_vip"|wc -l` -eq 1 ];then
        echo "keepalived出现了脑裂！！！"
    else
        echo "keepalived完全objk"
    fi
sleep 5
done
EOF

运行脚本，因为此时是脑裂的，keepalived主备上都有VIP，因为前期会报出现脑裂，当把主备上的firewalld关掉时就没有问题了

#脚本会一直运行，ctrl+c停止
$ sh /server/scripts/check_split_brain.sh
keepalived出现了脑裂！！！
keepalived出现了脑裂！！！
keepalived出现了脑裂！！！
keepalived出现了脑裂！！！
keepalived出现了脑裂！！！
keepalived出现了脑裂！！！
keepalived出现了脑裂！！！
keepalived完全objk
keepalived完全objk

5.源码安装keepalived

keepalived源码包官方下载地址

5.1 下载包

wget https://www.keepalived.org/software/keepalived-2.1.2.tar.gz

5.2 编译安装

安装依赖包

yum -y install openssl openssl-devel libnl3-devel pcre-devel

解压缩包

tar xf keepalived-2.1.2.tar.gz && cd keepalived-2.1.2

编译安装

./configure --prefix=/usr/local/keepalived
make && make install

5.3 配置keepalived

5.3.1 软连接keepalived命令

ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin

验证

$ keepalived -v
Keepalived v2.1.2 (06/14,2020)

Copyright(C) 2001-2020 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 3.10.0
Running on Linux 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020

configure options: --prefix=/etc/keepalived --sysconfdir=/etc/keepalived

Config options:  LVS VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTING

System options:  PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV6_ADVANCED_API RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA FRA_OIFNAME IFA_FLAGS IP_MULTICAST_ALL NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION VRRP_VMAC IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE SO_MARK SCHED_RESET_ON_FORK

5.3.2 keepalived.service文件

信息

keepalived编译安装完成后会自动生成文件/usr/lib/systemd/system/keepalived.service

5.3.3 拷贝安装目录下keepalived配置文件

否则会报错Unable to find configuration file /etc/keepalived/keepalived.conf (glob returned 3)

mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

5.3.4 拷贝启动文件(可选)

cp /root/keepalived-2.1.2/keepalived/etc/init.d/keepalived /etc/init.d

5.3.5 启动keepalived并设置开机自启

systemctl daemon-reload && systemctl enable keepalived && systemctl start keepalived

1.keepalived简介​

2.keepalived负载均衡高可用配置过程​

2.1 负载均衡端配置​

2.1.1 安装keepalived​

2.1.2 lb01(keepalived master)编辑配置文件​

2.1.3 lb02(keepalived backup)编辑配置文件​

2.1.4 启动keepalived​

2.1.5 检测lb01是否有vip​

2.1.6 验证VIP是否能够正常漂移​

2.1.7 编辑nginx配置文件​

2.2 后端真实服务web配置​

web01操作

2.2.1 编辑nginx配置文件​

2.2.2 创建网站根目录​

2.2.3 检测nginx语法并重载nginx​

web02操作

2.2.4 编辑nginx配置文件​

2.2.5 创建网站根目录​

2.2.6 检测nginx语法并重载nginx​

2.3 浏览器验证负载均衡是否正常工作​

3.nginx宕机问题​

3.1 编辑脚本​

3.2 验证脚本是否生效​

3.3 在keepalived中调用检测脚本​

4.keepalived高可用脑裂​

4.1 模拟脑裂​

4.2 测试脑裂问题​

5.源码安装keepalived​

5.1 下载包​

5.2 编译安装​

5.3 配置keepalived​

5.3.1 软连接keepalived命令​

5.3.2 keepalived.service文件​

5.3.3 拷贝安装目录下keepalived配置文件​

5.3.4 拷贝启动文件(可选)​

5.3.5 启动keepalived并设置开机自启​