redis集群搭建

Redis系列笔记

第一篇:redis基础命令
第二篇:redis常见应用场景

---

集群命令

cluster info打印集群的信息。
cluster nodes列出集群当前已知的所有节点（node）的相关信息。
cluster  meet  将ip和port所指定的节点添加到集群当中。
cluster addslots  [slot ...]将一个或多个槽（slot）指派（assign）给当前节点。
cluster delslots  [slot ...]移除一个或多个槽对当前节点的指派。
cluster slots列出槽位、节点信息。
cluster slaves 列出指定节点下面的从节点信息。 比如  cluster slaves 11f9169577352c33d85ad0d1ca5f5bf0deba3209  这个实际查的是nodes.conf
cluster replicate 将当前节点设置为指定节点的从节点。
cluster saveconfig   手动执行命令保存保存集群的配置文件，集群默认在配置修改的时候会自动保存配置文件。
cluster keyslot    列出key被放置在哪个槽上。 例如cluster keyslot 9223372036854742675
cluster flushslots   移除指派给当前节点的所有槽，让当前节点变成一个没有指派任何槽的节点。
cluster countkeysinslot    返回槽目前包含的键值对数量。
cluster getkeysinslot  返回count个槽中的键。  比如cluster getkeysinslot 202 3   
cluster setslot  node  将槽指派给指定的节点，如果槽已经指派给另一个节点，那么先让另一个节点删除该槽，然后再进行指派。  
cluster setslot  migrating  将本节点的槽迁移到指定的节点中。  
cluster setslot  importing  从 node_id 指定的节点中导入槽 slot 到本节点。  
cluster setslot  stable 取消对槽 slot 的导入（import）或者迁移（migrate）。 
cluster failover　　　　  手动进行故障转移。
cluster fet  从集群中移除指定的节点，这样就无法完成握手，过期时为60s，60s后两节点又会继续完成握手。
cluster reset [HARD|SOFT]重置集群信息，soft是清空其他节点的信息，但不修改自己的id，hard还会修改自己的id，不传该参数则使用soft方式。
cluster count-failure-reports 列出某个节点的故障报告的长度。
cluster SET-CONFIG-EPOCH 设置节点epoch，只有在节点加入集群前才能设置。

集群环境搭建

搭建集群至少需要6台服务器或虚拟机，如果想要搭建测试可以根据端口号模拟服务器虚拟运行6个redis实例。

（1）下载redis安装包，编译安装，指定目录/usr/local/redis/
（2）创建/usr/local/redis-cluster/目录，然后把安装目录下的bin目录复制到redis-cluster/下重命名为redis01~06
（3）修改配置文件

• #bind 127.0.0.1 注释，允许其他ip访问
• protected-mode no # 改为no; yes 保护模式，只允许本地链接，no 保护模式关闭
• port 7001/7002/7003/7004/7005/7006 # 六个配置文件分别配置不同端口
• daemonize: yes # 后台运行
• masterauth 1234 # 主机密码
• requirepass 1234 # 本机密码
• cluster-enabled yes # 开启集群
• cluster-enabled cluster-config-file nodes-7001.conf # 集群配置文件；集群创建成功会自动生成不配置默认nodes.conf
• cluster-node-timeout 15000 # 节点失联时间，超过该时间（毫秒），集群自动进行主从切换。
• appendonlyyes # 开启AOF持久化

（4）创建集群，中途会有个提示，输入yes回车即可

• /usr/local/redis/bin/redis-cli --cluster create --cluster-replicas 1 182.254.134.174:7001 182.254.134.174:7002 182.254.134.174:7003 182.254.134.174:7004 182.254.134.174:7005 182.254.134.174:7006 -a 1234

[root@VM-0-12-centos /]# cd /usr/local/redis
[root@VM-0-12-centos redis]# get https://github./redis/redis/archive/7.0.4.tar.gz
[root@VM-0-12-centos redis]# tar -xzvf 7.0.4.tar.gz
[root@VM-0-12-centos redis]# ll
total 2932
-r-r--r-- 1 root root 2994242 Aug  8 17:03 7.0.4.tar.gz
drxrxr-x 8 root root 4096 Jul 18 21:04 redis-7.0.4
[root@VM-0-12-centos redis]# cd redis-7.0.4
[root@VM-0-12-centos redis]# make && make install PREFIX=/usr/local/redis  # 指定安装目录,会在redis目录下生成一个bin/目录,如果不指定安装目录会在 redis-7.0.4/src/ 生成启动文件，我这里为了自己操作方便些直接在 redis/bin 目录下操作
[root@VM-0-12-centos redis]# cd ..
[root@VM-0-12-centos redis]# ls
7.0.4.tar.gz  bin  redis-7.0.4
[root@VM-0-12-centos redis]# cd bin/
[root@VM-0-12-centos bin]# ls
redis-benchmark  redis-check-aof  redis-check-rdb  redis-cli  redis-sentinel  redis-server
[root@VM-0-12-centos bin]# cp ../redis-7.0.4/redis.conf ./
[root@VM-0-12-centos bin]# ls
redis-benchmark  redis-check-aof  redis-check-rdb  redis-cli  redis.conf  redis-sentinel  redis-server
[root@VM-0-12-centos bin]# vim ./redis.conf # 修改daemonize 的值为 yes，开启后台启动
[root@VM-0-12-centos bin]# ./redis-server ./redis.conf
[root@VM-0-12-centos bin]# ./redis-cli
127.0.0.1:6379>
root@VM-0-12-centos bin]# cd ../..
[root@VM-0-12-centos local]# mkdir redis-cluster
[root@VM-0-12-centos local]# cd redis-cluster/
[root@VM-0-12-centos redis-cluster]# cp -r /usr/local/redis/bin/ ./redis01/
[root@VM-0-12-centos local]# rm -rf ./redis01/dump.rdb # 删除redis01目录下的快照文件dump.rdb
[root@VM-0-12-centos redis-cluster]# vim ./redis01/redis.conf
#bind 127.0.0.1 注释，允许其他ip访问
protected-mode no
port 7001
masterauth 1234
requirepass 1234 #最好设置一下密码
cluster-enabled yes
cluster-enabled yes 打开集群模式
cluster-config-file nodes-7001.conf 设定节点配置文件名
cluster-node-timeout 15000
appendonlyyes
[root@VM-0-12-centos redis-cluster]# cp -r ./redis01/ ./redis02/
[root@VM-0-12-centos redis-cluster]# cp -r ./redis01/ ./redis03/
[root@VM-0-12-centos redis-cluster]# cp -r ./redis01/ ./redis04/
[root@VM-0-12-centos redis-cluster]# cp -r ./redis01/ ./redis05/
[root@VM-0-12-centos redis-cluster]# cp -r ./redis01/ ./redis06/
[root@VM-0-12-centos redis-cluster]# vim ./redis01/redis.conf # 逐个修改端口号
[root@VM-0-12-centos redis-cluster]# vim start-all.sh # 创建启动文件，也可手动启动所有服务
cd redis01
./redis-server ./redis.conf
cd ..
cd redis02
./redis-server ./redis.conf
cd ..
cd redis03
./redis-server ./redis.conf
cd ..
cd redis04
./redis-server ./redis.conf
cd ..
cd redis05
./redis-server ./redis.conf
cd ..
cd redis06
./redis-server ./redis.conf
cd ..
[root@VM-0-12-centos redis-cluster]# vim s-all.sh # 创建关闭文件，也可手动停止所有服务
cd redis01
./redis-cli -h 127.0.0.1 -p 7001 -a 1234 shutdon
cd ..
cd redis02
./redis-cli -h 127.0.0.1 -p 7002 -a 1234 shutdon
cd ..
cd redis03
./redis-cli -h 127.0.0.1 -p 7003 -a 1234 shutdon
cd ..
cd redis04
./redis-cli -h 127.0.0.1 -p 7004 -a 1234 shutdon
cd ..
cd redis05
./redis-cli -h 127.0.0.1 -p 7005 -a 1234 shutdon
cd ..
cd redis06
./redis-cli -h 127.0.0.1 -p 7006 -a 1234 shutdon
cd ..
[root@VM-0-12-centos redis-cluster]# chmod +x start-all.sh
[root@VM-0-12-centos redis-cluster]# chmod +x s-all.sh
[root@VM-0-12-centos redis-cluster]# ./start-all.sh
[root@VM-0-12-centos redis-cluster]# ps aux | grep redis
root  17885  0.1  0.1 163152  3724 ?  Ssl  17:52   0:01 ./redis-server :6379
root  23399  0.0  0.1 165712  3664 ?  Ssl  18:20   0:00 ./redis-server :7001 [cluster]
root  23401  0.0  0.1 165712  3660 ?  Ssl  18:20   0:00 ./redis-server :7002 [cluster]
root  23403  0.0  0.1 165712  3664 ?  Ssl  18:20   0:00 ./redis-server :7003 [cluster]
root  23405  0.0  0.1 165712  3664 ?  Ssl  18:20   0:00 ./redis-server :7004 [cluster]
root  23411  0.0  0.1 171856  3664 ?  Ssl  18:20   0:00 ./redis-server :7005 [cluster]
root  23413  0.0  0.1 165712  3672 ?  Ssl  18:20   0:00 ./redis-server :7006 [cluster]
root  23495  0.0  0.0 112816   976 pts/0 R+   18:21   0:00 grep --color=auto redis
[root@VM-0-12-centos redis-cluster]# ls
redis01  redis02  redis03  redis04  redis05  redis06  start-all.sh  s-all.sh
[root@VM-0-12-centos redis-cluster]# /usr/local/redis/bin/redis-cli --cluster create --cluster-replicas 1 182.254.134.174:7001 182.254.134.174:7002 182.254.134.174:7003 182.254.134.174:7004 182.254.134.174:7005 182.254.134.174:7006 -a 1234
Warning: Using a passord ith '-a' or '-u' option on the mand line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 182.254.134.174:7005 to 182.254.134.174:7001
Adding replica 182.254.134.174:7006 to 182.254.134.174:7002
Adding replica 182.254.134.174:7004 to 182.254.134.174:7003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: e9750b2aa5a23b0d5e79e98b3a61c6f97a69f01d 182.254.134.174:7001
   slots:[0-5460] (5461 slots) master
M: 1194702cf2b7ef4f5d5e7854f526aa60d47f2577 182.254.134.174:7002
   slots:[5461-10922] (5462 slots) master
M: b120e1a401f11a06f8d81f19df2607c71c5cf9df 182.254.134.174:7003
   slots:[10923-16383] (5461 slots) master
S: 28362fb68d33eb50322f514daf4e7a447e3eb3 182.254.134.174:7004
   replicates 1194702cf2b7ef4f5d5e7854f526aa60d47f2577
S: 52b90d0718c06990de002dbe00be512c0f4ed1 182.254.134.174:7005
   replicates b120e1a401f11a06f8d81f19df2607c71c5cf9df
S: 2aedbda7dadc48e545ec0870b7d3585bb4637c47 182.254.134.174:7006
   replicates e9750b2aa5a23b0d5e79e98b3a61c6f97a69f01d
Can I set the above configuration? (type 'yes' to aept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..
>>> Performing Cluster Check (using node 182.254.134.174:7001)
M: e9750b2aa5a23b0d5e79e98b3a61c6f97a69f01d 182.254.134.174:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 1194702cf2b7ef4f5d5e7854f526aa60d47f2577 182.254.134.174:7002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 2aedbda7dadc48e545ec0870b7d3585bb4637c47 182.254.134.174:7006
   slots: (0 slots) slave
   replicates e9750b2aa5a23b0d5e79e98b3a61c6f97a69f01d
M: b120e1a401f11a06f8d81f19df2607c71c5cf9df 182.254.134.174:7003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 28362fb68d33eb50322f514daf4e7a447e3eb3 182.254.134.174:7004
   slots: (0 slots) slave
   replicates 1194702cf2b7ef4f5d5e7854f526aa60d47f2577
S: 52b90d0718c06990de002dbe00be512c0f4ed1 182.254.134.174:7005
   slots: (0 slots) slave
   replicates b120e1a401f11a06f8d81f19df2607c71c5cf9df
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@VM-0-12-centos redis-cluster]#

模拟主机宕机情况，看看是否自动 slave自动升为master

[root@VM-0-12-centos redis-cluster]# ./redis01/redis-cli -p 7001 -a 1234
Warning: Using a passord ith '-a' or '-u' option on the mand line interface may not be safe.
127.0.0.1:7001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_knon_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:886
cluster_stats_messages_pong_sent:887
cluster_stats_messages_sent:1773
cluster_stats_messages_ping_received:882
cluster_stats_messages_pong_received:886
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1773
total_cluster_links_buffer_limit_exceeded:0
127.0.0.1:7001> cluster nodes
78b3bb404c37850c1cb7409b6d1fd1d994ee2582 182.254.134.174:7003@17003 master - 0 1660036015000 3 connected 10923-16383
0238999b8de448562d28eb23f7be491f0603afe5 182.254.134.174:7005@17005 slave 78b3bb404c37850c1cb7409b6d1fd1d994ee2582 0 1660036017989 3 connected
079b10ae2ff6e0870250beb91be5f0a8d5fda2 182.254.134.174:7002@17002 master - 0 1660036016986 2 connected 5461-10922
232fb95877a5f2b3093b1f7d94b632c10f55c976 172.17.0.12:7001@17001 myself,master - 0 1660036015000 1 connected 0-5460
532e2e3322dc71f281349d18d562a2a00d201b5c 182.254.134.174:7006@17006 slave 232fb95877a5f2b3093b1f7d94b632c10f55c976 0 1660036016000 1 connected
2e939c53fbc695e136bd0e63a07d6f86f9e25cb3 182.254.134.174:7004@17004 slave 079b10ae2ff6e0870250beb91be5f0a8d5fda2 0 1660036015982 2 connected

重新开一个ssh远程连接,连接7002服务，关闭redis，等待15秒

[root@VM-0-12-centos redis-cluster]# ./redis02/redis-cli -p 7002 -a 1234
Warning: Using a passord ith '-a' or '-u' option on the mand line interface may not be safe.
127.0.0.1:7002> shutdon

查看节点信息，会发现7004已经已经自动升为master

127.0.0.1:7001> cluster nodes
78b3bb404c37850c1cb7409b6d1fd1d994ee2582 182.254.134.174:7003@17003 master - 0 1660036206861 3 connected 10923-16383
0238999b8de448562d28eb23f7be491f0603afe5 182.254.134.174:7005@17005 slave 78b3bb404c37850c1cb7409b6d1fd1d994ee2582 0 1660036204840 3 connected
079b10ae2ff6e0870250beb91be5f0a8d5fda2 182.254.134.174:7002@17002 master,fail - 1660036178557 1660036172498 2 disconnected
232fb95877a5f2b3093b1f7d94b632c10f55c976 172.17.0.12:7001@17001 myself,master - 0 1660036203000 1 connected 0-5460
532e2e3322dc71f281349d18d562a2a00d201b5c 182.254.134.174:7006@17006 slave 232fb95877a5f2b3093b1f7d94b632c10f55c976 0 1660036207000 1 connected
2e939c53fbc695e136bd0e63a07d6f86f9e25cb3 182.254.134.174:7004@17004 master - 0 1660036207872 7 connected 5461-10922
127.0.0.1:7001>

---

操作中遇到的问题 1、[ERR] Node 182.254.134.174:7001 is not empty. Either the node already knos other nodes (check ith CLUSTER NODES) or contains some key in database 0.

[root@VM-0-12-centos redis-cluster]# /usr/local/redis/bin/redis-cli --cluster create --cluster-replicas 1 182.254.134.174:7001 182.254.134.174:7002 182.254.134.174:7003 182.254.134.174:7004 182.254.134.174:7005 182.254.134.174:7006 -a 1234
Warning: Using a passord ith '-a' or '-u' option on the mand line interface may not be safe.
[ERR] Node 182.254.134.174:7001 is not empty. Either the node already knos other nodes (check ith CLUSTER NODES) or contains some key in database 0.

原因上面的命令只能在新创健集群的时候执行一次，目的是为了建立内部各个节点的对应关系，比如主从关系，这些关系仅且只能在一个集群中初始化时对应一次；
解决方法删除aof，rdb，nodes.conf文件重新启动，我写了一个清除文件脚本 clear.sh

cd ..
cd redis02
rm -f nodes-7002.conf
rm -f nodes.conf
rm -f dump.rdb
rm -f appendonly.aof
cd ..
cd redis03
rm -f nodes-7003.conf
rm -f nodes.conf
rm -f dump.rdb
rm -f appendonly.aof
cd ..
cd redis04
rm -f nodes-7004.conf
rm -f nodes.conf
rm -f dump.rdb
rm -f appendonly.aof
cd ..
cd redis05
rm -f nodes-7005.conf
rm -f nodes.conf
rm -f dump.rdb
rm -f appendonly.aof
cd ..
cd redis06
rm -f nodes-7006.conf
rm -f nodes.conf
rm -f dump.rdb
rm -f appendonly.aof
cd ..

[root@VM-0-12-centos redis-cluster]# ./clear.sh
[root@VM-0-12-centos redis-cluster]# ./start-all.sh
/usr/local/redis/bin/redis-cli --cluster create --cluster-replicas 1 182.254.134.174:7001 182.254.134.174:7002 182.254.134.174:7003 182.254.134.174:7004 182.254.134.174:7005 182.254.134.174:7006 -a 1234
正常启动，重复代码就不粘贴了...

2. 0@0 master,fail,noaddr - 1660036178557 1660036172498 2 disconnected

7004自动升为master后重新启动7002查看节点状态后发现，发现7002已脱离集群，且id都已发生了变化

127.0.0.1:7001> cluster nodes
78b3bb404c37850c1cb7409b6d1fd1d994ee2582 182.254.134.174:7003@17003 master - 0 1660039460000 3 connected 10923-16383
0238999b8de448562d28eb23f7be491f0603afe5 182.254.134.174:7005@17005 slave 78b3bb404c37850c1cb7409b6d1fd1d994ee2582 0 1660039461528 3 connected
079b10ae2ff6e0870250beb91be5f0a8d5fda2 :0@0 master,fail,noaddr - 1660036178557 1660036172498 2 disconnected
232fb95877a5f2b3093b1f7d94b632c10f55c976 172.17.0.12:7001@17001 myself,master - 0 1660039460000 1 connected 0-5460
532e2e3322dc71f281349d18d562a2a00d201b5c 182.254.134.174:7006@17006 slave 232fb95877a5f2b3093b1f7d94b632c10f55c976 0 1660039460000 1 connected
2e939c53fbc695e136bd0e63a07d6f86f9e25cb3 182.254.134.174:7004@17004 master - 0 1660039460525 7 connected 5461-10922

原因7002已脱离集群；
解决方法将该从节点剔出集群，重新加入
任意节点上执行 cluster fet命令

127.0.0.1:7001> cluster fet 079b10ae2ff6e0870250beb91be5f0a8d5fda2 # 踢出集群
OK
127.0.0.1:7001> cluster meet 182.254.134.174 7002 # 握手加入集群
OK
127.0.0.1:7001> cluster nodes
37d2363fbc60a64a3cb5be0cb75c4b418795b02e 182.254.134.174:7002@17002 master - 0 1660040192000 0 connected
78b3bb404c37850c1cb7409b6d1fd1d994ee2582 182.254.134.174:7003@17003 master - 0 1660040195000 3 connected 10923-16383
0238999b8de448562d28eb23f7be491f0603afe5 182.254.134.174:7005@17005 slave 78b3bb404c37850c1cb7409b6d1fd1d994ee2582 0 1660040193966 3 connected
232fb95877a5f2b3093b1f7d94b632c10f55c976 172.17.0.12:7001@17001 myself,master - 0 1660040195000 1 connected 0-5460
532e2e3322dc71f281349d18d562a2a00d201b5c 182.254.134.174:7006@17006 slave 232fb95877a5f2b3093b1f7d94b632c10f55c976 0 1660040196975 1 connected
2e939c53fbc695e136bd0e63a07d6f86f9e25cb3 182.254.134.174:7004@17004 master - 0 1660040195972 7 connected 5461-10922

然再在7002上和7004重新配置主从关系,

127.0.0.1:7002> cluster replicate 2e939c53fbc695e136bd0e63a07d6f86f9e25cb3
127.0.0.1:7002> cluster nodes
232fb95877a5f2b3093b1f7d94b632c10f55c976 182.254.134.174:7001@17001 master - 0 1660040830000 1 connected 0-5460
78b3bb404c37850c1cb7409b6d1fd1d994ee2582 182.254.134.174:7003@17003 master - 0 1660040828781 3 connected 10923-16383
37d2363fbc60a64a3cb5be0cb75c4b418795b02e 172.17.0.12:7002@17002 myself,slave 2e939c53fbc695e136bd0e63a07d6f86f9e25cb3 0 1660040827000 7 connected
0238999b8de448562d28eb23f7be491f0603afe5 182.254.134.174:7005@17005 slave 78b3bb404c37850c1cb7409b6d1fd1d994ee2582 0 1660040829000 3 connected
2e939c53fbc695e136bd0e63a07d6f86f9e25cb3 182.254.134.174:7004@17004 master - 0 1660040829785 7 connected 5461-10922
532e2e3322dc71f281349d18d562a2a00d201b5c 182.254.134.174:7006@17006 slave 232fb95877a5f2b3093b1f7d94b632c10f55c976 0 1660040830789 1 connected

---