2
0
0
0
专栏/.../

Region 创建、分裂及合并的原理

 张鱼小丸子-PingCAP  发表于  2020-04-14
原创

Region 创建、分裂、合并

TiDB 在 2.1 以及后续版本中引入了 Raft PreVote、Raft Learner、Raft Region Merge 等特性，接下来我们了解下这些特性作用与使用场景

TiDB 2.1 新特性使用场景

下文中部分引用 The Way to TiDB 3.0 and Beyond

Raft PreVote

2.1 版本已经支持 PreVote 功能，并默认打开。

Raft Prevote

考虑一种意外情况，就是在 Raft group 中出现了网络隔离，有 1 个节点和另外 2 个节点隔离掉了，然后它现在发现「我找不到 Leader 了，Leader 可能已经挂掉了」，然后就开始投票，不断投票，但是因为它和其他节点是隔离开的，所以没有办法选举成功。

它每次失败，都会把自己的 term 加 1，每次失败加 1，网络隔离发生一段时间之后，它的 term 就会很高。当网络分区恢复之后，它的选举消息就能发出去了，并且这个选举消息里面的 term 是比较高的。

根据 Raft 的协议，当遇到一个 term 比较高的时候，可能就会同意发起选举，当前的 Leader 就会下台来参与选举。 但是因为发生网络隔离这段时间他是没有办法同步数据的，此时它的 Raft Log 一定是落后的，所以即使它的 term 很高，也不可能被选成新的 Leader。

所以这个时候经过一次选举之后，它不会成为新 Leader，只有另外两个有机会成为新的 Leader。这个选举是对整个 Raft Group 造成了危害：首先它不可能成为新的 Leader，第二它把原有的 Leader 赶下台了，并且在这个选举过程中是没有 Leader 的，这时的 Raft Group 是不能对外提供服务的。虽然这个时间会很短，但如果该 TiKV 上 Region 数据较多时可能会造成比较大的抖动。

所以有了 PreVote 这个特性。具体是这样做的：在进行选举之前，先用 PreVote 这套机制来进行预选举，每个成员把自己的信息，包括 term，Raft Log Index 放进去，发给其它成员，其它成员有这个信息之后，认为「我可以选你为 Leader」，才会发起真正的选举。

有了 PreVote 之后，我们就可以避免这种大规模的一个节点上很多数据、很多 Raft Group、很多 Peer 的情况下突然出现网络分区，在恢复之后造成大量的 Region 出现选举，导致整个服务有抖动。 因此 PreVote 能极大的提升稳定性。

Raft Learner

2.1 版本已经支持 Learner 功能，并默认打开。

该功能主要优化 balance region、region offline 两个场景，当数据副本迁移或者下线时，出现网络隔离；此时会有较大的几率造成该 region 不可用。

Raft Learner

Raft 会有 Leader 和 Follower 这两个概念，Leader 来负责读写，Follower 来作为 Backup，然后随时找机会成为新的 Leader。如果你想加一个新的节点，比如说在扩容或者故障恢复，新加了一个 Follower 进来，这个时候 Raft Group 有 4 个成员， Leader、Follower 都是 Voter，都能够在写入数据时候对日志进行投票，或者是要在成员变更的时候投票的。 这时一旦发生意外情况，比如网络变更或者出现网络分区，假设 2 个被隔离掉的节点都在一个物理位置上，就会导致 4 个 Voter 中 2 个不可用，那这时这个 Raft Group 就不可用了。

有了 Learner 之后，我们在扩容时不会先去加一个 Follower（也就是一个 Voter），而是增加一个 Learner 的角色，它不是 Voter，所以它只会同步数据不会投票，所以无论在做数据写入还是成员变更的时候都不会算上它。 当同步完所有数据时（因为数据量大的时候同步时间会比较长），拿到所有数据之后，再把它变成一个 Voter，同时再把另一个我们想下线的 Follower 下掉就好了。 这样就能极大的缩短同时存在 4 个 Voter 的时间，整个 Raft Group 的可用性就得到了提升。

Region Merge 场景

2.1 版本已经支持 Region Merge 功能，未默认打开。

数据库的主要四个操作 增删修改 ，当删除数据较多时会出现较多文件或磁盘碎片( 单机数据库 )，通常会使用运维工具对文件碎片重新排序修复。分布式数据库也有类似现象，TiDB 是基于逻辑 Region 单元存储(可以想象为磁盘 inode 单元)，如果我们通过 delete 、drop、Truncate 等操作删除数据时，会在 Region 内删除一段数据或者整个数据空间，但是 Region 本身是不会被删除的，这样会加大 TiKV Raft stone 线程压力，同时在 PD balance region 时也会造成一些压力。

久而久之，持续运行一段时间后，Region 数量不断增加，为缓解 Region Group 心跳压力以及降低 PD balance region 压力，将低于一定空间大小的 Region merge 合并在一起。这样可以降低大部分压力，提升集群 Region 可用性以及稳定性。 在处理 Region Merge 时，假设低于 20MB ( Region 默认大小 96MB) 的 Region 将触发 Region Merge ；假设 A Region 19MB 、 B Region 18MB、C Region 40MB 分别在三个 TiKV 上；根据前面的规则，PD 会将 A 和 B 的 Leader 调度到同一个 KV 节点，然后触发 Region Merge ，将 A B 合并成一个新 Region。 目前版本中 Region Merge 会占用 PD 中的 Region balance schedule limit ，且需要先将相邻的两个 Region Leader 调度到相同的 KV 节点才可以工作。目前 Region Merge 只能基于 Region 两两合并，无法一步将 5 个低于 20MB 的 Region 合并成一个接近 96MB 的 Region。

部署架构

组件 IP 备注
TiKV 10.0.1.4:20171 ID: 1 / post1
TiKV 10.0.1.4:20172 ID: 2 / post2
TiKV 10.0.1.4:20173 ID: 7 / post3

日志分析

Prevote 场景

由于实际中 Prevote 最终成功可用场景难以复现，下面通过相近方案展现

  • Pre-Vote 算法的引入，网络隔离节点由于无法获得大部分节点的许可，因此无法增加 term，重新加入集群时不会导致重新选主。

  • 在 Pre-Vote 算法中，Candidate 首先要确认自己能赢得集群中大多数节点的投票，才会把自己的 term 增加，然后发起真正的投票，其他节点同意发起重新选举的条件更严格，必须同时满足 ：

  • 以下是某 tikv 创建 region 25 的过程 (实际 region 25 是由其他 region 数据满足一定条件后 split，本文暂不扩展)

    • 起初会随机选择某 KV 创建一个新的 Region，指定该 Region 数据范围、 Region ID、Region Peer ID、Store ID 等信息
    • 该 Region group 内仅有一个成员，自己向自己投票( Peer 投票时会携带自身信息 / 第四行 )即可，自己最后会被选举为 Leader // 间接等同满足了 Prevote 的两个条件
[14:46:52.781 +08:00] [INFO] [peer.rs:1570] ["insert new region"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 1 version: 10 } peers { id: 26 store_id: 1 }"] [region_id=25]
[14:46:52.781 +08:00] [INFO] [peer.rs:142] ["create peer"] [peer_id=26] [region_id=25]
[14:46:52.781 +08:00] [INFO] [raft.rs:723] ["[region 25] 26 became follower at term 5"]
[14:46:52.781 +08:00] [INFO] [raft.rs:295] ["[region 25] 26 newRaft [peers: [26], term: 5, commit: 5, applied: 5, last_index: 5, last_term: 5]"]
[14:46:52.781 +08:00] [INFO] [raft.rs:1094] ["[region 25] 26 is starting a new election at term 5"]
[14:46:52.781 +08:00] [INFO] [raft.rs:765] ["[region 25] 26 became pre-candidate at term 5"]
[14:46:52.781 +08:00] [INFO] [raft.rs:858] ["[region 25] 26 received MsgRequestPreVoteResponse from 26 at term 5"]
[14:46:52.781 +08:00] [INFO] [raft.rs:743] ["[region 25] 26 became candidate at term 6"]
[14:46:52.781 +08:00] [INFO] [raft.rs:858] ["[region 25] 26 received MsgRequestVoteResponse from 26 at term 6"]
[14:46:52.781 +08:00] [INFO] [raft.rs:793] ["[region 25] 26 became leader at term 6"]

Raft Learner 场景

  • 更多信息可阅读 Raft Learner – Raft - Membership Change
  • Raft Group 需要补充多个成员时，Learner 是一个个添加，并不是单次添加多个
  • add Learner --> Leader --> Follower --> delete Follower / 四个状态之间非原子性操作，但 Leader 后续步骤间隔时间会足够的短，相比之前版本会提升高可用性

TiKV 1 添加 Store 2 上的副本

[14:47:23.631 +08:00] [INFO] [pd.rs:551] ["try to change peer"] [peer="id: 74 store_id: 2 is_learner: true"] [change_type=AddLearnerNode] [region_id=25]
[14:47:23.631 +08:00] [INFO] [peer.rs:2076] ["propose conf change peer"] [change_peer=74] [change_type=AddLearnerNode] [peer_id=26] [region_id=25]
[14:47:23.639 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { change_type: AddLearnerNode peer { id: 74 store_id: 2 is_learner: true } }"] [index=7] [term=6] [peer_id=26] [region_id=25]
[14:47:23.639 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 1 version: 10"] [type=AddLearner] [peer_id=26] [region_id=25]
[14:47:23.639 +08:00] [INFO] [apply.rs:1508] ["add learner successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 1 version: 10 } peers { id: 26 store_id: 1 }"] [peer="id: 74 store_id: 2 is_learner: true"] [peer_id=26] [region_id=25]
[14:47:23.646 +08:00] [INFO] [peer.rs:1468] ["notify pd with change peer region"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 2 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 is_learner: true }"] [peer_id=26] [region_id=25]
[14:47:24.835 +08:00] [INFO] [peer_storage.rs:788] ["requesting snapshot"] [peer_id=26] [region_id=25]
[14:47:24.836 +08:00] [INFO] [snap.rs:674] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=default] [snapshot=/data/post1/deploy/data/snap/gen_25_6_7_(default|lock|write).sst] [region_id=25]
[14:47:24.836 +08:00] [INFO] [snap.rs:674] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data/post1/deploy/data/snap/gen_25_6_7_(default|lock|write).sst] [region_id=25]
[14:47:24.837 +08:00] [INFO] [snap.rs:674] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data/post1/deploy/data/snap/gen_25_6_7_(default|lock|write).sst] [region_id=25]
[14:47:24.845 +08:00] [INFO] [snap.rs:722] ["scan snapshot"] [takes=9.31693ms] [size=0] [key_count=0] [snapshot=/data/post1/deploy/data/snap/gen_25_6_7_(default|lock|write).sst] [region_id=25]
[14:47:26.858 +08:00] [INFO] [snap.rs:377] ["sent snapshot"] [duration=21.123849ms] [size=0] [snap_key=25_6_7] [region_id=25]
[14:47:26.858 +08:00] [INFO] [peer.rs:595] ["report snapshot status"] [status=Finish] [to="id: 74 store_id: 2 is_learner: true"] [peer_id=26] [region_id=25]

[14:47:27.839 +08:00] [INFO] [pd.rs:551] ["try to change peer"] [peer="id: 74 store_id: 2"] [change_type=AddNode] [region_id=25]
[14:47:27.839 +08:00] [INFO] [peer.rs:2076] ["propose conf change peer"] [change_peer=74] [change_type=AddNode] [peer_id=26] [region_id=25]
[14:47:27.847 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { peer { id: 74 store_id: 2 } }"] [index=8] [term=6] [peer_id=26] [region_id=25]
[14:47:27.848 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 2 version: 10"] [type=AddNode] [peer_id=26] [region_id=25]
[14:47:27.848 +08:00] [INFO] [apply.rs:1423] ["add peer successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 2 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 is_learner: true }"] [peer="id: 74 store_id: 2"] [peer_id=26] [region_id=25]
[14:47:27.856 +08:00] [INFO] [peer.rs:1468] ["notify pd with change peer region"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 3 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 }"] [peer_id=26] [region_id=25]

TiKV 2 副本 peer 74

  • TiKV 2 是按照 Raft Learner 方式将 peer 74 添加到 Raft group 中

    • 以下是日志记录
[2019/06/12 14:47:24.834 +08:00] [INFO] [peer.rs:175] ["replicate peer"] [peer_id=74] [region_id=25]
[2019/06/12 14:47:24.834 +08:00] [INFO] [raft.rs:723] ["[region 25] 74 became follower at term 0"]
[2019/06/12 14:47:24.834 +08:00] [INFO] [raft.rs:295] ["[region 25] 74 newRaft [peers: [], term: 0, commit: 0, applied: 0, last_index: 0, last_term: 0]"]
[2019/06/12 14:47:24.834 +08:00] [INFO] [raft.rs:723] ["[region 25] 74 became follower at term 1"]
[2019/06/12 14:47:24.834 +08:00] [INFO] [raft.rs:924] ["[region 25] 74 [term: 1] received a MsgHeartbeat message with higher term from 26 [term: 6]"]
[2019/06/12 14:47:24.834 +08:00] [INFO] [raft.rs:723] ["[region 25] 74 became follower at term 6"]
[2019/06/12 14:47:26.838 +08:00] [INFO] [snap.rs:223] ["saving snapshot file"] [file=/data/post2/deploy/data/snap/rev_25_6_7_(default|lock|write).sst] [snap_key=25_6_7]
[2019/06/12 14:47:26.858 +08:00] [INFO] [raft.rs:1829] ["[region 25] 74 [commit: 0, lastindex: 0, lastterm: 0] starts to restore snapshot [index: 7, term: 6]"]
[2019/06/12 14:47:26.858 +08:00] [INFO] [raft.rs:1853] ["[region 25] 74 restored progress of 26 [Some(Progress { matched: 0, next_idx: 1, state: Probe, paused: false, pending_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count:
 0, buffer: [] }, is_learner: false })]"]
[2019/06/12 14:47:26.858 +08:00] [INFO] [raft.rs:1853] ["[region 25] 74 restored progress of 74 [Some(Progress { matched: 0, next_idx: 1, state: Probe, paused: false, pending_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count:
 0, buffer: [] }, is_learner: true })]"]
[2019/06/12 14:47:26.858 +08:00] [INFO] [raft_log.rs:474] ["[region 25] 74 log [committed=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 7, term: 6]"]
[2019/06/12 14:47:26.858 +08:00] [INFO] [raft.rs:1773] ["[region 25] 74 [commit: 7] restored snapshot [index: 7, term: 6]"]
[2019/06/12 14:47:26.858 +08:00] [INFO] [peer_storage.rs:901] ["begin to apply snapshot"] [peer_id=74] [region_id=25]
[2019/06/12 14:47:26.858 +08:00] [INFO] [peer_storage.rs:941] ["apply snapshot with state ok"] [state="applied_index: 7 truncated_state { index: 7 term: 6 }"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"00
0\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 2 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 is_learner: 
true }"] [peer_id=74] [region_id=25]
[2019/06/12 14:47:26.873 +08:00] [INFO] [peer.rs:2054] ["snapshot is applied"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"00
0\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 2 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 is_learner: true }"] [peer_id=74] [region_id=25]
[2019/06/12 14:47:26.873 +08:00] [INFO] [region.rs:272] ["begin apply snap data"] [region_id=25]
[2019/06/12 14:47:26.874 +08:00] [INFO] [region.rs:341] ["apply new data"] [time_takes=45.202µs] [region_id=25]
[2019/06/12 14:47:27.848 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { peer { id: 74 store_id: 2 } }"] [index=8] [term=6] [peer_id=74] [region_id=25]
[2019/06/12 14:47:27.848 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 2 version: 10"] [type=AddNode] [peer_id=74] [region_id=25]
[2019/06/12 14:47:27.848 +08:00] [INFO] [apply.rs:1423] ["add peer successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000
�00\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 2 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 is_learner: true }"] [peer="id: 74 store_id: 2"] [peer_id=74] [region_id=25]

[2019/06/12 14:47:32.462 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { change_type: AddLearnerNode peer { id: 84 store_id: 7 is_learner: true } }"] [index=9] [term=6] [peer_id=74] [region_id=25]
[2019/06/12 14:47:32.462 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 3 version: 10"] [type=AddLearner] [peer_id=74] [region_id=25]
[2019/06/12 14:47:32.462 +08:00] [INFO] [apply.rs:1508] ["add learner successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"0
00\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 3 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 }"] [peer="id: 84 store_id: 7 is_learner: true"] [peer_id=74] [region_id=25]


[2019/06/12 14:47:35.863 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { peer { id: 84 store_id: 7 } }"] [index=10] [term=6] [peer_id=74] [region_id=25]
[2019/06/12 14:47:35.864 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 4 version: 10"] [type=AddNode] [peer_id=74] [region_id=25]
[2019/06/12 14:47:35.864 +08:00] [INFO] [apply.rs:1423] ["add peer successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000
�00\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 4 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 } peers { id: 84 store_id: 7 is_learner: true }"] [peer="id: 84 store_id: 7"] [peer_id=74
] [region_id=25]

[2019/06/12 14:47:42.696 +08:00] [INFO] [peer.rs:520] ["deleting applied snap file"] [snap_file=25_6_7] [peer_id=74] [region_id=25]

TiKV 1 添加 Store 7 上的副本

  • TiKV 1 Region 25 继续向 TiKV 7 添加 peer
[14:47:32.446 +08:00] [INFO] [pd.rs:551] ["try to change peer"] [peer="id: 84 store_id: 7 is_learner: true"] [change_type=AddLearnerNode] [region_id=25]
[14:47:32.446 +08:00] [INFO] [peer.rs:2076] ["propose conf change peer"] [change_peer=84] [change_type=AddLearnerNode] [peer_id=26] [region_id=25]
[14:47:32.462 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { change_type: AddLearnerNode peer { id: 84 store_id: 7 is_learner: true } }"] [index=9] [term=6] [peer_id=26] [region_id=25]
[14:47:32.462 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 3 version: 10"] [type=AddLearner] [peer_id=26] [region_id=25]
[14:47:32.462 +08:00] [INFO] [apply.rs:1508] ["add learner successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 3 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 }"] [peer="id: 84 store_id: 7 is_learner: true"] [peer_id=26] [region_id=25]
[14:47:32.470 +08:00] [INFO] [peer.rs:1468] ["notify pd with change peer region"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 4 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 } peers { id: 84 store_id: 7 is_learner: true }"] [peer_id=26] [region_id=25]
[14:47:32.844 +08:00] [INFO] [peer_storage.rs:788] ["requesting snapshot"] [peer_id=26] [region_id=25]
[14:47:32.844 +08:00] [INFO] [snap.rs:674] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=default] [snapshot=/data/post1/deploy/data/snap/gen_25_6_9_(default|lock|write).sst] [region_id=25]
[14:47:32.845 +08:00] [INFO] [snap.rs:674] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=lock] [snapshot=/data/post1/deploy/data/snap/gen_25_6_9_(default|lock|write).sst] [region_id=25]
[14:47:32.845 +08:00] [INFO] [snap.rs:674] ["scan snapshot of one cf"] [size=0] [key_count=0] [cf=write] [snapshot=/data/post1/deploy/data/snap/gen_25_6_9_(default|lock|write).sst] [region_id=25]
[14:47:32.854 +08:00] [INFO] [snap.rs:722] ["scan snapshot"] [takes=9.971653ms] [size=0] [key_count=0] [snapshot=/data/post1/deploy/data/snap/gen_25_6_9_(default|lock|write).sst] [region_id=25]
[14:47:34.868 +08:00] [INFO] [snap.rs:377] ["sent snapshot"] [duration=22.964814ms] [size=0] [snap_key=25_6_9] [region_id=25]
[14:47:34.868 +08:00] [INFO] [peer.rs:595] ["report snapshot status"] [status=Finish] [to="id: 84 store_id: 7 is_learner: true"] [peer_id=26] [region_id=25]

[14:47:35.846 +08:00] [INFO] [pd.rs:551] ["try to change peer"] [peer="id: 84 store_id: 7"] [change_type=AddNode] [region_id=25]
[14:47:35.847 +08:00] [INFO] [peer.rs:2076] ["propose conf change peer"] [change_peer=84] [change_type=AddNode] [peer_id=26] [region_id=25]
[14:47:35.863 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { peer { id: 84 store_id: 7 } }"] [index=10] [term=6] [peer_id=26] [region_id=25]
[14:47:35.863 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 4 version: 10"] [type=AddNode] [peer_id=26] [region_id=25]
[14:47:35.863 +08:00] [INFO] [apply.rs:1423] ["add peer successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 4 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 } peers { id: 84 store_id: 7 is_learner: true }"] [peer="id: 84 store_id: 7"] [peer_id=26] [region_id=25]
[14:47:35.872 +08:00] [INFO] [peer.rs:1468] ["notify pd with change peer region"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 5 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 } peers { id: 84 store_id: 7 }"] [peer_id=26] [region_id=25]

TiKV 7 副本 peer 84

  • TiKV 7 是按照 Raft Learner 方式将 peer 84 添加到 Raft group 中

    • 以下是日志记录
[14:47:32.842 +08:00] [INFO] [peer.rs:175] ["replicate peer"] [peer_id=84] [region_id=25]
[14:47:32.843 +08:00] [INFO] [raft.rs:723] ["[region 25] 84 became follower at term 0"]
[14:47:32.843 +08:00] [INFO] [raft.rs:295] ["[region 25] 84 newRaft [peers: [], term: 0, commit: 0, applied: 0, last_index: 0, last_term: 0]"]
[14:47:32.843 +08:00] [INFO] [raft.rs:723] ["[region 25] 84 became follower at term 1"]
[14:47:32.843 +08:00] [INFO] [raft.rs:924] ["[region 25] 84 [term: 1] received a MsgHeartbeat message with higher term from 26 [term: 6]"]
[14:47:32.843 +08:00] [INFO] [raft.rs:723] ["[region 25] 84 became follower at term 6"]
[14:47:34.847 +08:00] [INFO] [snap.rs:223] ["saving snapshot file"] [file=/data/post3/deploy/data/snap/rev_25_6_9_(default|lock|write).sst] [snap_key=25_6_9]
[14:47:34.868 +08:00] [INFO] [raft.rs:1829] ["[region 25] 84 [commit: 0, lastindex: 0, lastterm: 0] starts to restore snapshot [index: 9, term: 6]"]
[14:47:34.868 +08:00] [INFO] [raft.rs:1853] ["[region 25] 84 restored progress of 26 [Some(Progress { matched: 0, next_idx: 1, state: Probe, paused: false, pending_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count: 0, buffer: [] }, is_learner: false })]"]
[14:47:34.868 +08:00] [INFO] [raft.rs:1853] ["[region 25] 84 restored progress of 74 [Some(Progress { matched: 0, next_idx: 1, state: Probe, paused: false, pending_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count: 0, buffer: [] }, is_learner: false })]"]
[14:47:34.868 +08:00] [INFO] [raft.rs:1853] ["[region 25] 84 restored progress of 84 [Some(Progress { matched: 0, next_idx: 1, state: Probe, paused: false, pending_snapshot: 0, recent_active: false, ins: Inflights { start: 0, count: 0, buffer: [] }, is_learner: true })]"]
[14:47:34.868 +08:00] [INFO] [raft_log.rs:474] ["[region 25] 84 log [committed=0, applied=0, unstable.offset=1, unstable.entries.len()=0] starts to restore snapshot [index: 9, term: 6]"]
[14:47:34.868 +08:00] [INFO] [raft.rs:1773] ["[region 25] 84 [commit: 9] restored snapshot [index: 9, term: 6]"]
[14:47:34.868 +08:00] [INFO] [peer_storage.rs:901] ["begin to apply snapshot"] [peer_id=84] [region_id=25]
[14:47:34.868 +08:00] [INFO] [peer_storage.rs:941] ["apply snapshot with state ok"] [state="applied_index: 9 truncated_state { index: 9 term: 6 }"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 4 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 } peers { id: 84 store_id: 7 is_learner: true }"] [peer_id=84] [region_id=25]
[14:47:34.885 +08:00] [INFO] [peer.rs:2054] ["snapshot is applied"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 4 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 } peers { id: 84 store_id: 7 is_learner: true }"] [peer_id=84] [region_id=25]
[14:47:34.885 +08:00] [INFO] [region.rs:272] ["begin apply snap data"] [region_id=25]
[14:47:34.885 +08:00] [INFO] [region.rs:341] ["apply new data"] [time_takes=43.701µs] [region_id=25]
[14:47:35.864 +08:00] [INFO] [apply.rs:1060] ["execute admin command"] [command="cmd_type: ChangePeer change_peer { peer { id: 84 store_id: 7 } }"] [index=10] [term=6] [peer_id=84] [region_id=25]
[14:47:35.864 +08:00] [INFO] [apply.rs:1368] ["exec ConfChange"] [epoch="conf_ver: 4 version: 10"] [type=AddNode] [peer_id=84] [region_id=25]
[14:47:35.864 +08:00] [INFO] [apply.rs:1423] ["add peer successfully"] [region="id: 25 start_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"023\"000\"000\"000\"000\"000\"000\"000\"370" end_key: "t\"200\"000\"000\"000\"000\"000\"000\"377\"025\"000\"000\"000\"000\"000\"000\"000\"370" region_epoch { conf_ver: 4 version: 10 } peers { id: 26 store_id: 1 } peers { id: 74 store_id: 2 } peers { id: 84 store_id: 7 is_learner: true }"] [peer="id: 84 store_id: 7"] [peer_id=84] [region_id=25]


[14:47:42.609 +08:00] [INFO] [peer.rs:520] ["deleting applied snap file"] [snap_file=25_6_9] [peer_id=84] [region_id=25]

Raft Merge 场景

  • 本段日志暂无

Region status

Raft Leader 选举

  • Raft Leader 触发选举，本段日志是三副本均正常状态下触发 transfer leader 过程

TiKV 1 节点

[14:50:25.729 +08:00] [INFO] [peer.rs:1693] ["transfer leader"] [peer="id: 74 store_id: 2"] [peer_id=26] [region_id=25]
[14:50:25.729 +08:00] [INFO] [raft.rs:1294] ["[region 25] 26 [term 6] starts to transfer leadership to 74"]
[14:50:25.729 +08:00] [INFO] [raft.rs:1304] ["[region 25] 26 sends MsgTimeoutNow to 74 immediately as 74 already has up-to-date log"]
[14:50:25.729 +08:00] [INFO] [raft.rs:924] ["[region 147] 148 [term: 6] received a MsgRequestVote message with higher term from 149 [term: 7]"]
[14:50:25.729 +08:00] [INFO] [raft.rs:723] ["[region 147] 148 became follower at term 7"]
[14:50:25.729 +08:00] [INFO] [raft.rs:1108] ["[region 147] 148 [logterm: 6, index: 6, vote: 0] cast MsgRequestVote for 149 [logterm: 6, index: 6] at term 7"]
[14:50:25.745 +08:00] [INFO] [raft.rs:924] ["[region 25] 26 [term: 6] received a MsgRequestVote message with higher term from 74 [term: 7]"]
[14:50:25.745 +08:00] [INFO] [raft.rs:723] ["[region 25] 26 became follower at term 7"]
[14:50:25.745 +08:00] [INFO] [raft.rs:1108] ["[region 25] 26 [logterm: 6, index: 30, vote: 0] cast MsgRequestVote for 74 [logterm: 6, index: 30] at term 7"]
[14:50:27.355 +08:00] [ERROR] [endpoint.rs:452] [error-response] [err="region message: "peer is not leader" not_leader { region_id: 25 leader { id: 74 store_id: 2 } }"]

[14:57:56.933 +08:00] [INFO] [raft.rs:924] ["[region 25] 26 [term: 7] received a MsgRequestVote message with higher term from 84 [term: 8]"]
[14:57:56.933 +08:00] [INFO] [raft.rs:723] ["[region 25] 26 became follower at term 8"]
[14:57:56.933 +08:00] [INFO] [raft.rs:1108] ["[region 25] 26 [logterm: 7, index: 72, vote: 0] cast MsgRequestVote for 84 [logterm: 7, index: 72] at term 8"]

TiKV 2 节点

[14:50:25.729 +08:00] [INFO] [raft.rs:1662] ["[region 25] 74 [term 6] received MsgTimeoutNow from 26 and starts an election to get leadership."]
[14:50:25.730 +08:00] [INFO] [raft.rs:1094] ["[region 25] 74 is starting a new election at term 6"]
[14:50:25.730 +08:00] [INFO] [raft.rs:743] ["[region 25] 74 became candidate at term 7"]
[14:50:25.730 +08:00] [INFO] [raft.rs:858] ["[region 25] 74 received MsgRequestVoteResponse from 74 at term 7"]
[14:50:25.730 +08:00] [INFO] [raft.rs:832] ["[region 25] 74 [logterm: 6, index: 30] sent MsgRequestVote request to 84 at term 7"]
[14:50:25.730 +08:00] [INFO] [raft.rs:832] ["[region 25] 74 [logterm: 6, index: 30] sent MsgRequestVote request to 26 at term 7"]
[14:50:25.786 +08:00] [INFO] [raft.rs:858] ["[region 25] 74 received MsgRequestVoteResponse from 84 at term 7"]
[14:50:25.786 +08:00] [INFO] [raft.rs:1587] ["[region 25] 74 [quorum:2] has received 2 MsgRequestVoteResponse votes and 0 vote rejections"]
[14:50:25.786 +08:00] [INFO] [raft.rs:793] ["[region 25] 74 became leader at term 7"]

[14:57:56.905 +08:00] [INFO] [pd.rs:568] ["try to transfer leader"] [to_peer="id: 84 store_id: 7"] [from_peer="id: 74 store_id: 2"] [region_id=25]
[14:57:56.910 +08:00] [INFO] [peer.rs:1693] ["transfer leader"] [peer="id: 84 store_id: 7"] [peer_id=74] [region_id=25]
[14:57:56.910 +08:00] [INFO] [raft.rs:1294] ["[region 25] 74 [term 7] starts to transfer leadership to 84"]
[14:57:56.910 +08:00] [INFO] [raft.rs:1304] ["[region 25] 74 sends MsgTimeoutNow to 84 immediately as 84 already has up-to-date log"]
[14:57:56.933 +08:00] [INFO] [raft.rs:924] ["[region 25] 74 [term: 7] received a MsgRequestVote message with higher term from 84 [term: 8]"]
[14:57:56.933 +08:00] [INFO] [raft.rs:723] ["[region 25] 74 became follower at term 8"]
[14:57:56.933 +08:00] [INFO] [raft.rs:1108] ["[region 25] 74 [logterm: 7, index: 72, vote: 0] cast MsgRequestVote for 84 [logterm: 7, index: 72] at term 8"]

TiKV 7 节点

[14:50:25.744 +08:00] [INFO] [raft.rs:924] ["[region 25] 84 [term: 6] received a MsgRequestVote message with higher term from 74 [term: 7]"]
[14:50:25.744 +08:00] [INFO] [raft.rs:723] ["[region 25] 84 became follower at term 7"]
[14:50:25.744 +08:00] [INFO] [raft.rs:1108] ["[region 25] 84 [logterm: 6, index: 30, vote: 0] cast MsgRequestVote for 74 [logterm: 6, index: 30] at term 7"]

[14:57:56.910 +08:00] [INFO] [raft.rs:1662] ["[region 25] 84 [term 7] received MsgTimeoutNow from 74 and starts an election to get leadership."]
[14:57:56.910 +08:00] [INFO] [raft.rs:1094] ["[region 25] 84 is starting a new election at term 7"]
[14:57:56.910 +08:00] [INFO] [raft.rs:743] ["[region 25] 84 became candidate at term 8"]
[14:57:56.911 +08:00] [INFO] [raft.rs:858] ["[region 25] 84 received MsgRequestVoteResponse from 84 at term 8"]
[14:57:56.911 +08:00] [INFO] [raft.rs:832] ["[region 25] 84 [logterm: 7, index: 72] sent MsgRequestVote request to 26 at term 8"]
[14:57:56.911 +08:00] [INFO] [raft.rs:832] ["[region 25] 84 [logterm: 7, index: 72] sent MsgRequestVote request to 74 at term 8"]
[14:57:56.941 +08:00] [INFO] [raft.rs:858] ["[region 25] 84 received MsgRequestVoteResponse from 26 at term 8"]
[14:57:56.941 +08:00] [INFO] [raft.rs:1587] ["[region 25] 84 [quorum:2] has received 2 MsgRequestVoteResponse votes and 0 vote rejections"]
[14:57:56.941 +08:00] [INFO] [raft.rs:793] ["[region 25] 84 became leader at term 8"]

Region 25 status

{
  "id": 25,
  "start_key": "7480000000000000FF1300000000000000F8",
  "end_key": "7480000000000000FF1500000000000000F8",
  "epoch": {
    "conf_ver": 5,
    "version": 10
  },
  "peers": [
    {
      "id": 26,
      "store_id": 1
    },
    {
      "id": 74,
      "store_id": 2
    },
    {
      "id": 84,
      "store_id": 7
    }
  ],
  "leader": {
    "id": 84,
    "store_id": 7
  },
  "approximate_size": 1,
  "approximate_keys": 51
}

后记

  • 3.0 或者 4.0 时，可以通过 Learner 提供更多的场景使用，比如列存副本，这样分析任务就可以在 Learner 上去做
  • TiFlash 这个项目其实就是用 Learner 这个特性来增加只读副本，同时保证不会影响线上写入的延迟，因为它并不参与写入的时候投票。这样的好处是第一不影响写入延迟，第二有 Raft 实时同步数据，第三我们还能在上面快速地做很复杂的分析，同时线上 OLTP 业务有物理上的隔离
2
0
0
0

版权声明：本文为 TiDB 社区用户原创文章，遵循 CC BY-NC-SA 4.0 版权协议，转载请附上原文出处链接和本声明。

评论
暂无评论
Region 创建、分裂、合并TiDB 2.1 新特性使用场景Raft PreVoteRaft LearnerRegion Merge 场景部署架构日志分析Prevote 场景Raft Learner 场景Raft Merge 场景Region statusRaft Leader 选举后记