说明：测试环境v4.0.15，对于cdc 来说是一个非常老的版本，可能存在比较多的问题，如果是生产环境，尽量升级到比较新的版本，比如是v6.1.6，v6.5.1 这些版本无论是在性能和功能上面都有非常大的提升。下面的问题在v5.4.1测试就没有问题，所以推荐使用新的稳定的LTS版本。
cdc 的基本操作命令
#创建cdc
tiup ctl:v4.0.15 cdc changefeed create --pd=http://10.2.103.115:32379 --sink-uri="tidb://root:tidb@10.2.103.116:34000/" --changefeed-id="simple-replication-task" --config=cdc.toml
#查看cdc 任务状态
tiup ctl:v4.0.15 cdc changefeed list --pd=http://10.2.103.115:32379
#查看具体任务状态
tiup ctl:v4.0.15 cdc changefeed query -c simple-replication-task --pd=http://10.2.103.115:32379
#移除任务
tiup ctl:v4.0.15 cdc changefeed remove -c simple-replication-task --pd=http://10.2.103.115:32379
PD的状态
[tidb@vm115 ~]$ tiup cluster display tidb-dev
tiup is checking updates for component cluster ...
A new version of cluster is available:
The latest version: v1.12.0
Local installed version: v1.11.3
Update current component: tiup update cluster
Update all components: tiup update --all
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster display tidb-dev
Cluster type: tidb
Cluster name: tidb-dev
Cluster version: v4.0.15
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://10.2.103.115:32379/dashboard
Grafana URL: http://10.2.103.115:7000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
10.2.103.115:9793 alertmanager 10.2.103.115 9793/9794 linux/x86_64 Up /data1/tidb-data/alertmanager-9793 /data1/tidb-deploy/alertmanager-9793
10.2.103.115:9893 alertmanager 10.2.103.115 9893/9894 linux/x86_64 Up /data1/tidb-data/alertmanager-9893 /data1/tidb-deploy/alertmanager-9893
10.2.103.115:8400 cdc 10.2.103.115 8400 linux/x86_64 Up /data1/tidb-data/cdc-8400 /data1/tidb-deploy/cdc-8400
10.2.103.115:7000 grafana 10.2.103.115 7000 linux/x86_64 Up - /data1/tidb-deploy/grafana-7000
10.2.103.115:32379 pd 10.2.103.115 32379/3380 linux/x86_64 Up|L|UI /data1/tidb-data/pd-32379 /data1/tidb-deploy/pd-32379
10.2.103.115:35379 pd 10.2.103.115 35379/3580 linux/x86_64 Up /data1/tidb-data/pd-35379 /data1/tidb-deploy/pd-35379
10.2.103.115:36379 pd 10.2.103.115 36379/3680 linux/x86_64 Up /data1/tidb-data/pd-36379 /data1/tidb-deploy/pd-36379
10.2.103.115:9590 prometheus 10.2.103.115 9590/35020 linux/x86_64 Up /data1/tidb-data/prometheus-9590 /data1/tidb-deploy/prometheus-9590
10.2.103.115:43000 tidb 10.2.103.115 43000/20080 linux/x86_64 Up - /data1/tidb-deploy/tidb-34000
10.2.103.115:30160 tikv 10.2.103.115 30160/30180 linux/x86_64 Up /data1/tidb-data/tikv-30160 /data1/tidb-deploy/tikv-30160
Total nodes: 10
cdc 任务状态
[tidb@vm115 ~]$ tiup ctl:v4.0.15 cdc changefeed list --pd=http://10.2.103.115:32379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.15/ctl cdc changefeed list --pd=http://10.2.103.115:32379
[
{
"id": "simple-replication-task2",
"summary": {
"state": "normal",
"tso": 440757324149424131,
"checkpoint": "2023-04-13 11:15:59.237",
"error": null
}
}
]
切换PD leader
[tidb@vm115 ~]$ tiup ctl:v4.0.15 pd -u http://10.2.103.115:32379 member leader transfer pd-10.2.103.115-35379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.15/ctl pd -u http://10.2.103.115:32379 member leader transfer pd-10.2.103.115-35379
Success!
[tidb@vm115 ~]$ tiup cluster display tidb-dev
tiup is checking updates for component cluster ...
A new version of cluster is available:
The latest version: v1.12.0
Local installed version: v1.11.3
Update current component: tiup update cluster
Update all components: tiup update --all
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster display tidb-dev
Cluster type: tidb
Cluster name: tidb-dev
Cluster version: v4.0.15
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://10.2.103.115:32379/dashboard
Grafana URL: http://10.2.103.115:7000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
10.2.103.115:9793 alertmanager 10.2.103.115 9793/9794 linux/x86_64 Up /data1/tidb-data/alertmanager-9793 /data1/tidb-deploy/alertmanager-9793
10.2.103.115:9893 alertmanager 10.2.103.115 9893/9894 linux/x86_64 Up /data1/tidb-data/alertmanager-9893 /data1/tidb-deploy/alertmanager-9893
10.2.103.115:8400 cdc 10.2.103.115 8400 linux/x86_64 Up /data1/tidb-data/cdc-8400 /data1/tidb-deploy/cdc-8400
10.2.103.115:7000 grafana 10.2.103.115 7000 linux/x86_64 Up - /data1/tidb-deploy/grafana-7000
10.2.103.115:32379 pd 10.2.103.115 32379/3380 linux/x86_64 Up|UI /data1/tidb-data/pd-32379 /data1/tidb-deploy/pd-32379
10.2.103.115:35379 pd 10.2.103.115 35379/3580 linux/x86_64 Up|L /data1/tidb-data/pd-35379 /data1/tidb-deploy/pd-35379
10.2.103.115:36379 pd 10.2.103.115 36379/3680 linux/x86_64 Up /data1/tidb-data/pd-36379 /data1/tidb-deploy/pd-36379
10.2.103.115:9590 prometheus 10.2.103.115 9590/35020 linux/x86_64 Up /data1/tidb-data/prometheus-9590 /data1/tidb-deploy/prometheus-9590
10.2.103.115:43000 tidb 10.2.103.115 43000/20080 linux/x86_64 Up - /data1/tidb-deploy/tidb-34000
10.2.103.115:30160 tikv 10.2.103.115 30160/30180 linux/x86_64 Up /data1/tidb-data/tikv-30160 /data1/tidb-deploy/tikv-30160
Total nodes: 10
切换PD leader 对cdc 没有影响
[tidb@vm115 ~]$ tiup ctl:v4.0.15 cdc changefeed list --pd=http://10.2.103.115:32379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.15/ctl cdc changefeed list --pd=http://10.2.103.115:32379
[
{
"id": "simple-replication-task2",
"summary": {
"state": "normal",
"tso": 440757380830461955,
"checkpoint": "2023-04-13 11:19:35.458",
"error": null
}
}
]
[tidb@vm115 ~]$
缩容PD节点
cdc 任务报错
[tidb@vm115 ~]$ tiup cluster scale-in tidb-dev -N 10.2.103.115:35379
tiup is checking updates for component cluster ...
A new version of cluster is available:
The latest version: v1.12.1
Local installed version: v1.11.3
Update current component: tiup update cluster
Update all components: tiup update --all
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster scale-in tidb-dev -N 10.2.103.115:35379
This operation will delete the 10.2.103.115:35379 nodes in `tidb-dev` and all their data.
Do you want to continue? [y/N]:(default=N) y
Scale-in nodes...
+ [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-dev/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-dev/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [ Serial ] - ClusterOperate: operation=DestroyOperation, options={Roles:[] Nodes:[10.2.103.115:35379] Force:false SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:tidb SSHProxyIdentity:/home/tidb/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}
Stopping component pd
Stopping instance 10.2.103.115
Stop pd 10.2.103.115:35379 success
Destroying component pd
Destroying instance 10.2.103.115
Destroy 10.2.103.115 success
- Destroy pd paths: [/data1/tidb-data/pd-35379 /data1/tidb-deploy/pd-35379/log /data1/tidb-deploy/pd-35379 /etc/systemd/system/pd-35379.service]
+ [ Serial ] - UpdateMeta: cluster=tidb-dev, deleted=`'10.2.103.115:35379'`
+ [ Serial ] - UpdateTopology: cluster=tidb-dev
+ Refresh instance configs
- Generate config pd -> 10.2.103.115:32379 ... Done
- Generate config pd -> 10.2.103.115:36379 ... Done
- Generate config tikv -> 10.2.103.115:30160 ... Done
- Generate config tidb -> 10.2.103.115:43000 ... Done
- Generate config cdc -> 10.2.103.115:8400 ... Done
- Generate config prometheus -> 10.2.103.115:9590 ... Done
- Generate config grafana -> 10.2.103.115:7000 ... Done
- Generate config alertmanager -> 10.2.103.115:9793 ... Done
- Generate config alertmanager -> 10.2.103.115:9893 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 10.2.103.115:9590 ... Done
- Reload grafana -> 10.2.103.115:7000 ... Done
Scaled cluster `tidb-dev` in successfully
cdc任务报错
[tidb@vm115 ~]$ tiup ctl:v4.0.15 cdc changefeed list --pd=http://10.2.103.115:32379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.15/ctl cdc changefeed list --pd=http://10.2.103.115:32379
[2023/04/14 09:18:52.714 +08:00] [WARN] [client_changefeed.go:170] ["query changefeed info failed"] [error="owner not found"]
[
{
"id": "simple-replication-task2",
"summary": null
}
]
[tidb@vm115 ~]$ tiup ctl:v4.0.15 cdc changefeed list --pd=http://10.2.103.115:32379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.15/ctl cdc changefeed list --pd=http://10.2.103.115:32379
[2023/04/14 09:19:00.720 +08:00] [WARN] [client_changefeed.go:170] ["query changefeed info failed"] [error="owner not found"]
[
{
"id": "simple-replication-task2",
"summary": null
}
]
[tidb@vm115 ~]$ tiup ctl:v4.0.15 cdc changefeed list --pd=http://10.2.103.115:32379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.15/ctl cdc changefeed list --pd=http://10.2.103.115:32379
[
{
"id": "simple-replication-task2",
"summary": {
"state": "stopped",
"tso": 440778127778512897,
"checkpoint": "2023-04-14 09:18:38.784",
"error": {
"addr": "10.2.103.115:8400",
"code": "CDC:ErrProcessorUnknown",
"message": "failed to update info: [CDC:ErrReachMaxTry]reach maximum try: 3"
}
}
}
]
[tidb@vm115 ~]$
cdc 报错日志
[2023/04/14 09:18:40.348 +08:00] [ERROR] [processor.go:497] ["failed to flush task position"] [changefeed=simple-replication-task2] [error="[CDC:ErrPDEtcdAPIError]rpc error: code = Unknown desc = raft: stopped"] [errorVerbose="[CDC:ErrPDEtcdAPIError]rpc error: code = Unknown desc = raft: stopped\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByCause\n\tgithub.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:279\ngithub.com/pingcap/ticdc/pkg/errors.WrapError\n\tgithub.com/pingcap/ticdc@/pkg/errors/helper.go:30\ngithub.com/pingcap/ticdc/cdc/kv.CDCEtcdClient.PutTaskPositionOnChange\n\tgithub.com/pingcap/ticdc@/cdc/kv/etcd.go:739\ngithub.com/pingcap/ticdc/cdc.(*oldProcessor).flushTaskPosition\n\tgithub.com/pingcap/ticdc@/cdc/processor.go:494\ngithub.com/pingcap/ticdc/cdc.(*oldProcessor).flushTaskStatusAndPosition\n\tgithub.com/pingcap/ticdc@/cdc/processor.go:560\ngithub.com/pingcap/ticdc/cdc.(*oldProcessor).positionWorker.func1.1\n\tgithub.com/pingcap/ticdc@/cdc/processor.go:318\ngithub.com/pingcap/ticdc/pkg/retry.run\n\tgithub.com/pingcap/ticdc@/pkg/retry/retry_with_opt.go:54\ngithub.com/pingcap/ticdc/pkg/retry.Do\n\tgithub.com/pingcap/ticdc@/pkg/retry/retry_with_opt.go:32\ngithub.com/pingcap/ticdc/cdc.(*oldProcessor).positionWorker.func1\n\tgithub.com/pingcap/ticdc@/cdc/processor.go:317\ngithub.com/pingcap/ticdc/cdc.(*oldProcessor).positionWorker.func2\n\tgithub.com/pingcap/ticdc@/cdc/processor.go:349\ngithub.com/pingcap/ticdc/cdc.(*oldProcessor).positionWorker\n\tgithub.com/pingcap/ticdc@/cdc/processor.go:418\ngithub.com/pingcap/ticdc/cdc.(*oldProcessor).Run.func1\n\tgithub.com/pingcap/ticdc@/cdc/processor.go:251\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20201020160332-67f06af15bc9/errgroup/errgroup.go:57\nruntime.goexit\n\truntime/asm_amd64.s:1357"]
解决方案
重新的resume cdc 任务
[tidb@vm115 ~]$ tiup ctl:v4.0.15 cdc changefeed resume -c simple-replication-task2 --pd=http://10.2.103.115:32379
[tidb@vm115 ~]$ tiup ctl:v4.0.15 cdc changefeed list --pd=http://10.2.103.115:32379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v4.0.15/ctl cdc changefeed list --pd=http://10.2.103.115:32379
[
{
"id": "simple-replication-task2",
"summary": {
"state": "normal",
"tso": 440778162669879297,
"checkpoint": "2023-04-14 09:20:51.884",
"error": null
}
}
]
[tidb@vm115 ~]$
升级到v5.4.1测试
缩容PD
同样的操作，cdc 任务不报错
[tidb@vm115 ~]$ tiup cluster scale-in tidb-dev -N 10.2.103.115:35379
tiup is checking updates for component cluster ...
A new version of cluster is available:
The latest version: v1.12.1
Local installed version: v1.11.3
Update current component: tiup update cluster
Update all components: tiup update --all
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster scale-in tidb-dev -N 10.2.103.115:35379
This operation will delete the 10.2.103.115:35379 nodes in `tidb-dev` and all their data.
Do you want to continue? [y/N]:(default=N) y
Scale-in nodes...
+ [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-dev/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-dev/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [Parallel] - UserSSH: user=tidb, host=10.2.103.115
+ [ Serial ] - ClusterOperate: operation=DestroyOperation, options={Roles:[] Nodes:[10.2.103.115:35379] Force:false SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:tidb SSHProxyIdentity:/home/tidb/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}
Stopping component pd
Stopping instance 10.2.103.115
Stop pd 10.2.103.115:35379 success
Destroying component pd
Destroying instance 10.2.103.115
Destroy 10.2.103.115 success
- Destroy pd paths: [/data1/tidb-data/pd-35379 /data1/tidb-deploy/pd-35379/log /data1/tidb-deploy/pd-35379 /etc/systemd/system/pd-35379.service]
+ [ Serial ] - UpdateMeta: cluster=tidb-dev, deleted=`'10.2.103.115:35379'`
+ [ Serial ] - UpdateTopology: cluster=tidb-dev
+ Refresh instance configs
- Generate config pd -> 10.2.103.115:32379 ... Done
- Generate config pd -> 10.2.103.115:36379 ... Done
- Generate config tikv -> 10.2.103.115:30160 ... Done
- Generate config tidb -> 10.2.103.115:43000 ... Done
- Generate config cdc -> 10.2.103.115:8400 ... Done
- Generate config prometheus -> 10.2.103.115:9590 ... Done
- Generate config grafana -> 10.2.103.115:7000 ... Done
- Generate config alertmanager -> 10.2.103.115:9793 ... Done
- Generate config alertmanager -> 10.2.103.115:9893 ... Done
+ Reload prometheus and grafana
- Reload prometheus -> 10.2.103.115:9590 ... Done
- Reload grafana -> 10.2.103.115:7000 ... Done
Scaled cluster `tidb-dev` in successfully
[tidb@vm115 ~]$
cdc 状态正常
[tidb@vm115 ~]$ tiup ctl:v5.4.1 cdc changefeed list --pd=http://10.2.103.115:32379
Starting component `ctl`: /home/tidb/.tiup/components/ctl/v5.4.1/ctl cdc changefeed list --pd=http://10.2.103.115:32379
[
{
"id": "simple-replication-task2",
"summary": {
"state": "normal",
"tso": 440778284890324994,
"checkpoint": "2023-04-14 09:28:38.118",
"error": null
}
}
]
[tidb@vm115 ~]$
总结：
1、任务生产上面的变更，如果有条件都要在测试环境模拟、测试一下。
2、生产集群尽量升级到一些主流、稳定的版本上，过老的版本可能存在一些BUG。
3、最新的LTS版本 cdc 功能和性能都要质的飞跃，推荐使用新的LTS版本。