背景
随着业务的变更，可能经常会遇到TiDB数据库的TiKV或TIDB Server节点扩缩容的需求。下面记录了在虚机环境下，完整的TiDBv5.3.0数据库的安装，扩缩容和升级过程，也记录了TiSark的部署过程。给有需要的小伙伴提供个参考。
安装TiDBv5.3.0
为了方便后面的升级测试，选择安装了较低版本的TiDB-v5.3.0。
安装环境
操作系统版本：CentOS 7.9
TiDB数据库版本：TIDB v5.3.0
TIDB数据库安装拓扑：
PD：3节点
TiDB：1节点（这里安装了最小节点数，原因是想后面进行TiDB节点扩容操作）
TiKV：3节点
PS: 以上PD,TiDB和TiKV的最小安装节点个数都是1节点。
系统配置
安装TiDB前要进行一些准备工作，创建安装目录，创建安装用户以及系统参数优化等工作。
挂载数据盘
- 查看数据盘。/dev/sdb
fdisk -l
- 创建分区。
parted -s -a optimal /dev/sdb mklabel gpt -- mkpart primary ext4 1 -1
- 格式化文件系统。
mkfs.ext4 /dev/sdb1
- 查看数据盘分区 UUID。
# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1 xfs 278ae69e-30a6-40ca-a764-cd8862ef1527 /boot
└─sda2 LVM2_member tIRIrI-soRh-5TBf-q6tn-cMTd-DMdZ-FucCk7
├─centos-root xfs 13d08fa9-88bc-4b0a-bcab-4e1c1072afd2 /
└─centos-swap swap 488f568c-b72f-414e-9394-0765cbb9e5a2 [SWAP]
sdb
└─sdb1 ext4 87d0467c-d3a1-4916-8112-aed259bf8c8c
本例中 nvme0n1p1 的 UUID 为 87d0467c-d3a1-4916-8112-aed259bf8c8c。
- 编辑 /etc/fstab 文件，添加 nodelalloc 挂载参数。
# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Wed Apr 7 15:27:42 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=278ae69e-30a6-40ca-a764-cd8862ef1527 /boot xfs defaults 0 0
/dev/mapper/centos-swap swap swap defaults 0 0
UUID=87d0467c-d3a1-4916-8112-aed259bf8c8c /u02 ext4 defaults,nodelalloc,noatime 0 2
- 挂载数据盘。
mkdir /data1 && mount -a
- 执行以下命令，如果文件系统为 ext4，并且挂载参数中包含 nodelalloc，则表示已生效。
# mount -t ext4
/dev/sdb1 on /u02 type ext4 (rw,noatime,nodelalloc,data=ordered)
关闭防火墙
- 检查防火墙状态（以 CentOS Linux release 7.7.1908 (Core) 为例）
sudo firewall-cmd --state
sudo systemctl status firewalld.service
- 关闭防火墙服务
sudo systemctl stop firewalld.service
- 关闭防火墙自动启动服务
sudo systemctl disable firewalld.service
- 检查防火墙状态
sudo systemctl status firewalld.service
开启时钟同步
检查chronyd服务状态
[root@cen7-pg-01 ~]# systemctl status chronyd.service
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:chronyd(8)
man:chrony.conf(5)
[root@cen7-pg-01 ~]# chronyc tracking
506 Cannot talk to daemon
设置chronyd同步服务器，开启所有节点时钟同步功能
# 修改同步服务器配置文件：
# vi /etc/chrony.conf
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 192.168.56.10 iburst
# Allow NTP client access from local network.
allow 192.168.0.0/16
# Serve time even if not synchronized to a time source.
local stratum 10
# 修改其他节点chronyd服务配置文件：
# vi /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 192.168.56.10 iburst
# 启动chronyd服务:
# systemctl start chronyd.service
# systemctl status chronyd.service
● chronyd.service - NTP client/server
Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2022-07-22 14:39:39 CST; 4s ago
Docs: man:chronyd(8)
man:chrony.conf(5)
Process: 5505 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 5501 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 5504 (chronyd)
Tasks: 1
CGroup: /system.slice/chronyd.service
└─5504 /usr/sbin/chronyd
Jul 22 14:39:39 cen7-mysql-01 systemd[1]: Stopped NTP client/server.
Jul 22 14:39:39 cen7-mysql-01 systemd[1]: Starting NTP client/server...
Jul 22 14:39:39 cen7-mysql-01 chronyd[5504]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG)
Jul 22 14:39:39 cen7-mysql-01 chronyd[5504]: Frequency 0.156 +/- 235.611 ppm read from /var/lib/chrony/drift
Jul 22 14:39:39 cen7-mysql-01 systemd[1]: Started NTP client/server.
Jul 22 14:39:44 cen7-mysql-01 chronyd[5504]: Selected source 192.168.56.10
-- 检查系统时钟同步状态：Leap status = Normal表示状态正常
[root@cen7-mysql-01 ~]# chronyc tracking
Reference ID : C0A8380A (cen7-mysql-01)
Stratum : 11
Ref time (UTC) : Fri Jul 22 06:39:43 2022
System time : 0.000000000 seconds fast of NTP time
Last offset : -0.000003156 seconds
RMS offset : 0.000003156 seconds
Frequency : 0.107 ppm fast
Residual freq : -0.019 ppm
Skew : 250.864 ppm
Root delay : 0.000026199 seconds
Root dispersion : 0.001826834 seconds
Update interval : 0.0 seconds
Leap status : Normal
-- 设置chronyd服务开机自启动
# systemctl enable chronyd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/chronyd.service to /usr/lib/systemd/system/chronyd.service.
优化系统参数
在生产系统的 TiDB 中，建议按照官方文档TiDB 环境与系统配置检查 | PingCAP Docs对操作系统进行配置优化。
本次在虚机环境进行安装测试，只关闭了透明大页：
-- 检查操作系统时候关闭透明大页：
# cat /sys/kernel/mm/transparent_hugepage/enabled
# cat /sys/kernel/mm/transparent_hugepage/defrag
如果查询为：
[always] madvise never
表示透明大页处于启用状态，需要关闭。
-- 关闭操作系统的透明大页方法：
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
-- 永久关闭操作系统的透明大页方法：
# vi /etc/rc.d/rc.local
-- 在文件尾追加如下内容
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
# chmod +x /etc/rc.d/rc.local
创建TiDB用户
# groupadd tidb
# useradd -g tidb tidb
# passwd tidb
Changing password for user tidb.
New password:-- 输入tidb用户密码
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: -- 再次输入tidb用户密码
passwd: all authentication tokens updated successfully.
-- 创建数据库安装目录，并赋权
# mkdir /u02/tidb-data
# mkdir /u02/tidb-deploy
# chown -R tidb: /u02
修改 sysctl 参数
--修改/etc/sysctl.conf 设置操作系统参数
# echo "fs.file-max = 1000000">> /etc/sysctl.conf
# echo "net.core.somaxconn = 32768">> /etc/sysctl.conf
# echo "net.ipv4.tcp_tw_recycle = 0">> /etc/sysctl.conf
# echo "net.ipv4.tcp_syncookies = 0">> /etc/sysctl.conf
# echo "vm.overcommit_memory = 1">> /etc/sysctl.conf
-- 使修改生效
# sysctl -p
配置tidb用户的 limits.conf 文件
# cat << EOF >>/etc/security/limits.conf
tidb soft nofile 1000000
tidb hard nofile 1000000
tidb soft stack 32768
tidb hard stack 32768
EOF
安装TiDB
部署离线环境 TiUP 组件
-- 解压TIDB安装文件
# tar -zxf tidb-community-server-v5.3.0-linux-amd64.tar.gz
# tar -zxf tidb-community-toolkit-v5.3.0-linux-amd64.tar.gz
# chown -R tidb: /u01/soft/ti*
# su - tidb
-- 配置TIDB相关环境变量
$ cd /u01/soft/tidb-community-server-v5.3.0-linux-amd64/
$ sh ./local_install.sh && source /home/tidb/.bash_profile
Disable telemetry success
Successfully set mirror to /u01/soft/tidb-community-server-v5.3.0-linux-amd64
Detected shell: bash
Shell profile: /home/tidb/.bash_profile
/home/tidb/.bash_profile has been modified to to add tiup to PATH
open a new terminal or source /home/tidb/.bash_profile to use it
Installed path: /home/tidb/.tiup/bin/tiup
===============================================
1. source /home/tidb/.bash_profile
2. Have a try: tiup playground
===============================================
$ which tiup
~/.tiup/bin/tiup
创建安装拓扑文件
$ vi topology.yaml
global:
user: "tidb"
ssh_port: 22
deploy_dir: "/u02/tidb-deploy"
data_dir: "/u02/tidb-data"
#server_configs: {}
pd_servers:
- host: 192.168.56.10
- host: 192.168.56.11
- host: 192.168.56.12
tidb_servers:
- host: 192.168.56.10
tikv_servers:
- host: 192.168.56.10
- host: 192.168.56.11
- host: 192.168.56.12
monitoring_servers:
- host: 192.168.56.10
grafana_servers:
- host: 192.168.56.10
alertmanager_servers:
- host: 192.168.56.10
安装tidb集群
这里使用TiUP离线部署方式安装TIDB：
-- 安装环境检查
$ tiup cluster check ./topology.yaml --user root -p
详细的检查结果已省略
在日志的最后可以看到哪些检查项未通过
-- 修复系统环境中存在的问题
$ tiup cluster deploy tidb-test v5.3.0 ./topology.yaml --user root -p
详细的检查结果已省略
在日志的最后如果看到的所有- Apply change on <IP> 的结果都为Done,表示修复成功
-- 开始安装TIDB集群
$ tiup cluster deploy tidb-test v5.3.0 ./topology.yaml --user root -p
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster deploy tidb-test v5.3.0 ./topology.yaml --user root -p
Input SSH password: root密码
+ Detect CPU Arch
- Detecting node 192.168.56.10 ... Done
- Detecting node 192.168.56.11 ... Done
- Detecting node 192.168.56.12 ... Done
Please confirm your topology:
Cluster type: tidb
Cluster name: tidb-test
Cluster version: v5.3.0
Role Host Ports OS/Arch Directories
---- ---- ----- ------- -----------
pd 192.168.56.10 2379/2380 linux/x86_64 /u02/tidb-deploy/pd-2379,/u02/tidb-data/pd-2379
pd 192.168.56.11 2379/2380 linux/x86_64 /u02/tidb-deploy/pd-2379,/u02/tidb-data/pd-2379
pd 192.168.56.12 2379/2380 linux/x86_64 /u02/tidb-deploy/pd-2379,/u02/tidb-data/pd-2379
tikv 192.168.56.10 20160/20180 linux/x86_64 /u02/tidb-deploy/tikv-20160,/u02/tidb-data/tikv-20160
tikv 192.168.56.11 20160/20180 linux/x86_64 /u02/tidb-deploy/tikv-20160,/u02/tidb-data/tikv-20160
tikv 192.168.56.12 20160/20180 linux/x86_64 /u02/tidb-deploy/tikv-20160,/u02/tidb-data/tikv-20160
tidb 192.168.56.10 4000/10080 linux/x86_64 /u02/tidb-deploy/tidb-4000
prometheus 192.168.56.10 9090 linux/x86_64 /u02/tidb-deploy/prometheus-9090,/u02/tidb-data/prometheus-9090
grafana 192.168.56.10 3000 linux/x86_64 /u02/tidb-deploy/grafana-3000
alertmanager 192.168.56.10 9093/9094 linux/x86_64 /u02/tidb-deploy/alertmanager-9093,/u02/tidb-data/alertmanager-9093
Attention:
1. If the topology is not what you expected, check your yaml file.
2. Please confirm there is no port/directory conflicts in same host.
Do you want to continue? [y/N]: (default=N) y --按照提示输入y,继续安装
...这里省略部分日志
Cluster `tidb-test` deployed successfully, you can start it with command: `tiup cluster start tidb-test`
看到安装过程最后一行提示‘deployed successfully’，表示安装成功
启动tidb集群
-- 查看当前从系统中的TiDB集群列表
$ tiup cluster list
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster list
Name User Version Path PrivateKey
---- ---- ------- ---- ----------
tidb-test tidb v5.3.0 /home/tidb/.tiup/storage/cluster/clusters/tidb-test /home/tidb/.tiup/storage/cluster/clusters/tidb-test/ssh/id_rsa
-- 启动tidb-test集群
$ tiup cluster start tidb-test
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster start tidb-test
Starting cluster tidb-test...
......中间的日志已省略
+ [ Serial ] - UpdateTopology: cluster=tidb-test
Started cluster `tidb-test` successfully
-- 查看集群状态
$ tiup cluster display tidb-test
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster display tidb-test
Cluster type: tidb
Cluster name: tidb-test
Cluster version: v5.3.0
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.56.11:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.56.10:9093 alertmanager 192.168.56.10 9093/9094 linux/x86_64 Up /u02/tidb-data/alertmanager-9093 /u02/tidb-deploy/alertmanager-9093
192.168.56.10:3000 grafana 192.168.56.10 3000 linux/x86_64 Up - /u02/tidb-deploy/grafana-3000
192.168.56.10:2379 pd 192.168.56.10 2379/2380 linux/x86_64 Up|L /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.11:2379 pd 192.168.56.11 2379/2380 linux/x86_64 Up|UI /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.12:2379 pd 192.168.56.12 2379/2380 linux/x86_64 Up /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.10:9090 prometheus 192.168.56.10 9090 linux/x86_64 Up /u02/tidb-data/prometheus-9090 /u02/tidb-deploy/prometheus-9090
192.168.56.10:4000 tidb 192.168.56.10 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.10:20160 tikv 192.168.56.10 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.11:20160 tikv 192.168.56.11 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.12:20160 tikv 192.168.56.12 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
Total nodes: 10
以上集群状态正常。
至此，TiDB集群安装完毕。
我们可以使用mysql 客户端工具连接TIDB数据库，进行数据库操作。
TIDB集群部署总结
在CentOS7.9 操作系统上进行TiDB v5.3.0数据库安装操作，参考官方文档的操作步骤基本可以顺利完成安装：
TiDB 软件和硬件环境建议配置 | PingCAP Docs
其中，TiDB 环境与系统配置检查部分，需要根据实际的安装环境进行调整。
附，安装TiDBv5.3.0过程中遇到的问题
按照官方文档将 server 和 toolkit 两个离线镜像合并时，报错了。但是不进行server 和 toolkit 的合并操作也可以顺利完成安装过程。
报错信息如下:
$ cd /u01/soft/tidb-community-server-v5.3.0-linux-amd64/
$ cp -rp keys ~/.tiup/
$ tiup mirror merge ../tidb-community-toolkit-v5.3.0-linux-amd64
Error: resource snapshot.json: not found
TiDB集群扩缩容
TiDB 集群可以在不中断线上服务的情况下进行扩容和缩容。
扩缩容操作参考官方文档：使用 TiUP 扩容缩容 TiDB 集群 | PingCAP Docs
扩容：
修改扩容配置文件，扩容一个tidb server节点和一个tikv节点
注：这里如果新TiDB节点信息中存在新加入TiDB集群的主机，在扩容前，需要按照上面TIDB安装步骤中系统配置部分的内容对新节点进行配置。
vi scale-out.yaml
tidb_servers:
- host: 192.168.56.11
tikv_servers:
- host: 192.168.56.150
进行扩容操作
--使用tiup cluster scale-out命令进行TiDB集群扩容
$ tiup cluster scale-out tidb-test ./scale-out.yaml -p -u root
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster scale-out tidb-test ./scale-out.yaml -p -u root
Input SSH password: root密码
+ Detect CPU Arch
- Detecting node 192.168.56.150 ... Done
...省略中间日志
+ [ Serial ] - UpdateTopology: cluster=tidb-test
Scaled cluster `tidb-test` out successfully
--看到扩容过程最后一行提示'Scaled cluster `集群名称` out successfully',表示扩容成功
-- 扩容后再次查看集群状态，确认新节点已经成功加入集群
$ tiup cluster display tidb-test
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster display tidb-test
Cluster type: tidb
Cluster name: tidb-test
Cluster version: v5.3.0
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.56.11:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.56.10:9093 alertmanager 192.168.56.10 9093/9094 linux/x86_64 Up /u02/tidb-data/alertmanager-9093 /u02/tidb-deploy/alertmanager-9093
192.168.56.10:3000 grafana 192.168.56.10 3000 linux/x86_64 Up - /u02/tidb-deploy/grafana-3000
192.168.56.10:2379 pd 192.168.56.10 2379/2380 linux/x86_64 Up /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.11:2379 pd 192.168.56.11 2379/2380 linux/x86_64 Up|UI /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.12:2379 pd 192.168.56.12 2379/2380 linux/x86_64 Up|L /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.10:9090 prometheus 192.168.56.10 9090 linux/x86_64 Up /u02/tidb-data/prometheus-9090 /u02/tidb-deploy/prometheus-9090
192.168.56.10:4000 tidb 192.168.56.10 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.11:4000 tidb 192.168.56.11 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.10:20160 tikv 192.168.56.10 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.11:20160 tikv 192.168.56.11 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.12:20160 tikv 192.168.56.12 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.150:20160 tikv 192.168.56.150 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
Total nodes: 12
缩容：
-- 使用tiup cluster scale-in命令进行TiDB集群缩容操作，--node参数指定要移除的节点的IP和端口号
$ tiup cluster scale-in tidb-test --node 192.168.56.150:20160
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster scale-in tidb-test --node 192.168.56.150:20160
This operation will delete the 192.168.56.150:20160 nodes in `tidb-test` and all their data.
Do you want to continue? [y/N]:(default=N) y
Scale-in nodes...
...省略中间日志
Scaled cluster `tidb-test` in successfully
--看到缩容过程最后一行提示'Scaled cluster `集群名称` in successfully',表示缩容成功
-- 扩容后再次查看集群状态，确认被移除节点状态已经更改为Tombstone
$ tiup cluster display tidb-test
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster display tidb-test
Cluster type: tidb
Cluster name: tidb-test
Cluster version: v5.3.0
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.56.11:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.56.10:9093 alertmanager 192.168.56.10 9093/9094 linux/x86_64 Up /u02/tidb-data/alertmanager-9093 /u02/tidb-deploy/alertmanager-9093
192.168.56.10:3000 grafana 192.168.56.10 3000 linux/x86_64 Up - /u02/tidb-deploy/grafana-3000
192.168.56.10:2379 pd 192.168.56.10 2379/2380 linux/x86_64 Up /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.11:2379 pd 192.168.56.11 2379/2380 linux/x86_64 Up|UI /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.12:2379 pd 192.168.56.12 2379/2380 linux/x86_64 Up|L /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.10:9090 prometheus 192.168.56.10 9090 linux/x86_64 Up /u02/tidb-data/prometheus-9090 /u02/tidb-deploy/prometheus-9090
192.168.56.10:4000 tidb 192.168.56.10 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.11:4000 tidb 192.168.56.11 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.10:20160 tikv 192.168.56.10 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.11:20160 tikv 192.168.56.11 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.12:20160 tikv 192.168.56.12 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.150:20160 tikv 192.168.56.150 20160/20180 linux/x86_64 Tombstone /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
Total nodes: 12
There are some nodes can be pruned:
Nodes: [192.168.56.150:20160]
You can destroy them with the command: `tiup cluster prune tidb-test`
-- 根据上面提示，我们使用tiup cluster prune <集群名称> 命令，彻底删除已被移除集群的节点
$ tiup cluster prune tidb-test
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster prune tidb-test
...省略中间日志
Destroy success
-- 看到命令最后一行提示'Destroy success'.可再次查询集群状态，确认被移除节点信息已被清理
TiSpark部署方法
这里介绍两种部署TiSpark的方法
方法一，Tiup Cluster部署TiSpark
关于使用Tiup工具安装TiSpark的部署说明可参考官方文档：
首先安装系统java环境
--安装jdk8过程略
修改扩容配置文件
注：这里如果新TiSpark节点信息中存在新加入TiDB集群的主机，在扩容前，需要按照上面TIDB安装步骤中系统配置部分的内容对新节点进行配置。
$ vi scale-out.yaml
tispark_masters:
- host: 192.168.56.150
tispark_workers:
- host: 192.168.56.12
使用Tiup工具部署TiSpark（目前官方还是实验阶段）
[tidb@cen7-pg-01 bin]$ pwd
/u02/tidb-deploy/tispark-worker-7078/bin
[tidb@cen7-mysql-01 ~]$ tiup cluster scale-out tidb-test ./scale-out.yaml -p -u root
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster scale-out tidb-test ./scale-out.yaml -p -u root
Input SSH password:
.........此处省略较长的输出日志
Scaled cluster `tidb-test` out successfully
-- 扩容后查看集群状态
[tidb@cen7-mysql-01 ~]$ tiup cluster display tidb-test
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.7.0/tiup-cluster display tidb-test
Cluster type: tidb
Cluster name: tidb-test
Cluster version: v5.3.0
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.56.11:2379/dashboard
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.56.10:9093 alertmanager 192.168.56.10 9093/9094 linux/x86_64 Up /u02/tidb-data/alertmanager-9093 /u02/tidb-deploy/alertmanager-9093
192.168.56.10:3000 grafana 192.168.56.10 3000 linux/x86_64 Up - /u02/tidb-deploy/grafana-3000
192.168.56.10:2379 pd 192.168.56.10 2379/2380 linux/x86_64 Up /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.11:2379 pd 192.168.56.11 2379/2380 linux/x86_64 Up|UI /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.12:2379 pd 192.168.56.12 2379/2380 linux/x86_64 Up|L /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.10:9090 prometheus 192.168.56.10 9090 linux/x86_64 Up /u02/tidb-data/prometheus-9090 /u02/tidb-deploy/prometheus-9090
192.168.56.10:4000 tidb 192.168.56.10 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.11:4000 tidb 192.168.56.11 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.10:20160 tikv 192.168.56.10 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.11:20160 tikv 192.168.56.11 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.12:20160 tikv 192.168.56.12 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.150:7077 tispark-master 192.168.56.150 7077/8080 linux/x86_64 Up - /u02/tidb-deploy/tispark-master-7077
192.168.56.12:7078 tispark-worker 192.168.56.12 7078/8081 linux/x86_64 Up - /u02/tidb-deploy/tispark-worker-7078
Total nodes: 13
-- tispark已成功部署
tispark查询测试
-- 测试使用spark-sql工具连接TiDB数据库
[tidb@cen7-pg-01 bin]$ pwd
/u02/tidb-deploy/tispark-worker-7078/bin
[tidb@cen7-pg-01 bin]$ ./spark-sql
......省略过长的输出日志
spark-sql> show databases;
22/07/28 10:17:27 INFO PDClient: Switched to new leader: [leaderInfo: 192.168.56.11:2379]
22/07/28 10:17:29 INFO ReflectionUtil$: tispark class url: file:/u02/tidb-deploy/tispark-worker-7078/jars/tispark-assembly-2.4.1.jar
22/07/28 10:17:29 INFO ReflectionUtil$: spark wrapper class url: jar:file:/u02/tidb-deploy/tispark-worker-7078/jars/tispark-assembly-2.4.1.jar!/resources/spark-wrapper-spark-2_4/
22/07/28 10:17:29 INFO HiveMetaStore: 0: get_databases: *
22/07/28 10:17:29 INFO audit: ugi=tidb ip=unknown-ip-addr cmd=get_databases: *
22/07/28 10:17:29 INFO CodeGenerator: Code generated in 342.766519 ms
default
test
mysql
Time taken: 4.469 seconds, Fetched 3 row(s)
22/07/28 10:17:29 INFO SparkSQLCLIDriver: Time taken: 4.469 seconds, Fetched 3 row(s)
--设置当前数据库
spark-sql> use test;
22/07/28 10:17:35 INFO HiveMetaStore: 0: get_database: test
22/07/28 10:17:35 INFO audit: ugi=tidb ip=unknown-ip-addr cmd=get_database: test
Time taken: 0.081 seconds
22/07/28 10:17:35 INFO SparkSQLCLIDriver: Time taken: 0.081 seconds
-- 查询t1表内容
spark-sql> select * from t1;
....省略中间日志
1 a
2 c
3 e
4 f
5 g
Time taken: 0.245 seconds, Fetched 5 row(s)
22/07/28 10:25:22 INFO SparkSQLCLIDriver: Time taken: 0.245 seconds, Fetched 5 row(s)
22/07/28 10:25:22 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
spark-sql>
方法二，在Tiup Cluster外单独部署TiSpark
参考文档：
官方文档中单独部署TiSpark指南:TiSpark 用户指南 | PingCAP Docs
GitHub中TiSpark的UserGuide:tispark/userguide.md at master · pingcap/tispark · GitHub
安装时要注意根据spark的安装版本，选择对应的TiSpark版本：TiSpark 用户指南 | PingCAP Docs
安装Spark3.1.3+TiSpark3.1
-- 在安装spark之前要先搭建将Java环境
安装jdk8过程略
-- 修改 /etc/hosts文件，配置Spark节点的主机名
# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.56.18 TiSpark01
192.168.56.19 TiSpark02
-- 创建spark 安装目录并设置访问权限
# mkdir /tidb-data
# groupadd tidb
# useradd -g tidb tidb
# chown -R tidb: /tidb-data
-- 解压Spark安装包
$ tar -zxf /tidb-data/spark-3.1.3-bin-hadoop3.2.tgz -C /home/tidb
-- 将tispark的jar包复制到到spark的jars目录中
$ cp /tidb-data/tispark-assembly-3.1-2.5.1.jar /home/tidb/spark-3.1.3-bin-hadoop3.2/jars/
-- 修改环境变量
$ vim .bash_profile
export SPARK_HOME=/home/tidb/spark-3.1.3-bin-hadoop3.2
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
--修改spark配置文件
$ cd spark-3.1.3-bin-hadoop3.2/conf/
$ ls -ltr
total 36
-rw-r--r-- 1 tidb tidb 865 Feb 6 18:05 workers.template
-rwxr-xr-x 1 tidb tidb 4428 Feb 6 18:05 spark-env.sh.template
-rw-r--r-- 1 tidb tidb 1292 Feb 6 18:05 spark-defaults.conf.template
-rw-r--r-- 1 tidb tidb 9141 Feb 6 18:05 metrics.properties.template
-rw-r--r-- 1 tidb tidb 2371 Feb 6 18:05 log4j.properties.template
-rw-r--r-- 1 tidb tidb 1105 Feb 6 18:05 fairscheduler.xml.template
-- 配置worker节点信息
$ mv spark-env.sh.template spark-env.sh
$ mv spark-defaults.conf.template spark-defaults.conf
$ mv workers.template workers
$ vi workers
注释localhost行
添加行：
192.168.56.19
-- 修改spark默认配置
--在spark-defaults.conf文件中添加TiSpark配置参数：
-- spark.tispark.pd.addresses 允许输入按逗号 (',') 分隔的多个 PD 服务器，请指定每个服务器的端口号。
[tidb@TiSpark01 conf]$ vi spark-defaults.conf
spark.tispark.pd.addresses 192.168.56.11:2379,192.168.56.13:2379,192.168.56.14:2379
spark.sql.extensions org.apache.spark.sql.TiExtensions
##对于 TiSpark 版本 >= 2.5.0，请添加以下附加配置以启用catalog
spark.sql.catalog.tidb_catalog org.apache.spark.sql.catalyst.catalog.TiCatalog
spark.sql.catalog.tidb_catalog.pd.addresses 192.168.56.11:2379,192.168.56.13:2379,192.168.56.14:2379
-- 设置spark运行参数
$ vi spark-env.sh
#设置运行master进程的节点
export SPARK_MASTER_HOST=master
#设置master的通信端口
export SPARK_MASTER_PORT=7077
#每个worker使用的核数
export SPARK_WORKER_CORES=1
#每个worker使用的内存大小
export SPARK_WORKER_MEMORY=1024M
#master的webui端口
export SPARK_MASTER_WEBUI_PORT=8080
#spark的配置文件目录
export SPARK_CONF_DIR=/home/tidb/spark-3.1.3-bin-hadoop3.2/conf
#jdk安装路径
export JAVA_HOME=/home/tidb/java/jdk1.8
-- 将修改后的spark目录复制到worker节点
$ scp -r ~/spark-3.1.3-bin-hadoop3.2/ 192.168.56.19:/home/tidb/
--启动spark
$ ~/spark-3.1.3-bin-hadoop3.2/sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/tidb/spark-3.1.3-bin-hadoop3.2/logs/spark-tidb-org.apache.spark.deploy.master.Master-1-TiSpark01.out
192.168.56.19: Welcome to SjCredit
192.168.56.19: starting org.apache.spark.deploy.worker.Worker, logging to /home/tidb/spark-3.1.3-bin-hadoop3.2/logs/spark-tidb-org.apache.spark.deploy.worker.Worker-1-TiSpark02.out
--查看spark进程，正常启动
$ ps -ef|grep spark
tidb 11868 1 16 04:55 ? 00:00:05 /home/ysops/java/jdk1.8/bin/java -cp /home/tidb/spark-3.1.3-bin-hadoop3.2/conf/:/home/tidb/spark-3.1.3-bin-hadoop3.2/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master:7077
tidb 11910 11823 0 04:56 pts/0 00:00:00 grep --color=auto spark
-- 分别在spark master和worker节点查看进程日志,确认spark正常运行
$ tail -100f /home/tidb/spark-3.1.3-bin-hadoop3.2/logs/spark-tidb-org.apache.spark.deploy.master.Master-1-TiSpark01.out
Spark Command: /home/ysops/java/jdk1.8/bin/java -cp /home/tidb/spark-3.1.3-bin-hadoop3.2/conf/:/home/tidb/spark-3.1.3-bin-hadoop3.2/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host TiSpark01 --port 7077 --webui-port 8080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/07/26 05:31:49 INFO Master: Started daemon with process name: 12394@TiSpark01
22/07/26 05:31:49 INFO SignalUtils: Registering signal handler for TERM
22/07/26 05:31:49 INFO SignalUtils: Registering signal handler for HUP
22/07/26 05:31:49 INFO SignalUtils: Registering signal handler for INT
22/07/26 05:31:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/07/26 05:31:49 INFO SecurityManager: Changing view acls to: tidb
22/07/26 05:31:49 INFO SecurityManager: Changing modify acls to: tidb
22/07/26 05:31:49 INFO SecurityManager: Changing view acls groups to:
22/07/26 05:31:49 INFO SecurityManager: Changing modify acls groups to:
22/07/26 05:31:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tidb); groups with view permissions: Set(); users with modify permissions: Set(tidb); groups with modify permissions: Set()
22/07/26 05:31:50 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
22/07/26 05:31:50 INFO Master: Starting Spark master at spark://TiSpark01:7077
22/07/26 05:31:50 INFO Master: Running Spark version 3.1.3
22/07/26 05:31:50 INFO Utils: Successfully started service 'MasterUI' on port 8080.
22/07/26 05:31:50 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://TiSpark01:8080
22/07/26 05:31:51 INFO Master: I have been elected leader! New state: ALIVE
22/07/26 05:31:57 INFO Master: Registering worker 192.168.56.19:45587 with 1 cores, 1024.0 MiB RAM
$ tail -100f /home/tidb/spark-3.1.3-bin-hadoop3.2/logs/spark-tidb-org.apache.spark.deploy.worker.Worker-1-TiSpark02.out
Spark Command: /home/ysops/tidb/jdk1.8/bin/java -cp /home/tidb/spark-3.1.3-bin-hadoop3.2/conf/:/home/tidb/spark-3.1.3-bin-hadoop3.2/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://TiSpark01:7077
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/07/26 05:31:53 INFO Worker: Started daemon with process name: 12418@TiSpark02
22/07/26 05:31:53 INFO SignalUtils: Registering signal handler for TERM
22/07/26 05:31:53 INFO SignalUtils: Registering signal handler for HUP
22/07/26 05:31:53 INFO SignalUtils: Registering signal handler for INT
22/07/26 05:31:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/07/26 05:31:54 INFO SecurityManager: Changing view acls to: tidb
22/07/26 05:31:54 INFO SecurityManager: Changing modify acls to: tidb
22/07/26 05:31:54 INFO SecurityManager: Changing view acls groups to:
22/07/26 05:31:54 INFO SecurityManager: Changing modify acls groups to:
22/07/26 05:31:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tidb); groups with view permissions: Set(); users with modify permissions: Set(tidb); groups with modify permissions: Set()
22/07/26 05:31:55 INFO Utils: Successfully started service 'sparkWorker' on port 45587.
22/07/26 05:31:55 INFO Worker: Worker decommissioning not enabled, SIGPWR will result in exiting.
22/07/26 05:31:56 INFO Worker: Starting Spark worker 192.168.56.19:45587 with 1 cores, 1024.0 MiB RAM
22/07/26 05:31:56 INFO Worker: Running Spark version 3.1.3
22/07/26 05:31:56 INFO Worker: Spark home: /home/tidb/spark-3.1.3-bin-hadoop3.2
22/07/26 05:31:56 INFO ResourceUtils: ==============================================================
22/07/26 05:31:56 INFO ResourceUtils: No custom resources configured for spark.worker.
22/07/26 05:31:56 INFO ResourceUtils: ==============================================================
22/07/26 05:31:57 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
22/07/26 05:31:57 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://TiSpark02:8081
22/07/26 05:31:57 INFO Worker: Connecting to master TiSpark01:7077...
22/07/26 05:31:57 INFO TransportClientFactory: Successfully created connection to TiSpark01/192.168.56.18:7077 after 125 ms (0 ms spent in bootstraps)
22/07/26 05:31:57 INFO Worker: Successfully registered with master spark://TiSpark01:7077
-- 到此Spark+TiSpark部署成功完成。
使用TiSpark连接TiDB数据库测试
-- 使用spark-shell工具连接TiDB数据库
$ spark-shell
.....省略输出日志
scala> spark.sql("use tidb_catalog.test")
......省略输出日志
scala> spark.sql("show tables").show
+---------+---------+
|namespace|tableName|
+---------+---------+
| test| nation|
| test| region|
| test| part|
| test| supplier|
| test| partsupp|
| test| customer|
| test| orders|
| test| lineitem|
+---------+---------+
scala> spark.sql("select count (*) from lineitem").show
+--------+
|count(1)|
+--------+
| 1818624|
+--------+
scala>
-- 使用spark-sql工具连接TiDB数据库
$ spark-sql
......省略输出日志
spark-sql> use tidb_catalog.test;
Time taken: 0.333 seconds
spark-sql> show tables;
test nation
test region
test part
test supplier
test partsupp
test customer
test orders
test lineitem
Time taken: 1.454 seconds, Fetched 8 row(s)
spark-sql> select count (*) from lineitem;
1941504
Time taken: 2.808 seconds, Fetched 1 row(s)
spark-sql>
TiSpark部署总结
两种部署方法的比对，很显然，使用TiUP工具部署TiSpark的操作会简易很多。但是在目前的6.1版本中TiUP Cluster 的 TiSpark 支持目前为实验特性，不建议在生产环境中使用。
TiDB5.3升级到TiDB6.1
升级的方式有两种：不停机升级和停机升级。TiUP Cluster 默认的升级 TiDB 集群的方式是不停机升级，即升级过程中集群仍然可以对外提供服务。升级时会对各节点逐个迁移 leader 后再升级和重启，因此对于大规模集群需要较长时间才能完成整个升级操作。如果业务有维护窗口可供数据库停机维护，则可以使用停机升级的方式快速进行升级操作。
升级操作参考官方文档：使用 TiUP 升级 TiDB | PingCAP Docs
下面是采用不停机升级的方式的操作过程。
解压安装包，更新 TiUP 离线镜像
-- 解压新版本的安装包和工具包
# tar -zxf tidb-community-server-v6.1.0-linux-amd64.tar.gz
# tar -zxf tidb-community-toolkit-v6.1.0-linux-amd64.tar.gz
-- 修改目录权限
# chown -R tidb: /u01/soft/tidb*
-- 配置环境变量
$ sh tidb-community-server-v6.1.0-linux-amd64/local_install.sh && source /home/tidb/.bash_profile
Disable telemetry success
Successfully set mirror to /u01/soft/tidb-community-server-v6.1.0-linux-amd64
Detected shell: bash
Shell profile: /home/tidb/.bash_profile
Installed path: /home/tidb/.tiup/bin/tiup
===============================================
1. source /home/tidb/.bash_profile
2. Have a try: tiup playground
===============================================
-- 合并server 和 toolkit 两个离线镜像
$ cd tidb-community-server-v6.1.0-linux-amd64/
$ cp -rp keys ~/.tiup/
$ tiup mirror merge ../tidb-community-toolkit-v6.1.0-linux-amd64
-- 升级cluster组件
$ tiup update cluster
Updated successfully!
升级TiDB集群
-- 编辑 TiUP Cluster 拓扑配置文件（测试环境没有修改过参数，所以升级测试时就没有对配置文件做任何修改）
$ tiup cluster edit-config tidb-test
tiup is checking updates for component cluster ...
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.10.2/tiup-cluster edit-config tidb-test
The file has nothing changed
-- 检查当前集群的健康状况
$ tiup cluster check tidb-test --cluster
tiup is checking updates for component cluster ...
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.10.2/tiup-cluster check tidb-test --cluster
+ Download necessary tools
- Downloading check tools for linux/amd64 ... Done
...省略中间日志
Checking region status of the cluster tidb-test...
All regions are healthy.
-- 输出结果最后一行‘All regions are healthy.’，表示集群状态正常，可以进行后续升级操作。
-- 使用tiup cluster upgrade 命令将集群升级到指定版本
$ tiup cluster upgrade tidb-test v6.1.0
tiup is checking updates for component cluster ...
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.10.2/tiup-cluster upgrade tidb-test v6.1.0
This operation will upgrade tidb v5.3.0 cluster tidb-test to v6.1.0.
Do you want to continue? [y/N]:(default=N) y
Upgrading cluster...
...省略中间日志
Upgraded cluster `tidb-test` successfully
-- 输出结果最后一行‘Upgraded cluster `集群名称` successfully’，表示集群升级正常完成。
-- 查看集群状态：
$ tiup cluster display tidb-test
tiup is checking updates for component cluster ...
Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.10.2/tiup-cluster display tidb-test
Cluster type: tidb
Cluster name: tidb-test
Cluster version: v6.1.0
Deploy user: tidb
SSH type: builtin
Dashboard URL: http://192.168.56.11:2379/dashboard
Grafana URL: http://192.168.56.10:3000
ID Role Host Ports OS/Arch Status Data Dir Deploy Dir
-- ---- ---- ----- ------- ------ -------- ----------
192.168.56.10:9093 alertmanager 192.168.56.10 9093/9094 linux/x86_64 Up /u02/tidb-data/alertmanager-9093 /u02/tidb-deploy/alertmanager-9093
192.168.56.10:3000 grafana 192.168.56.10 3000 linux/x86_64 Up - /u02/tidb-deploy/grafana-3000
192.168.56.10:2379 pd 192.168.56.10 2379/2380 linux/x86_64 Up /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.11:2379 pd 192.168.56.11 2379/2380 linux/x86_64 Up|L|UI /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.12:2379 pd 192.168.56.12 2379/2380 linux/x86_64 Up /u02/tidb-data/pd-2379 /u02/tidb-deploy/pd-2379
192.168.56.10:9090 prometheus 192.168.56.10 9090/12020 linux/x86_64 Up /u02/tidb-data/prometheus-9090 /u02/tidb-deploy/prometheus-9090
192.168.56.10:4000 tidb 192.168.56.10 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.11:4000 tidb 192.168.56.11 4000/10080 linux/x86_64 Up - /u02/tidb-deploy/tidb-4000
192.168.56.10:20160 tikv 192.168.56.10 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.11:20160 tikv 192.168.56.11 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.12:20160 tikv 192.168.56.12 20160/20180 linux/x86_64 Up /u02/tidb-data/tikv-20160 /u02/tidb-deploy/tikv-20160
192.168.56.150:7077 tispark-master 192.168.56.150 7077/8080 linux/x86_64 Up - /u02/tidb-deploy/tispark-master-7077
192.168.56.12:7078 tispark-worker 192.168.56.12 7078/8081 linux/x86_64 Up - /u02/tidb-deploy/tispark-worker-7078
Total nodes: 13
-- 集群状态正常，成功升级到6.1.0
升级总结
由于是测试环境，升级前没有对数据库参数做过调整。所以本文章中没有比对新旧版本数据库参数的过程。
但是正式环境，升级前一定要确认旧版本到新版本之间所有数据库参数的变化，是否会对升级后数据库的使用产生不良影响。
参考官方文档的相关版本的ReleaseNote，比如6.1版本：
TiDB 6.1.0 Release Notes | PingCAP Docs
个人觉得，如果官方可以给提供个，可以在升级操作前比对升级前后数据库参数变化的脚本，就会方便很多。