如何处理损坏的sst文件
作者：h5n1
概述
TiDB在运行过程中可能由于文件系统或操作系统等原因出现sst文件损坏情况，从tidb 5.2版本开始支持tikv-ctl bad-ssts命令用于处理损坏的sst文件，官方文档中已经给出了比较详细的解释，但在按照官方文档操作时会有些问题。
本文记录通过dd破坏sst文件模拟损坏后使用tikv-ctl bad-ssts处理过程，本次测试数据库版本为5.3.3-ARM平台。
操作过程
1、 初始化一张1000万的数据表，随意查看该表上的region，本次模拟破坏的是store 12上的sst文件。
$ pd-ctl region 348
{
"id": 348,
"start_key": "7480000000000000FF3B5F728000000000FF8A88290000000000FA",
"end_key": "7480000000000000FF3B5F728000000000FF90D3980000000000FA",
"epoch": {
"conf_ver": 59,
"version": 51
},
"peers": [
{
"id": 350,
"store_id": 11,
"role_name": "Voter"
},
{
"id": 2532,
"store_id": 5,
"role_name": "Voter"
},
{
"id": 14052,
"store_id": 12,
"role_name": "Voter"
}
],
"leader": {
"id": 350,
"store_id": 11,
"role_name": "Voter"
},
"written_bytes": 156,
"read_bytes": 0,
"written_keys": 2,
"read_keys": 0,
"approximate_size": 114,
"approximate_keys": 821741
}
2、 使用region-properties 查找该region的sst文件，并使用dd 命令进行破坏
$ ./tikv-ctl --host xx.xxx.144.16:20160 region-properties -r 348
[2022/10/13 21:39:37.318 +08:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/10/13 21:39:37.319 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0xfffb600d0120 for subchannel 0xfffc200d2bc0"]
mvcc.min_ts: 436638484783693826
mvcc.max_ts: 436638579797524498
mvcc.num_rows: 410870
mvcc.num_puts: 410870
mvcc.num_deletes: 0
mvcc.num_versions: 821741
mvcc.max_row_versions: 3
num_entries: 821741
num_deletes: 0
num_files: 1
sst_files: 000417.sst
region.start_key: 7480000000000000ff3b5f728000000000ff8a88290000000000fa
region.end_key: 7480000000000000ff3b5f728000000000ff90d3980000000000fa
region.middle_key_by_approximate_size: 7480000000000000ff3b5f728000000000ff8dd4ba0000000000faf9f0bf9c25a7fffe
$ ls -l /data/v5.0.3/tikv/data/db/000417.sst
-rw-r--r-- 1 tidb tidb 41907560 Oct 13 21:29 /data/v5.0.3/tikv/data/db/000417.sst
[tidb@tgypt-xx13d002-cs76w v5.3.3]$ dd if=/dev/zero of=/data/v5.0.3/tikv/data/db/000417.sst bs=1M count=42
42+0 records in
42+0 records out
44040192 bytes (44 MB, 42 MiB) copied, 0.0284878 s, 1.5 GB/s
[tidb@tgypt-xx13d002-cs76w v5.3.3]$ ls -l /data/v5.0.3/tikv/data/db/000417.sst
-rw-r--r-- 1 tidb tidb 44040192 Oct 13 21:42 /data/v5.0.3/tikv/data/db/000417.sst
3、 破坏sst文件后tikv并没有立即崩溃，重启tikv后tikv.log报错如下，无法启动：
[2022/10/13 21:47:34.605 +08:00] [ERROR] [server.rs:1056] ["failed to init io snooper"] [err_code=KV:Unknown] [err="\"IO snooper is not started due to not compiling with BCC\""]
[2022/10/13 21:47:34.624 +08:00] [FATAL] [server.rs:1281] ["failed to create kv engine: Storage Engine Corruption: Sst file size mismatch: /data/v5.0.3/tikv/data/db/000417.sst. Size recorded in manifest 41907560, actual size 44040192\n"]
4、根据官方文档提供的命令使用bad-ssts检测有问题的sst文件，尝试使用--db 指定db目录 和--data-dir指定data-dir目录均报参数错误，于是使用了5.3.1的tikv-ctl + --db参数后执行正常(之所以使用5.3.1是因为之前使用过该版本命令)。
$ ./tikv-ctl bad-ssts --db /data/v5.0.3/tikv/data/db --pd 10.125.144.17:2379
error: Found argument '--db' which wasn't expected, or isn't valid in this context
$ /data/soft/v5.3.1/tikv-ctl bad-ssts --db /data/v5.0.3/tikv/data/db --pd 10.125.144.17:2379
[2022/10/13 22:07:44.853 +08:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=10.125.144.17:2379]
[2022/10/13 22:07:44.854 +08:00] [INFO] [<unknown>] ["TCP_USER_TIMEOUT is available. TCP_USER_TIMEOUT will be used thereafter"]
[2022/10/13 22:07:44.854 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0xfffba02701b0 for subchannel 0xfffc200d2bc0"]
[2022/10/13 22:07:44.855 +08:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://10.125.144.17:2379]
[2022/10/13 22:07:44.855 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0xfffba02702d0 for subchannel 0xfffc200d2a00"]
[2022/10/13 22:07:44.856 +08:00] [INFO] [util.rs:544] ["connecting to PD endpoint"] [endpoints=http://10.125.144.18:2379]
[2022/10/13 22:07:44.856 +08:00] [INFO] [<unknown>] ["New connected subchannel at 0xfffba02703f0 for subchannel 0xfffc200d2bc0"]
[2022/10/13 22:07:44.856 +08:00] [INFO] [util.rs:668] ["connected to PD member"] [endpoints=http://10.125.144.18:2379]
[2022/10/13 22:07:44.856 +08:00] [INFO] [util.rs:536] ["all PD endpoints are consistent"] [endpoints="[\"10.125.144.17:2379\"]"]
--------------------------------------------------------
corruption info:
/data/v5.0.3/tikv/data/db/000417.sst: Corruption: Bad table magic number: expected 9863518390377041911, found 0 in /data/v5.0.3/tikv/data/db/000417.sst
sst meta:
417:41907560[0 .. 0]['7A7480000000000000FF3B5F728000000000FF8A88290000000000FAF9F0BF866BEFFFEE' seq:0, type:1 .. '7A7480000000000000FF3B5F728000000000FF90D3970000000000FAF9F0BF9BCB07FFFD' seq:0, type:1] at level 6 for Column family "write" (ID 2)
overlap region:
RegionInfo { region: id: 348 start_key: 7480000000000000FF3B5F728000000000FF8A88290000000000FA end_key: 7480000000000000FF3B5F728000000000FF90D3980000000000FA region_epoch { conf_ver: 59 version: 51 } peers { id: 350 store_id: 11 } peers { id: 2532 store_id: 5 } peers { id: 14052 store_id: 12 }, leader: Some(id: 350 store_id: 11) }
suggested operations:
tikv-ctl ldb --db=/data/v5.0.3/tikv/data/db unsafe_remove_sst_file "/data/v5.0.3/tikv/data/db/000417.sst"
tikv-ctl --db=/data/v5.0.3/tikv/data/db tombstone -r 348 --pd <endpoint>
--------------------------------------------------------
corruption analysis has completed
上述检查数的详细信息参加官方文档说明
5、 按照上述输出的建议命令执行tikv-ctl ldb --db=/data/v5.0.3/tikv/data/db unsafe_remove_sst_file "/data/v5.0.3/tikv/data/db/000417.sst"
报错：Failed: Failed to parse SST file number /data/v5.0.3/tikv/data/db/000417.sst 。分析该错误提示 不能解析 SST file number ，看起来这里应该指定sst的文件号。而前面bad-sst检查时的输出Meta信息中包含sst文件号，和sst文件名中一致。
$ /data/soft/v5.3.1/tikv-ctl ldb --db=/data/v5.0.3/tikv/data/db unsafe_remove_sst_file "/data/v5.0.3/tikv/data/db/000417.sst"
[2022/10/13 22:09:21.574 +08:00] [INFO] [mod.rs:118] ["encryption: none of key dictionary and file dictionary are found."]
[2022/10/13 22:09:21.574 +08:00] [INFO] [mod.rs:479] ["encryption is disabled."]
Failed: Failed to parse SST file number /data/v5.0.3/tikv/data/db/000417.sst
6、 使用 sst文件号执行成功
$ /data/soft/v5.3.1/tikv-ctl ldb --db=/data/v5.0.3/tikv/data/db unsafe_remove_sst_file 417
[2022/10/13 22:09:44.747 +08:00] [INFO] [mod.rs:118] ["encryption: none of key dictionary and file dictionary are found."]
[2022/10/13 22:09:44.748 +08:00] [INFO] [mod.rs:479] ["encryption is disabled."]
unsafely removed SST file
再次执行该命令提示已经在rockdb中找不到该文件
Failed: failed to unsafely remove SST file: NotFound: File not present in any level
7、 执行 region tombstone 命令从有问题的tikv上删除sst文件的region peer，该命令同样在--data-dir/--db/--pd参数使用上报错，最后使用--force处理成功
$ /data/soft/v5.3.1/tikv-ctl --data-dir /data/v5.0.3/tikv/data tombstone -r 348 --force
[2022/10/13 22:14:19.246 +08:00] [INFO] [mod.rs:118] ["encryption: none of key dictionary and file dictionary are found."]
[2022/10/13 22:14:19.246 +08:00] [INFO] [mod.rs:479] ["encryption is disabled."]
[2022/10/13 22:14:19.268 +08:00] [WARN] [config.rs:587] ["compaction guard is disabled due to region info provider not available"]
[2022/10/13 22:14:19.268 +08:00] [WARN] [config.rs:682] ["compaction guard is disabled due to region info provider not available"]
success!
8、再次检查region，发下tidb中已经删除store 12上的peer 而在store 上补了副本
$ pd-ctl region 348
{
"id": 348,
"start_key": "7480000000000000FF3B5F728000000000FF8A88290000000000FA",
"end_key": "7480000000000000FF3B5F728000000000FF90D3980000000000FA",
"epoch": {
"conf_ver": 65,
"version": 51
},
"peers": [
{
"id": 350,
"store_id": 11,
"role_name": "Voter"
},
{
"id": 2532,
"store_id": 5,
"role_name": "Voter"
},
{
"id": 17637,
"store_id": 2,
"role_name": "Voter"
}
],
"leader": {
"id": 350,
"store_id": 11,
"role_name": "Voter"
},
"written_bytes": 157,
"read_bytes": 0,
"written_keys": 2,
"read_keys": 0,
"approximate_size": 114,
"approximate_keys": 821741
}
9、重新启动tikv，tikv能正常启动，检查表中数据无丢失
3 总结
使用tikv-ctl bad-ssts 处理损坏的sst文件在官方文档已经有说明，但命令参数、输出结果上有些错误，导致按照文档操作有些问题，建议官方进行一下修改。