生产环境突发单节点磁盘使用率持续飙升是运维高频故障，本次基于 OceanBase 4.2.1 真实故障案例，拆解由大 SQL 中间结果落地临时文件引发磁盘暴涨近 1TB 完整排查思路、定位命令、处置方案与版本优化方案，快速收藏备用。
问题现象
集群单节点磁盘使用率上升!
环境版本：OceanBase V4.2.1
关键诊断信息
触发条件
大SQL中间结果落盘
事后诊断
对于临时文件可以使用下面的方法，确认临时文件宏块是否发生了空间放大。
● V4.1.x 及之前版本
grep 'ob_tmp_file*' observer.log.2023031022* | grep 'succeed to wash a block' | grep -o 'free_page_nums:[0-9]*' | awk -F ':' '{ sum += $2; } END { print "total_free_page_num = " sum; print "avg_free_page_nums = " sum/NR; print "enlargement_factor = " (252*NR)/(252*NR - sum); print "macro_block_count = " NR }'
● V4.2.x 版本
grep 'ob_tmp_file*' observer.log.2023031022* | grep 'succeed to wash a block' | grep -o 'free_page_nums=[0-9]*' | awk -F '=' '{ sum += $2; } END { print "total_free_page_num = " sum; print "avg_free_page_nums = " sum/NR; print "enlargement_factor = " (252*NR)/(252*NR - sum); print "macro_block_count = " NR }'
● V4.3.x 及 V4.4.0 版本
grep 'block usage for temporary files' observer.log
问题版本为4.2.1版本，使用对应语句可以看到临时文件占用的宏块数量 562719*2M约1.07T，和上涨的大小大致符合
$grep 'ob_tmp_file*' observer.log.20250821* | grep 'succeed to wash a block' | grep -o 'free_page_nums=[0-9]*' | awk -F '=' '{ sum += $2; } END { print "total_free_page_num = " sum; print "avg_free_page_nums = " sum/NR; print "enlargement_factor = " (252*NR)/(252*NR - sum); print "macro_block_count = " NR }'
total_free_page_num = 15732112
avg_free_page_nums = 27.9573
enlargement_factor = 1.12479
macro_block_count = 562719
通过succeed to open a tmp file关键字过滤日志如下， dir_id和fd为写临时文件指定目录和文件 id
$grep "succeed to open a tmp file" observer.log.*
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51306][T1034_L0_G0][T1034][YB420A6F3001-000631F0C3EF1616-0-0] [lt=14] succeed to open a tmp file(fd=101389, dir=524536455, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfedda2 0xcfed36c 0xcfef9c8 0xe20463d 0xe1feba6 0xe1fde82 0xacfe19b 0xad04a1f 0xad08116 0x555ddf7 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [48675][T1034_L0_G0][T1034][YB420A6F3001-000631F0B09F4B5B-0-0] [lt=13] succeed to open a tmp file(fd=101390, dir=524537270, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfedda2 0xcfed36c 0xcfef9c8 0xe20463d 0xe1feba6 0xe1fde82 0xacfe19b 0xad04a1f 0xad08116 0x555ddf7 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51306][T1034_L0_G0][T1034][YB420A6F3001-000631F0C3EF1634-0-0] [lt=27] succeed to open a tmp file(fd=101391, dir=524547193, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51306][T1034_L0_G0][T1034][YB420A6F3001-000631F0C3EF164B-0-0] [lt=29] succeed to open a tmp file(fd=101392, dir=524551743, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfedda2 0xcfed36c 0xcfef9c8 0xe20463d 0xe1feba6 0xe1fde82 0xacfe19b 0xad04a1f 0xad08116 0x555ddf7 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=21] succeed to open a tmp file(fd=101393, dir=524676377, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101394, dir=524676391, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101395, dir=524676394, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=4] succeed to open a tmp file(fd=101396, dir=524676456, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101397, dir=524676461, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101398, dir=524676483, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=4] succeed to open a tmp file(fd=101399, dir=524676488, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=12] succeed to open a tmp file(fd=101400, dir=524676495, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=12] succeed to open a tmp file(fd=101401, dir=524676504, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101402, dir=524676508, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=4] succeed to open a tmp file(fd=101403, dir=524676539, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd400d85 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101404, dir=524676543, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101405, dir=524676545, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd400d85 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101406, dir=524676548, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=4] succeed to open a tmp file(fd=101407, dir=524676562, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=5] succeed to open a tmp file(fd=101408, dir=524676621, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51303][T1034_L0_G0][T1034][YB420A6F3001-000631E1E4AEBD31-0-0] [lt=6] succeed to open a tmp file(fd=101409, dir=524676654, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0x556da20 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [51411][T1034_L0_G0][T1034][YB420A6F3001-000631F0A5DF1B78-0-0] [lt=11] succeed to open a tmp file(fd=101410, dir=524682405, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfedda2 0xcfed36c 0xcfef9c8 0xe20463d 0xe1feba6 0xe1fde82 0xacfe19b 0xad04a1f 0xad08116 0x555ddf7 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=20] succeed to open a tmp file(fd=101411, dir=524790249, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101412, dir=524790255, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=4] succeed to open a tmp file(fd=101413, dir=524790257, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101414, dir=524790262, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=6] succeed to open a tmp file(fd=101415, dir=524790280, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=4] succeed to open a tmp file(fd=101416, dir=524790285, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101417, dir=524790290, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101418, dir=524790292, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101419, dir=524790299, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd400d85 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101420, dir=524790306, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101421, dir=524790315, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101422, dir=524790317, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101423, dir=524790324, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101424, dir=524790362, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=4] succeed to open a tmp file(fd=101425, dir=524790368, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
INFO [STORAGE] open (ob_tmp_file.cpp:1393) [1133][T1034_L0_G100][T1034][YB420A6F3001-000631F06FAF8656-0-0] [lt=5] succeed to open a tmp file(fd=101426, dir=524790376, common::lbt()="0x13980025 0xfb170bf 0xfb163e3 0xcfe3707 0xcfde430 0xcfdf0bf 0xd401696 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd41235f 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0xd3a3a02 0x5ac0eba 0x5a54e19 0xd3a9eaf 0x55b90ae 0x57208c9 0x56f2652 0x55b905c 0xd2d55dd 0x55b90ae 0x55b3b7d 0x55afdb7 0x55aef1f 0xace331a 0x55701a9 0x5566296 0x55591ca 0x554223c 0xa98e814 0x13c3e48f 0x7fe2ca111dc5 0x7fe2c9e3eced")
查看对应租户的top1 sql执行详情，执行时间也与监控上涨时段符合，基本确定问题SQL。
问题原因
临时文件导致磁盘占用1T是SQL产生的中间结果落盘导致，磁盘碎片率大约10%，是符合预期的磁盘占用；如果需要减少磁盘占用，在43x版本上有支持中间结果压缩，或425版本有支持限制临时文件磁盘占用总量。
问题的风险及影响
客户磁盘使用率上升。
解决方法
临时文件导致的磁盘占用升高，找到产生临时文件的SQL并kill掉之后，文件会被释放。
