×
技术社区 >  技术博客 >  【OceanBase 排查案例】insert all 语句引发租户内存告急?速看解决方案!

【OceanBase 排查案例】insert all 语句引发租户内存告急?速看解决方案!

在 OceanBase 数据库的日常运维中,内存占用异常是极易引发业务故障的高频问题。近期我们遇到一例典型场景:某业务执行 insert all 语句时频繁报内存超限错误,租户 SqlExecutor 模块内存占用飙升至近 200G,直接触发 OCP 告警。今天就带大家拆解问题根源,给出落地可行的解决方案!

问题现象

当insert all into tab语句中存在大量的into tab values子句,执行SQL报错ERROR 4013 (HY001): No memory or reach tenant memory limit,ocp告警中出现OBServer内存分配出错的告警信息。

告警详情:[OBServer 内存分配出错] 集群:yx_arm_clu01,主机:10.xx.xx.52,日志类型:observer,日志文件:/home/admin/oceanbase/log/observer.log,日志级别:WARN,关键字=No memory or reach tenant memory limit.*\[OOPS\]=,错误码=4013,日志详情=[2026-02-12 13:12:15.495254] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [2490563][pnio1][T0][YB420A50A235-00062BA62D7FC190-0-0] [lt=6][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=tenant memory has reached the upper limit(tenant_id: 1016, tenant_hold: 247389028352, tenant_limit: 247390116250, alloc_size: 2097152))。

OCP租户 SQL诊断 TOP SQL如下

SQL示例语句如下:

insert all    into tab (col1, col2) values ('a', 1)    into tab (col1, col2) values ('b', 2)    into tab (col1, col2) values ('c', 3) select * from dual;

OBServer日志内存信息如下,DEFAULT_CTX_ID的内存占用高达近200G。

[2026-02-12 11:46:18.507119] INFO  [LIB] operator() (ob_malloc_allocator.cpp:567) [2490491][MemDumpTimer][T1016][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant: 1016, limit: 247,390,116,250 hold: 231,218,413,568 rpc_hold: 0 cache_hold: 178,257,920 cache_used: 178,257,920 cache_item_count: 85 [MEMORY] ctx_id=           DEFAULT_CTX_ID hold_bytes=208,756,539,392 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=          MEMSTORE_CTX_ID hold_bytes=  1,679,818,752 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=         TRANS_CTX_MGR_ID hold_bytes=      4,194,304 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=        PLAN_CACHE_CTX_ID hold_bytes= 18,944,884,736 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=                WORK_AREA hold_bytes=    100,663,296 limit=            49,478,023,240[MEMORY] ctx_id=                    GLIBC hold_bytes=    153,092,096 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=                 CO_STACK hold_bytes=    603,979,776 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=          META_OBJ_CTX_ID hold_bytes=    161,480,704 limit=            49,478,023,240[MEMORY] ctx_id=       TX_CALLBACK_CTX_ID hold_bytes=    538,968,064 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=               LOB_CTX_ID hold_bytes=      2,097,152 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=          PS_CACHE_CTX_ID hold_bytes=     12,582,912 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=               RPC_CTX_ID hold_bytes=      2,097,152 limit=           247,390,116,200[MEMORY] ctx_id=            TX_DATA_TABLE hold_bytes=     71,368,704 limit= 9,223,372,036,854,775,807[MEMORY] ctx_id=              MDS_DATA_ID hold_bytes=      8,388,608 limit= 9,223,372,036,854,775,807[MEMORY][PM] tid=    593265 used=              0 hold=      4,194,304 pm=0xffbfff30f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593266 used=              0 hold=      4,194,304 pm=0xffbfff39f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593269 used=              0 hold=      2,097,152 pm=0xffbfff19f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593274 used=              0 hold=      4,194,304 pm=0xffbffed0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593275 used=              0 hold=      4,194,304 pm=0xffbffed9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593281 used=              0 hold=      4,194,304 pm=0xffbffe99f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593280 used=              0 hold=      4,194,304 pm=0xffbffe90f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593283 used=              0 hold=      4,194,304 pm=0xffbffe70f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593285 used=              0 hold=      4,194,304 pm=0xffbffe46f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593286 used=              0 hold=      4,194,304 pm=0xffbffe50f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593288 used=              0 hold=      4,194,304 pm=0xffbffe26f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593289 used=              0 hold=      4,194,304 pm=0xffbffe30f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593290 used=              0 hold=      4,194,304 pm=0xffbffe39f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593291 used=              0 hold=      4,194,304 pm=0xffbffe06f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593292 used=              0 hold=      4,194,304 pm=0xffbffe10f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593294 used=              0 hold=      4,194,304 pm=0xffbffde6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593296 used=              0 hold=      4,194,304 pm=0xffbffdf9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593297 used=              0 hold=      4,194,304 pm=0xffbffdc6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593298 used=              0 hold=      4,194,304 pm=0xffbffdd0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593301 used=              0 hold=      4,194,304 pm=0xffbffdb0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593302 used= 18,780,424,224 hold= 18,935,578,624 pm=0xffbffdb9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593305 used=              0 hold=      4,194,304 pm=0xffbffd99f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593309 used=              0 hold=      4,194,304 pm=0xffbffd46f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593310 used=              0 hold=      4,194,304 pm=0xffbffd50f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593314 used=              0 hold=      4,194,304 pm=0xffbffd39f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593318 used=              0 hold=      4,194,304 pm=0xffbffce6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593319 used=              0 hold=      4,194,304 pm=0xffbffcf0f000 ctx_name=DEFAULT_CTX_ID:[MEMORY][PM] tid=    593292 used=              0 hold=      4,194,304 pm=0xffbffe10f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593294 used=              0 hold=      4,194,304 pm=0xffbffde6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593296 used=              0 hold=      4,194,304 pm=0xffbffdf9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593297 used=              0 hold=      4,194,304 pm=0xffbffdc6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593298 used=              0 hold=      4,194,304 pm=0xffbffdd0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593301 used=              0 hold=      4,194,304 pm=0xffbffdb0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593302 used= 18,780,424,224 hold= 18,935,578,624 pm=0xffbffdb9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593305 used=              0 hold=      4,194,304 pm=0xffbffd99f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593309 used=              0 hold=      4,194,304 pm=0xffbffd46f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593310 used=              0 hold=      4,194,304 pm=0xffbffd50f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593314 used=              0 hold=      4,194,304 pm=0xffbffd39f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593318 used=              0 hold=      4,194,304 pm=0xffbffce6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593319 used=              0 hold=      4,194,304 pm=0xffbffcf0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593320 used=              0 hold=      4,194,304 pm=0xffbffcf9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593322 used=              0 hold=      4,194,304 pm=0xffbffcd0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593326 used=              0 hold=      4,194,304 pm=0xffbffcb9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593329 used=              0 hold=      4,194,304 pm=0xffbffc99f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593331 used=              0 hold=      4,194,304 pm=0xffbffc70f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593332 used=              0 hold=      4,194,304 pm=0xffbffc79f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593334 used=              0 hold=      4,194,304 pm=0xffbffc50f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593335 used=              0 hold=      4,194,304 pm=0xffbffc59f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593338 used=              0 hold=      4,194,304 pm=0xffbffc39f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593342 used=              0 hold=      4,194,304 pm=0xffbffbe6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593344 used=              0 hold=      4,194,304 pm=0xffbffbf9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593345 used=              0 hold=      4,194,304 pm=0xffbffbc6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593349 used=              0 hold=      4,194,304 pm=0xffbffbb0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593351 used=              0 hold=      4,194,304 pm=0xffbffb86f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593355 used= 95,993,336,496 hold= 96,785,399,808 pm=0xffbffb70f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593357 used= 37,978,587,072 hold= 38,292,488,192 pm=0xffbffb46f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593361 used=              0 hold=      4,194,304 pm=0xffbffb30f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593362 used= 25,274,339,360 hold= 25,483,083,776 pm=0xffbffb39f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593364 used=              0 hold=      4,194,304 pm=0xffbffb10f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593366 used=              0 hold=      4,194,304 pm=0xffbffae6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593367 used=              0 hold=      4,194,304 pm=0xffbffaf0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593368 used=              0 hold=      4,194,304 pm=0xffbffaf9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593369 used=              0 hold=      4,194,304 pm=0xffbffac6f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593371 used=              0 hold=      4,194,304 pm=0xffbffad9f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593373 used=              0 hold=      4,194,304 pm=0xffbffab0f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593375 used=              0 hold=      4,194,304 pm=0xffbffa86f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593380 used=              0 hold=      4,194,304 pm=0xffbffa79f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593383 used=              0 hold=      4,194,304 pm=0xffbffa59f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593384 used= 14,166,628,288 hold= 14,283,374,592 pm=0xffbffa26f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593385 used=              0 hold=      4,194,304 pm=0xffbffa30f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593386 used=      8,319,744 hold=      8,388,608 pm=0xffbffa39f000 ctx_name=DEFAULT_CTX_ID[MEMORY][PM] tid=    593391 used=              0 hold=      4,194,304 pm=0xffbff9f0f000 ctx_name=DEFAULT_CTX_ID

集群sys租户内存性能视图GV$OB_MEMORY结果如下

MySQL [oceanbase]> select * from GV$OB_MEMORY  where tenant_id=1016 order by used;+-----------+--------------+----------+-------------------------------+-----------------+---------+--------------+--------------+| TENANT_ID | SVR_IP       | SVR_PORT | CTX_NAME                      | MOD_NAME        | COUNT   | HOLD         | USED         |+-----------+--------------+----------+-------------------------------+-----------------+---------+--------------+--------------+|      1016 | 10.xx.xx.52 |     2882 | UNEXPECTED_IN_500             | KvstorCacheMb   |       0 |            0 |            0 ||      1016 | 10.xx.xx.52 |     2882 | TX_CALLBACK_CTX_ID            | KvstorCacheMb   |       0 |            0 |            0 ||      1016 | 10.xx.xx.52 |     2882 | TX_CALLBACK_CTX_ID            | KvstorCacheMb   |       0 |            0 |            0 |...|      1016 | 10.xx.xx.52 |     2882 | MEMSTORE_CTX_ID               | Memstore        |     516 |   1073676288 |   1073185056 ||      1016 | 10.xx.xx.52 |     2882 | PLAN_CACHE_CTX_ID             | SqlPhyPlan      |  305793 |   6853178816 |   6360646353 ||      1016 | 10.xx.xx.52 |     2882 | MEMSTORE_CTX_ID               | Memstore        |    4879 |  10152067072 |  10147422264 ||      1016 | 10.xx.xx.52 |     2882 | DEFAULT_CTX_ID                | MysqlRequesReco |    4895 |  10261233664 |  10256201088 ||      1016 | 10.xx.xx.52 |     2882 | MEMSTORE_CTX_ID               | Memstore        |    5122 |  10657693696 |  10652817552 ||      1016 | 10.xx.xx.52 |     2882 | DEFAULT_CTX_ID                | MysqlRequesReco |    5226 |  10949967872 |  10944596352 ||      1016 | 10.xx.xx.52 |     2882 | DEFAULT_CTX_ID                | MysqlRequesReco |    5273 |  11047763968 |  11042344320 ||      1016 | 10.xx.xx.52 |     2882 | PLAN_CACHE_CTX_ID             | SqlPhyPlan      | 1010678 |  17699686016 |  16976458135 ||      1016 | 10.xx.xx.52 |     2882 | PLAN_CACHE_CTX_ID             | SqlPhyPlObj     |  815592 |  19348716992 |  19077125294 ||      1016 | 10.xx.xx.52 |     2882 | DEFAULT_CTX_ID                | SqlExecutor     |   94760 | 199055097856 | 198953370500 |+-----------+--------------+----------+-------------------------------+-----------------+---------+--------------+--------------+878 rows in set (0.010 sec)

SQL执行时发现SqlExecutor模块的内存出现膨胀,随着会话连接的关闭,内存也会进行释放。

gv$ob_sql_audit中SQL执行耗时GET_PLAN_TIME的值较大,接近ELAPSED_TIME的时间,REQUEST_MEMORY_USED的内存值达到了90G。

select * from gv$ob_sql_audit where sql_id in ('66B25AF9022FD543841E0CF9A1934B53','CC8F9BD89C362010C9A1D07E4DB54A82') limit 10\G;*************************** 3. row ***************************                         SVR_IP: 10.xx.xx.52                       SVR_PORT: 2882                     REQUEST_ID: 11626260613                    SQL_EXEC_ID: 18113079752                       TRACE_ID: YB420A50A234-00062AD3381FAF22-0-0                            SID: 3221811103                      CLIENT_IP: 10.xx.xx.52                    CLIENT_PORT: 22156                      TENANT_ID: 1016                    TENANT_NAME: xxx_t02            EFFECTIVE_TENANT_ID: 1016                        USER_ID: 515777                      USER_NAME: xxx_user                     USER_GROUP: 0                 USER_CLIENT_IP: 192.xx.xx.23                          DB_ID: 515778                        DB_NAME: xxx_DBO                         SQL_ID: 66B25AF9022FD543841E0CF9A1934B53                      QUERY_SQL: insert all                  into TBL_SCM_FLOW_TOUCH_QUOTA (LIST_ID,ONE_ID,NODE_ID,ACTIVITY_CODE) values...into TBL_SCM_FLOW_TOUCH_QUOTA (LIST_ID,ONE_ID,NODE_ID,ACTIVITY_CODE) values        ( 7592334943,  'cabee4de-3c3b-412b-a73f-26b1486ba805', '23612587c9ae441895077e67c8d43954', '52b701b93                        PLAN_ID: 0                  AFFECTED_ROWS: 0                    RETURN_ROWS: 0                  PARTITION_CNT: 0                       RET_CODE: -5066                          QC_ID: 0                         DFO_ID: 0                         SQC_ID: 0                      WORKER_ID: 0                          EVENT:                          P1TEXT:                              P1: 0                         P2TEXT:                              P2: 0                         P3TEXT:                              P3: 0                          LEVEL: 0                  WAIT_CLASS_ID: 100                    WAIT_CLASS#: 0                     WAIT_CLASS: OTHER                          STATE: MAX_WAIT TIME ZERO                WAIT_TIME_MICRO: 0          TOTAL_WAIT_TIME_MICRO: 0                    TOTAL_WAITS: 0                      RPC_COUNT: 0                      PLAN_TYPE: 0                   IS_INNER_SQL: 0                IS_EXECUTOR_RPC: 0                    IS_HIT_PLAN: 0                   REQUEST_TIME: 1770826483379082                   ELAPSED_TIME: 48045733798                       NET_TIME: 0                  NET_WAIT_TIME: 8                     QUEUE_TIME: 61                    DECODE_TIME: 1                  GET_PLAN_TIME: 48045713897                   EXECUTE_TIME: 76          APPLICATION_WAIT_TIME: 0          CONCURRENCY_WAIT_TIME: 0              USER_IO_WAIT_TIME: 0                  SCHEDULE_TIME: 0                  ROW_CACHE_HIT: 0         BLOOM_FILTER_CACHE_HIT: 0                BLOCK_CACHE_HIT: 0                     DISK_READS: 0                      RETRY_CNT: 0                     TABLE_SCAN: 0              CONSISTENCY_LEVEL: -1        MEMSTORE_READ_ROW_COUNT: 0         SSSTORE_READ_ROW_COUNT: 0            DATA_BLOCK_READ_CNT: 0           DATA_BLOCK_CACHE_HIT: 0           INDEX_BLOCK_READ_CNT: 0          INDEX_BLOCK_CACHE_HIT: 0            BLOCKSCAN_BLOCK_CNT: 0              BLOCKSCAN_ROW_CNT: 0PUSHDOWN_STORAGE_FILTER_ROW_CNT: 0            REQUEST_MEMORY_USED: 98880409680          EXPECTED_WORKER_COUNT: 0              USED_WORKER_COUNT: 0                     SCHED_INFO: NULL             FUSE_ROW_CACHE_HIT: 0              PS_CLIENT_STMT_ID: -1               PS_INNER_STMT_ID: -1                          TX_ID: 0               SNAPSHOT_VERSION: 0                   REQUEST_TYPE: 2          IS_BATCHED_MULTI_STMT: 0                  OB_TRACE_INFO: NULL                      PLAN_HASH: 0             LOCK_FOR_READ_TIME: 0                   PARAMS_VALUE:                       RULE_NAME:                   PARTITION_HIT: 1            TX_INTERNAL_ROUTING: 0               TX_STATE_VERSION: 0                   FLT_TRACE_ID:               NETWORK_WAIT_TIME: 03 rows in set (58.455 sec)ERROR: No query specified

问题原因

insert all into values 类型 SQL 后跟大量多行数据时,在硬解析时,内存膨胀,另外行数不固定,可能导致不能命中 plan_cache,出现并发硬解析时,同时占用内存就很大,导致租户内存被挤占。

规避方案

  1. 改写为insert into values 多行,控制500-1000条提交一次,避免一次插入条数调大。

影响版本

数据库版本:OceanBase 4.2.1.10

租户模式:OB_Oracle

参考资料

执行内存占用高问题汇总-OceanBase数据库使用指南

如果你的业务也遇到 OceanBase 内存异常、SQL 执行效率低等问题,欢迎在评论区留言交流,我们会持续分享更多实战排查案例!

精选推荐