[Bug] be crash when stream load ,and jemalloc_cache mem use 800g

[Toms1999]

Search before asking

• I had searched in the issues and found no similar issues.

Version

2.1.6

What's Wrong?

time=2024-09-24 20:34:29 level=error msg=Load task on 10.72.120.222:/home/fdw/xuesong_temp/gpfdist_dir/ext_rimbi_dw_ods_ods_ris_event_dd_1_prt_m202101_2024924_202014/data_file_25 failed! Errmsg: File: /home/fdw/xuesong_temp/gpfdist_dir/ext_rimbi_dw_ods_ods_ris_event_dd_1_prt_m202101_2024924_202014/data_file_25, Size: 21121031305, Label: 07fae7cf-f296-40a7-b812-1dccb717debc, Status: Fail, Message: [INTERNAL_ERROR]cancelled: PreCatch error code:11, [E11] Allocator sys memory check failed: Cannot alloc:268435456, consuming tracker:<Load#Id=d14d079dff4cda8d-2fbffd7dcb21d5b4>, peak used 4628458272, current used 1985918388, exec node:<>, process memory used 474.88 GB exceed limit 866.08 GB or sys available memory 5.01 GB less than low water mark 6.40 GB.

    0#  doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173
    1#  Allocator<false, false, false, DefaultMemoryAllocator>::sys_memory_check(unsigned long) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/common/allocator.cpp:129
    2#  Allocator<false, false, false, DefaultMemoryAllocator>::realloc_impl(void*, unsigned long, unsigned long, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/common/allocator.cpp:187
    3#  doris::vectorized::ColumnStr<unsigned int>::insert_data(char const*, unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/vec/common/pod_array.h:171
    4#  doris::Status doris::vectorized::CsvReader::deserialize_nullable_string<true>(doris::vectorized::IColumn&, doris::Slice&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:350
    5#  doris::vectorized::CsvReader::_fill_dest_columns(doris::Slice const&, doris::vectorized::Block*, std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, unsigned long*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/format/csv/csv_reader.cpp:0
    6#  doris::vectorized::CsvReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
    7#  doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
    8#  doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
    9#  doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vscanner.cpp:0
    10# doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vscanner.cpp:100
    11# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:378
    12# std::_Function_handler<void (), doris::vectorized::ScannerScheduler::submit(std::shared_ptr<doris::vectorized::ScannerContext>, std::shared_ptr<doris::vectorized::ScanTask>)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701
    13# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
    14# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
    15# start_thread
    16# __clone

, FILE:/home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp, LINE:175, FUNCTION:auto doris::vectorized::ScannerScheduler::submit(std::shared_ptr, std::shared_ptr)::(anonymous class)::operator()()::(anonymous class)::operator()()::(anonymous class)::operator()() const, ErrorURL:

What You Expected?

no mem oom

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct