feat(query): window partition by spill to disk (#16441)

* dma write Signed-off-by: coldWater <[email protected]> * partition spill to disk Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * refine Signed-off-by: coldWater <[email protected]> * temp file Signed-off-by: coldWater <[email protected]> * refactor spill Signed-off-by: coldWater <[email protected]> * clear temp dir Signed-off-by: coldWater <[email protected]> * move dma Signed-off-by: coldWater <[email protected]> * config Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * bytes limit Signed-off-by: coldWater <[email protected]> * rustix Signed-off-by: coldWater <[email protected]> * rustix Signed-off-by: coldWater <[email protected]> * temp Signed-off-by: coldWater <[email protected]> * Location Signed-off-by: coldWater <[email protected]> * builder Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * SpillerConfig Signed-off-by: coldWater <[email protected]> * temp dir Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * drop_disk_spill_dir_unknown Signed-off-by: coldWater <[email protected]> * drop residual temp dir Signed-off-by: coldWater <[email protected]> * no copy read Signed-off-by: coldWater <[email protected]> * rename reserved_disk_ratio Signed-off-by: coldWater <[email protected]> * record the profile of LocalSpill and RemoteSpill separately Signed-off-by: coldWater <[email protected]> * defalut disable Signed-off-by: coldWater <[email protected]> * update setting Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * fix dma Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * refactor dma Signed-off-by: coldWater <[email protected]> * change config Signed-off-by: coldWater <[email protected]> * Alignment Signed-off-by: coldWater <[email protected]> * cloud test Signed-off-by: coldWater <[email protected]> * x Signed-off-by: coldWater <[email protected]> * clean Signed-off-by: coldWater <[email protected]> * spill_local_disk_path Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> * fix Signed-off-by: coldWater <[email protected]> --------- Signed-off-by: coldWater <[email protected]>
databendlabs · Oct 8, 2024 · 040e8fa · 040e8fa
1 parent d8aa952
commit 040e8fa
Show file tree

Hide file tree

Showing 39 changed files with 1,602 additions and 327 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -247,6 +247,7 @@ http = "1"
 itertools = "0.10.5"
 jsonb = "0.4.3"
 jwt-simple = "0.11.0"
+libc = { version = "0.2.158" }
 match-template = "0.0.1"
 mysql_async = { version = "0.34", default-features = false, features = ["native-tls-tls"] }
 object_store_opendal = "0.46"

diff --git a/src/common/arrow/src/arrow/array/mod.rs b/src/common/arrow/src/arrow/array/mod.rs
@@ -31,7 +31,7 @@
 //! to a concrete struct based on [`PhysicalType`](crate::arrow::datatypes::PhysicalType) available from [`Array::data_type`].
 //! All immutable arrays are backed by [`Buffer`](crate::arrow::buffer::Buffer) and thus cloning and slicing them is `O(1)`.
 //!
-//! Most arrays contain a [`MutableArray`] counterpart that is neither clonable nor sliceable, but
+//! Most arrays contain a [`MutableArray`] counterpart that is neither cloneable nor sliceable, but
 //! can be operated in-place.
 use std::any::Any;
 use std::sync::Arc;

diff --git a/src/common/base/Cargo.toml b/src/common/base/Cargo.toml
@@ -31,7 +31,7 @@ databend-common-exception = { workspace = true }
 enquote = "1.1.0"
 fastrace = { workspace = true }
 futures = { workspace = true }
-libc = "0.2.153"
+libc = { workspace = true }
 log = { workspace = true }
 logcall = { workspace = true }
 micromarshal = "0.5.0"
@@ -50,6 +50,7 @@ prometheus-parse = "0.2.3"
 rand = { workspace = true, features = ["serde1"] }
 regex = { workspace = true }
 replace_with = "0.1.7"
+rustix = "0.38.37"
 semver = { workspace = true }
 serde = { workspace = true }
 serde_json = { workspace = true }