-
Type: Task
-
Resolution: Works as Designed
-
Priority: Unknown
-
None
-
Affects Version/s: None
-
Component/s: None
Hi, when I'm trying to make parallel find in one collection(which contains 10190890 records, but I limit it to 509545 items), I'm sadly find out that
when find complete, and everything is dropped, the memory still remains to high,
When it first make a concurrent read, the memory is 2.9G, I think this is expected.
But everything is done, the memory still be 391M, which suprise to me.
I have no clue about what's going on here and why this happened, can you please help?
My server contains 40-core cpu, total memory is 120G.
Here is one record example in mongodb:
{ "_id" : ObjectId("6137150c554447359d6141a7"), "beta" : 22, "momentum" : 32, "size" : 11, "earnings_yield" : 21, "residual_volatility" : 1, "growth" : 15, "book_to_price" : 14, "leverage" : 13, "liquidity" : 21, "non_linear_size" : 129, "industry" : "adf", "comovement" : 1, "date" : ISODate("2005-01-06T00:00:00.000Z"), "order_book_id" : "fdas" }here is reproducing code example:
Unable to find source-code formatter for language: rust. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
rust use bson::{doc, Document}; use mongodb::options::FindOptions; use mongodb::sync::Client; use std::sync::mpsc; use std::thread; use std::io::{self, BufRead}; enum TaskResult { Success, Fail, } fn main() { { let source = Client::with_uri_str("mongodb://localhost:27017/?authSource=admin") .unwrap(); let (sender, receiver) = mpsc::channel(); for _ in 0..40 { let source_coll = source.database("betahub").collection::<Document>("factor_exposure"); let sender = sender.clone(); thread::spawn(move || { let res = source_coll .find(doc! {}, FindOptions::builder().batch_size(10000).limit(509545).build()) .and_then(|cursor| { let mut buf = vec![]; Ok(for _doc in cursor { buf.push(_doc); if buf.len() >= 10000 {buf.clear()}}) }); match res { Err(e) => { let _ = sender.send(TaskResult::Fail); } Ok(_) => { let _ = sender.send(TaskResult::Success); } } }); } let mut count = 0; while let Ok(event) = receiver.recv() { match event { TaskResult::Fail => { println!("get fail"); break; }, TaskResult::Success => { count += 1; if count == 40 { break; } } } } } println!("Every thing is done, press enter to exit."); let _line1 = io::stdin().lock().lines().next().unwrap().unwrap(); }