[SERVER-14905] mongod crashes without any error message Created: 14/Aug/14  Updated: 10/Dec/14  Resolved: 16/Aug/14

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.6.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dan Dascalescu Assignee: Ramon Fernandez Marina
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

On a 1GB RAM Ubuntu server running on Digital Ocean, a single-instance mongod crashes silently.

uname -a:

Linux stockbase 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

last lines of /var/mongodb/log:

2014-08-14T12:46:13.486-0700 [conn5] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001307617", url: "http://www.sec.gov/Archives/edgar/data/1307617/000130761714000004/0001307617-14-000004-index.htm", pubDate: new Date(1408045202000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:0 locks(micros) r:704798 reslen:48 704ms
2014-08-14T12:46:14.114-0700 [conn2] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001167507", url: "http://www.sec.gov/Archives/edgar/data/1167507/000114036114032723/0001140361-14-032723-index.htm", pubDate: new Date(1408045175000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:0 locks(micros) r:622573 reslen:48 622ms
2014-08-14T12:46:14.799-0700 [conn4] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000859804", url: "http://www.sec.gov/Archives/edgar/data/859804/000085980414000005/0000859804-14-000005-index.htm", pubDate: new Date(1408045167000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:1 locks(micros) r:1082196 reslen:48 679ms
2014-08-14T12:46:15.407-0700 [conn3] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001538375", url: "http://www.sec.gov/Archives/edgar/data/1538375/000104746914007004/0001047469-14-007004-index.htm", pubDate: new Date(1408045158000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:0 locks(micros) r:602606 reslen:48 602ms
2014-08-14T12:46:16.065-0700 [conn5] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0001492224", url: "http://www.sec.gov/Archives/edgar/data/1492224/000108514614001748/0001085146-14-001748-index.htm", pubDate: new Date(1408045126000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:1 locks(micros) r:792784 reslen:48 653ms
2014-08-14T12:47:20.622-0700 [clientcursormon] mem (MB) res:281 virt:8691
2014-08-14T12:47:20.622-0700 [clientcursormon]  mapped (incl journal view):8254
2014-08-14T12:47:20.622-0700 [clientcursormon]  connections:10
2014-08-14T12:48:27.858-0700 [conn4] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000811831", url: "http://www.sec.gov/Archives/edgar/data/811831/000081183114000003/0000811831-14-000003-index.htm", pubDate: new Date(1408045304000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:33 locks(micros) r:5274901 reslen:48 3324ms
2014-08-14T12:48:28.303-0700 [conn5] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000943034", url: "http://www.sec.gov/Archives/edgar/data/943034/000089710114001151/0000897101-14-001151-index.htm", pubDate: new Date(1408045124000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:24 locks(micros) r:5982433 reslen:48 3454ms
2014-08-14T12:48:29.390-0700 [conn3] command stockbase-prod.$cmd command: count { count: "content", query: { ct: "filing", cik: "0000943034", url: "http://www.sec.gov/Archives/edgar/data/943034/000089710114001151/0000897101-14-001151-index.htm", pubDate: new Date(1408045124000) }, fields: null } planSummary: IXSCAN { ct: 1 }, IXSCAN { pubDate: -1, title: 1 } keyUpdates:0 numYields:56 locks(micros) r:6455049 reslen:48 4445ms
2014-08-14T13:41:23.639-0700 ***** SERVER RESTARTED *****

I've run the application code again several times, and one time, the log file did show that mongod was running out of memory:

2014-08-14T13:41:51.054-0700 [conn9] insert stockbase-prod.content query: { ct: "filing", formNumber: "6-K", filerName: "WOORI FINANCE HOLDINGS CO LTD", cik: "0001264136", filerType: "Filer", url: "http://www.sec.gov/Archives/edgar/data/1264136/000119312514310069/0001193125-14-310069-index.htm", summary: "<b>Filed:</b> 2014-08-14 <b>AccNo:</b> 0001193125-14-310069 <b>Size:</b> 4 MB", accessionNumber: "0001193125-14-310069", size: "4", pubDate: new Date(1408046048000), score: 0, type: "investor", _id: "SKKrhjDxgiuGzewkt" } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:12606 1324ms
2014-08-14T13:41:51.055-0700 [conn9] command stockbase-prod.$cmd command: insert { $msg: "query not recording (too large)" } keyUpdates:0 numYields:0 locks(micros) w:284 reslen:40 1334ms
2014-08-14T13:41:51.296-0700 [conn8] query stockbase-prod.feedParsing.jobs query: { $query: { type: { $in: [ "parseFeed" ] }, status: "ready", runId: null, after: { $lte: new Date(1408048907796) }, retries: { $gt: 0 } }, orderby: { priority: 1, retryUntil: 1, after: 1 } } planSummary: COLLSCAN, COLLSCAN ntoreturn:1 ntoskip:0 keyUpdates:0 numYields: 9 locks(micros) r:101699 nreturned:0 reslen:20 107ms
2014-08-14T13:41:57.421-0700 [conn10] Assertion: 10000:out of memory BufBuilder
2014-08-14T13:41:57.439-0700 [conn8] Assertion: 10000:out of memory BufBuilder
2014-08-14T13:41:57.446-0700 [conn9] Assertion: 10000:out of memory BufBuilder
2014-08-14T13:41:57.736-0700 [conn6] SEVERE: out of memory, printing stack and exiting:
0x11e6111 0x11e5408 0x1239bfd 0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6 0x7f7b7e3dedb5 0x7f7b7e3df077 0x11e61c4 0x11e5408 0x1239bfd
0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6
Unable to collect backtrace symbols (errno:12 Cannot allocate memory)
2014-08-14T13:41:57.736-0700 [conn7] SEVERE: out of memory, printing stack and exiting:
0x11e6111 0x11e5408 0x1239bfd 0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6 0x7f7b7e3dedb5 0x7f7b7e3df077 0x11e61c4 0x11e5408 0x1239bfd
0x1643270 0x7f7b7e4043b9 0x7f7b7e404f7b 0x7f7b7e405014 0x7f7b7e3e3386 0x7f7b7e3e75a6
Unable to collect backtrace symbols (errno:12 Cannot allocate memory)

Would it be possible to dump to the log the cause of every crash?



 Comments   
Comment by Ramon Fernandez Marina [ 16/Aug/14 ]

dandv, it appears that the mongod process is not crashing, but it's being killed by the OOM (Out Of Memory) killer. When that happens, it is not possible to write anything to the log.

You can check wether this was the case by looking at your system logs; if you run

grep -i kill /var/log/messages /var/log/syslog

you may see messages like:

Out of memory: Kill process 1234 (mongod) ...

As for the second log, the Assertion line indicates that mongod found a critical problem, namely not being able to get more memory from the system, and shuts down as it can't continue to run.

I'd recommend you increase the RAM of the machine, as 1GB is a very low amount these days.

Generated at Thu Feb 08 03:36:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.