[SERVER-47039] Actual exception type: std::length_error Created: 23/Mar/20  Updated: 27/Oct/23  Resolved: 07/Jul/20

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 4.0.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kin Wai Cheung Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongod.log-20200323_crash.gz    
Operating System: ALL
Steps To Reproduce:

A deveoper did execute statements ( a lot of sessions) doing

2020-03-19T15:49:25.396+0000 I COMMAND [conn605664] command ctr.trademarks.chin command: getMore { getMore: 1385985404742, collection: "trademarks.chin", lsid:
 
{ id: UUID("d6a503d9-cbfe-4856-bf19-eb3ece04f3b7"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) }
 
, $clusterTime: { clusterTime: Timestamp(1584632964, 494), signature:
 
{ hash: BinData(0, F9BDF452767691F36DF3D1DFD37513245957FD8B), keyId: 6784397986496512001 }
 
}, $client: { driver:
 
{ name: "mongo-java-driver|legacy", version: "3.10.2" }
 
, os: \{ type: "Linux", name: "Linux", architecture: "amd64", version: "4.1.12-61.1.28.el6uek.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_202-b08", mongos: \{ host: "pmgo-pl203:27017", client: "10.178.96.14:46716", version: "4.0.9" } }, $configServerState: { opTime:
 
{ ts: Timestamp(1584632962, 549), t: 31 }
 
}, $db: "ctr" } originatingCommand: { find: "trademarks.chin", filter: { _id.guid:
 
{ $exists: true, $gte: "I4BA46448F76111E39F868C89E14BE28C", $lt: "I4CB4C96EAC8E11E8B600005056B74373" }
 
}, projection: { Doc.n-document.n-docbody.Document.id: 1, $sortKey:
 
{ $meta: "sortKey" }
 
}, sort: \{ _id.guid: -1, _id.version: -1 }, shardVersion: [ Timestamp(5932, 0), ObjectId('5a192f568425c9bb9d6072d9') ], lsid: \{ id: UUID("d6a503d9-cbfe-4856-bf19-eb3ece04f3b7"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) }, $clusterTime: { clusterTime: Timestamp(1584632964, 494), signature:
 
{ hash: BinData(0, F9BDF452767691F36DF3D1DFD37513245957FD8B), keyId: 6784397986496512001 }
 
}, $client: { driver:
 
{ name: "mongo-java-driver|legacy", version: "3.10.2" }
 
, os: \{ type: "Linux", name: "Linux", architecture: "amd64", version: "4.1.12-61.1.28.el6uek.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_202-b08", mongos: \{ host: "pmgo-pl203:27017", client: "10.178.96.14:46716", version: "4.0.9" } }, $configServerState: { opTime:
 
{ ts: Timestamp(1584632962, 549), t: 31 }
 
}, $db: "ctr" } planSummary: IXSCAN \{ _id.guid: 1, _id.version: 1 } cursorid:1385985404742 keysExamined:68556 docsExamined:68556 cursorExhausted:1 numYields:535 nreturned:68556 reslen:16771717 locks:{ Global: { acquireCount:
 
{ r: 536 }
 
}, Database: { acquireCount:
 
{ r: 536 }
 
}, Collection: { acquireCount:
 
{ r: 536 }
 
} } storage:{ data:
 
{ bytesRead: 108540709, timeReadingMicros: 163979 }
 
} protocol:op_msg 859ms
 2020-03-19T15:49:28.191+0000 I NETWORK [listener] connection accepted from 10.178.96.14:57573 #605805 (51 connections now open)
 2020-03-19T15:49:28.191+0000 I NETWORK [listener] connection accepted from 10.178.96.14:57574 #605806 (52 connections now open)
 2020-03-19T15:49:28.191+0000 I NETWORK [conn605805] received client metadata from 10.178.96.14:57573 conn605805: { driver:
 
{ name: "NetworkInterfaceTL", version: "4.0.9" }
 
, os: \{ type: "Linux", name: "Oracle Linux Server release 6.9", architecture: "x86_64", version: "Kernel 4.1.12-61.1.28.el6uek.x86_64" } }
 2020-03-19T15:49:28.191+0000 I NETWORK [conn605806] received client metadata from 10.178.96.14:57574 conn605806: { driver:
 
{ name: "NetworkInterfaceTL", version: "4.0.9" }
 
, os: \{ type: "Linux", name: "Oracle Linux Server release 6.9", architecture: "x86_64", version: "Kernel 4.1.12-61.1.28.el6uek.x86_64" } }
 2020-03-19T15:49:34.681+0000 I SHARDING [conn605770] request split points lookup for chunk bdt.normalized \{ : "44114VENE" } -->> \{ : "44162201CHIN" }
 2020-03-19T15:49:34.729+0000 I NETWORK [listener] connection accepted from 10.178.96.14:57582 #605807 (53 connections now open)
 2020-03-19T15:49:34.729+0000 I NETWORK [conn605807] received client metadata from 10.178.96.14:57582 conn605807: { driver:
 
{ name: "NetworkInterfaceTL", version: "4.0.9" }
 
, os: \{ type: "Linux", name: "Oracle Linux Server release 6.9", architecture: "x86_64", version: "Kernel 4.1.12-61.1.28.el6uek.x86_64" } }
 2020-03-19T15:49:34.730+0000 I SHARDING [conn605807] request split points lookup for chunk bdt.normalized \{ : "44114VENE" } -->> \{ : "44162201CHIN" }
 2020-03-19T15:49:38.293+0000 F - [conn605626] terminate() called. An exception is active; attempting to gather more information
 2020-03-19T15:49:38.360+0000 F - [conn605626] std::exception::what(): vector::reserve

Participants:

 Description   

 
 
2020-03-19T15:49:38.293+0000 F - [conn605626] terminate() called. An exception is active; attempting to gather more information
 2020-03-19T15:49:38.360+0000 F - [conn605626] std::exception::what(): vector::reserve
 Actual exception type: std::length_error
 0x55afb12e9da1 0x55afb12e9785 0x55afb13de7c6 0x55afb13de811 0x55afb13de918 0x55afb13f767f 0x55afb08f1ee4 0x55afb10d687c 0x55afb08a96eb 0x55afb08a5418 0x55afb08a5fbc 0x55afb08bff29 0x55afb090375d 0x55afb01bf265 0x55afb01bf18d 0x55afb01bf95b 0x55afb02107ed 0x55afb021117b 0x55afaffb4637 0x55afaffaa29a 0x55afb0d4dfb9 0x55afaf992088 0x55afaf99419e 0x55afaf995ab9 0x55afaf996a01 0x55afaf98209a 0x55afaf98ec3a 0x55afaf9899a7 0x55afaf98d101 0x55afb0b3c892 0x55afaf987b90 0x55afaf98ac85 0x55afaf9890e7 0x55afaf989a2d 0x55afaf98d101 0x55afb0b3cdf5 0x55afb1240e24 0x7f1dbe0c9aa1 0x7f1dbde16bcd
 ----- BEGIN BACKTRACE -----
 
{"backtrace":[\\{"b":"55AFAEEC3000","o":"2426DA1","s":"_ZN5mongo15printStackTraceERSo"}
 
,\{"b":"55AFAEEC3000","o":"2426785"},{"b":"55AFAEEC3000","o":"251B7C6","s":"_ZN10__cxxabiv111__terminateEPFvvE"},\{"b":"55AFAEEC3000","o":"251B811"},\{"b":"55AFAEEC3000","o":"251B918"},{"b":"55AFAEEC3000","o":"253467F","s":"_ZSt20__throw_length_errorPKc"},\{"b":"55AFAEEC3000","o":"1A2EEE4"},\{"b":"55AFAEEC3000","o":"221387C","s":"_ZN5mongo5Value20deserializeForSorterERNS_9BufReaderERKNS0_25SorterDeserializeSettingsE"},\{"b":"55AFAEEC3000","o":"19E66EB","s":"_ZN5mongo6sorter12FileIteratorINS_5ValueES2_E4nextEv"},\{"b":"55AFAEEC3000","o":"19E2418","s":"_ZN5mongo19DocumentSourceGroup10initializeEv"},\{"b":"55AFAEEC3000","o":"19E2FBC","s":"_ZN5mongo19DocumentSourceGroup7getNextEv"},\{"b":"55AFAEEC3000","o":"19FCF29","s":"_ZN5mongo19DocumentSourceMatch7getNextEv"},\{"b":"55AFAEEC3000","o":"1A4075D","s":"_ZN5mongo8Pipeline7getNextEv"},\{"b":"55AFAEEC3000","o":"12FC265","s":"_ZN5mongo18PipelineProxyStage11getNextBsonEv"},\{"b":"55AFAEEC3000","o":"12FC18D","s":"_ZN5mongo18PipelineProxyStage6doWorkEPm"},\{"b":"55AFAEEC3000","o":"12FC95B","s":"_ZN5mongo9PlanStage4workEPm"},\{"b":"55AFAEEC3000","o":"134D7ED","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},\{"b":"55AFAEEC3000","o":"134E17B","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},\{"b":"55AFAEEC3000","o":"10F1637","s":"_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_7BSONObjERNS_14BSONObjBuilderE"},\{"b":"55AFAEEC3000","o":"10E729A"},\{"b":"55AFAEEC3000","o":"1E8AFB9","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE"},\{"b":"55AFAEEC3000","o":"ACF088"},\{"b":"55AFAEEC3000","o":"AD119E"},\{"b":"55AFAEEC3000","o":"AD2AB9"},\{"b":"55AFAEEC3000","o":"AD3A01","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},\{"b":"55AFAEEC3000","o":"ABF09A","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},\{"b":"55AFAEEC3000","o":"ACBC3A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},\{"b":"55AFAEEC3000","o":"AC69A7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},\{"b":"55AFAEEC3000","o":"ACA101"},\{"b":"55AFAEEC3000","o":"1C79892","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},\{"b":"55AFAEEC3000","o":"AC4B90","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},\{"b":"55AFAEEC3000","o":"AC7C85","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},\{"b":"55AFAEEC3000","o":"AC60E7","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},\{"b":"55AFAEEC3000","o":"AC6A2D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},\{"b":"55AFAEEC3000","o":"ACA101"},\{"b":"55AFAEEC3000","o":"1C79DF5"},\{"b":"55AFAEEC3000","o":"237DE24"},\{"b":"7F1DBE0C2000","o":"7AA1"},\{"b":"7F1DBDD2E000","o":"E8BCD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.9", "gitVersion" : "fc525e2d9b0e4bceff5c2201457e564362909765", "compiledModules" : [], "uname" :
 
{ "sysname" : "Linux", "release" : "4.1.12-61.1.28.el6uek.x86_64", "version" : "#2 SMP Thu Feb 23 20:03:53 PST 2017", "machine" : "x86_64" }
 
, "somap" : [ \{ "b" : "55AFAEEC3000", "elfType" : 3, "buildId" : "1608990BA9F24FFB0C9133E50C74957A69393AE7" }, \{ "b" : "7FFFDBBDE000", "elfType" : 3, "buildId" : "61D4C01DF75DC089B47A6D2EE3A9EAE22EDA3691" }, \{ "b" : "7EE23FFF0000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "25589170D46DC1419D258BCEF8EBA45DFEBF90C9" }, \{ "b" : "7EE24A9D6000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F0BE1166EDCFFB2422B940D601A1BBD89352D80F" }, \{ "b" : "7EE2431F1000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "9B14622E8FD1C74B89FD318DD1ACDA27A0713FF8" }, \{ "b" : "7EE23FF85000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "2C58F69B34107BCE6E0E65B2642CBCE7B9571ECE" }, \{ "b" : "7EE24C181000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1F7E85410384392BC51FA7324961719A10125F31" }, \{ "b" : "7EE24B379000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "8CE34B59DECD0B577E05E1D9085F7EDD18AD6F17" }, \{ "b" : "7EE24ACF5000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1F4B47E6CD57B2A0E10406329448ED33C095FBAF" }, \{ "b" : "7EE2492DF000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "BC7550A8A7C2D706FE4E489058BADC963465DBB7" }, \{ "b" : "7EE24B0C2000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "9704352BD4F54A3839DA5AE80D9C4633FDC600D1" }, \{ "b" : "7EE24B12E000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "DC14F55D42D410E6BF92975130E0894F214B9A07" }, \{ "b" : "7EE24D245000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9C5AE7B449C51BEC0EA459850AA037931F524D26" }, \{ "b" : "7EE248EFC000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "65E4BD3DC00C6CB5B31BE35CECFAEC28C57FA65F" }, \{ "b" : "7EE23F8AB000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "D28AB8040B8D7DEFFF17AF81398D2E9F7937224E" }, \{ "b" : "7EE241E67000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2CBD7B734972204176F168E3BC983371F08D46B0" }, \{ "b" : "7EE243380000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "68864AF40B36D187E9B61CCAD2741DB1A1E647A9" }, \{ "b" : "7EE243D54000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "13CCE164ACE084DAB18CB6280557C0535FCBDE80" }, \{ "b" : "7EE244750000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D593146D2D8A933B3782E63E021B13E72169426F" }, \{ "b" : "7EE24913A000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, \{ "b" : "7EE243AED000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "0C02309C46E99FA217C6C969446C60F8BF2E4898" }, \{ "b" : "7EE243CC1000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "F2C9D1C53F5E37DDB1B27EC28013BCEC6CD4FA08" }, \{ "b" : "7EE24597E000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "97E2C4BA1FC045EEC14CD1D1C40CF50A35810384" }, \{ "b" : "7EE244F51000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "ECA4DB7B320155B81EB3358CBE79186277A24DDD" }, \{ "b" : "7EE245D4D000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "A5A515D1FF66E636267EC260BEB72918BC4E325A" }, \{ "b" : "7EE245748000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "280139E2286532655CC31EDAE943B8729E25DF0D" }, \{ "b" : "7EE244D0A000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "2BF6EF6A0BA1114869862223E8DCFCC30FC0B739" }, \{ "b" : "7EE23C2E2000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "52F6E18262B6DFAC418D2B474D51F8C927A3AC91" }, \{ "b" : "7EE23E4D3000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "04C0A5933873AB7117968F72138C68EA610BE27A" }, \{ "b" : "7EE23F2B9000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "90F78E5E1AF40EE481268BEAF15F9A2E3E8EFBD2" }, \{ "b" : "7EE240CAE000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "3E8E0EB7EDCBF0C22F201C8E56EB2CB7F5017490" }, \{ "b" : "7EE2406AB000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, \{ "b" : "7EE243474000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "F542C8ACD4AD1F2C6A551043BDFBAB051905DA1C" }, \{ "b" : "7EE246A55000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, \{ "b" : "7EE242C52000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" } ] }}
 
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55afb12e9da1]
 mongod(+0x2426785) [0x55afb12e9785]
 mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x55afb13de7c6]
 mongod(+0x251B811) [0x55afb13de811]
 mongod(+0x251B918) [0x55afb13de918]
 mongod(_ZSt20__throw_length_errorPKc+0x3F) [0x55afb13f767f]
 mongod(+0x1A2EEE4) [0x55afb08f1ee4]
 mongod(_ZN5mongo5Value20deserializeForSorterERNS_9BufReaderERKNS0_25SorterDeserializeSettingsE+0x62C) [0x55afb10d687c]
 mongod(_ZN5mongo6sorter12FileIteratorINS_5ValueES2_E4nextEv+0x5B) [0x55afb08a96eb]
 mongod(_ZN5mongo19DocumentSourceGroup10initializeEv+0x618) [0x55afb08a5418]
 mongod(_ZN5mongo19DocumentSourceGroup7getNextEv+0xBC) [0x55afb08a5fbc]
 mongod(_ZN5mongo19DocumentSourceMatch7getNextEv+0x49) [0x55afb08bff29]
 mongod(_ZN5mongo8Pipeline7getNextEv+0x3D) [0x55afb090375d]
 mongod(_ZN5mongo18PipelineProxyStage11getNextBsonEv+0x35) [0x55afb01bf265]
 mongod(_ZN5mongo18PipelineProxyStage6doWorkEPm+0xFD) [0x55afb01bf18d]
 mongod(_ZN5mongo9PlanStage4workEPm+0x6B) [0x55afb01bf95b]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x50D) [0x55afb02107ed]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x55afb021117b]
 mongod(_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_7BSONObjERNS_14BSONObjBuilderE+0x1EE7) [0x55afaffb4637]
 mongod(+0x10E729A) [0x55afaffaa29a]
 mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x55afb0d4dfb9]
 mongod(+0xACF088) [0x55afaf992088]
 mongod(+0xAD119E) [0x55afaf99419e]
 mongod(+0xAD2AB9) [0x55afaf995ab9]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x3D1) [0x55afaf996a01]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3A) [0x55afaf98209a]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55afaf98ec3a]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55afaf9899a7]
 mongod(+0xACA101) [0x55afaf98d101]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x55afb0b3c892]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x150) [0x55afaf987b90]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB05) [0x55afaf98ac85]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x55afaf9890e7]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55afaf989a2d]
 mongod(+0xACA101) [0x55afaf98d101]
 mongod(+0x1C79DF5) [0x55afb0b3cdf5]
 mongod(+0x237DE24) [0x55afb1240e24]
 libpthread.so.0(+0x7AA1) [0x7f1dbe0c9aa1]
 libc.so.6(clone+0x6D) [0x7f1dbde16bcd]
 ----- END BACKTRACE -----



 Comments   
Comment by Dmitry Agranat [ 07/Jul/20 ]

Hi kinwai.cheung@clarivate.com,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Carl Champain (Inactive) [ 26/Mar/20 ]

kinwai.cheung@clarivate.com,

We used internal tools to investigate the backtrace. Upgrading to the latest 4.0.17 would help us determine whether or not you are experiencing SERVER-41918.

Thank you,
Carl
 

Comment by Kin Wai Cheung [ 26/Mar/20 ]

Thanks Carl will check it out.

I tried make the backtrace more readable with c++filt (sse below) but it didn't help me further though.

How did you came to the conclusion that CollectionBulkLoaderImpl could be the issue?

When I checked for the std::exception::what(): vector::reserve on

https://en.cppreference.com/w/cpp/container/vector/reserve

it tells me that the new_cap value is greater than it's max_size()

--------------------------------------------------

2020-03-19T16:47:41.630+0000 F - [conn114] std::exception::what(): vector::reserve
Actual exception type: std::length_error
0x55adacd99da1 0x55adacd99785 0x55adace8e7c6 0x55adace8e811 0x55adace8e918 0x55adacea767f 0x55adac3a1ee4 0x55adacb8687c 0x55adac3596eb 0x55adac355418 0x55adac355fbc 0x55adac36ff29 0x55adac3b375d 0x55adabc6f265 0x55adabc6f18d 0x55adabc6f95b 0x55adabcc07ed 0x55adabcc117b 0x55adaba64637 0x55adaba5a29a 0x55adac7fdfb9 0x55adab442088 0x55adab44419e 0x55adab445ab9 0x55adab446a01 0x55adab43209a 0x55adab43ec3a 0x55adab4399a7 0x55adab43d101 0x55adac5ec892 0x55adab437b90 0x55adab43ac85 0x55adab4390e7 0x55adab439a2d 0x55adab43d101 0x55adac5ecdf5 0x55adaccf0e24 0x7fb432cbeaa1 0x7fb432a0bbcd
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"55ADAA973000","o":"2426DA1","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"55ADAA973000","o":"2426785"},{"b":"55ADAA973000","o":"251B7C6","s":"ZN10cxxabiv111terminateEPFvvE"},{"b":"55ADAA973000","o":"251B811"},{"b":"55ADAA973000","o":"251B918"},{"b":"55ADAA973000","o":"253467F","s":"_ZSt20_throw_length_errorPKc"},{"b":"55ADAA973000","o":"1A2EEE4"},{"b":"55ADAA973000","o":"221387C","s":"_ZN5mongo5Value20deserializeForSorterERNS_9BufReaderERKNS0_25SorterDeserializeSettingsE"},{"b":"55ADAA973000","o":"19E66EB","s":"_ZN5mongo6sorter12FileIteratorINS_5ValueES2_E4nextEv"},{"b":"55ADAA973000","o":"19E2418","s":"_ZN5mongo19DocumentSourceGroup10initializeEv"},{"b":"55ADAA973000","o":"19E2FBC","s":"_ZN5mongo19DocumentSourceGroup7getNextEv"},{"b":"55ADAA973000","o":"19FCF29","s":"_ZN5mongo19DocumentSourceMatch7getNextEv"},{"b":"55ADAA973000","o":"1A4075D","s":"_ZN5mongo8Pipeline7getNextEv"},{"b":"55ADAA973000","o":"12FC265","s":"_ZN5mongo18PipelineProxyStage11getNextBsonEv"},{"b":"55ADAA973000","o":"12FC18D","s":"_ZN5mongo18PipelineProxyStage6doWorkEPm"},{"b":"55ADAA973000","o":"12FC95B","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"55ADAA973000","o":"134D7ED","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"55ADAA973000","o":"134E17B","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"55ADAA973000","o":"10F1637","s":"_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"55ADAA973000","o":"10E729A"},{"b":"55ADAA973000","o":"1E8AFB9","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE"},{"b":"55ADAA973000","o":"ACF088"},{"b":"55ADAA973000","o":"AD119E"},{"b":"55ADAA973000","o":"AD2AB9"},{"b":"55ADAA973000","o":"AD3A01","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"55ADAA973000","o":"ABF09A","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55ADAA973000","o":"ACBC3A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55ADAA973000","o":"AC69A7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55ADAA973000","o":"ACA101"},{"b":"55ADAA973000","o":"1C79892","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55ADAA973000","o":"AC4B90","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55ADAA973000","o":"AC7C85","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55ADAA973000","o":"AC60E7","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55ADAA973000","o":"AC6A2D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55ADAA973000","o":"ACA101"},{"b":"55ADAA973000","o":"1C79DF5"},{"b":"55ADAA973000","o":"237DE24"},{"b":"7FB432CB7000","o":"7AA1"},{"b":"7FB432923000","o":"E8BCD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.9", "gitVersion" : "fc525e2d9b0e4bceff5c2201457e564362909765", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "4.1.12-61.1.28.el6uek.x86_64", "version" : "#2 SMP Thu Feb 23 20:03:53 PST 2017", "machine" : "x86_64" }

, "somap" : [ { "b" : "55ADAA973000", "elfType" : 3, "buildId" : "1608990BA9F24FFB0C9133E50C74957A69393AE7" }, { "b" : "7FFDC0D80000", "elfType" : 3, "buildId" : "61D4C01DF75DC089B47A6D2EE3A9EAE22EDA3691" }, { "b" : "7F78B4BE5000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "25589170D46DC1419D258BCEF8EBA45DFEBF90C9" }, { "b" : "7F78BF5CB000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F0BE1166EDCFFB2422B940D601A1BBD89352D80F" }, { "b" : "7F78B7DE6000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "9B14622E8FD1C74B89FD318DD1ACDA27A0713FF8" }, { "b" : "7F78B4B7A000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "2C58F69B34107BCE6E0E65B2642CBCE7B9571ECE" }, { "b" : "7F78C0D76000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1F7E85410384392BC51FA7324961719A10125F31" }, { "b" : "7F78BFF6E000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "8CE34B59DECD0B577E05E1D9085F7EDD18AD6F17" }, { "b" : "7F78BF8EA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1F4B47E6CD57B2A0E10406329448ED33C095FBAF" }, { "b" : "7F78BDED4000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "BC7550A8A7C2D706FE4E489058BADC963465DBB7" }, { "b" : "7F78BFCB7000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "9704352BD4F54A3839DA5AE80D9C4633FDC600D1" }, { "b" : "7F78BFD23000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "DC14F55D42D410E6BF92975130E0894F214B9A07" }, { "b" : "7F78C1E3A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9C5AE7B449C51BEC0EA459850AA037931F524D26" }, { "b" : "7F78BDAF1000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "65E4BD3DC00C6CB5B31BE35CECFAEC28C57FA65F" }, { "b" : "7F78B44A0000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "D28AB8040B8D7DEFFF17AF81398D2E9F7937224E" }, { "b" : "7F78B6A5C000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2CBD7B734972204176F168E3BC983371F08D46B0" }, { "b" : "7F78B7F75000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "68864AF40B36D187E9B61CCAD2741DB1A1E647A9" }, { "b" : "7F78B8949000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "13CCE164ACE084DAB18CB6280557C0535FCBDE80" }, { "b" : "7F78B9345000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D593146D2D8A933B3782E63E021B13E72169426F" }, { "b" : "7F78BDD2F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7F78B86E2000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "0C02309C46E99FA217C6C969446C60F8BF2E4898" }, { "b" : "7F78B88B6000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "F2C9D1C53F5E37DDB1B27EC28013BCEC6CD4FA08" }, { "b" : "7F78BA573000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "97E2C4BA1FC045EEC14CD1D1C40CF50A35810384" }, { "b" : "7F78B9B46000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "ECA4DB7B320155B81EB3358CBE79186277A24DDD" }, { "b" : "7F78BA942000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "A5A515D1FF66E636267EC260BEB72918BC4E325A" }, { "b" : "7F78BA33D000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "280139E2286532655CC31EDAE943B8729E25DF0D" }, { "b" : "7F78B98FF000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "2BF6EF6A0BA1114869862223E8DCFCC30FC0B739" }, { "b" : "7F78B0ED7000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "52F6E18262B6DFAC418D2B474D51F8C927A3AC91" }, { "b" : "7F78B30C8000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "04C0A5933873AB7117968F72138C68EA610BE27A" }, { "b" : "7F78B3EAE000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "90F78E5E1AF40EE481268BEAF15F9A2E3E8EFBD2" }, { "b" : "7F78B58A3000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "3E8E0EB7EDCBF0C22F201C8E56EB2CB7F5017490" }, { "b" : "7F78B52A0000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7F78B8069000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "F542C8ACD4AD1F2C6A551043BDFBAB051905DA1C" }, { "b" : "7F78BB64A000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, { "b" : "7F78B7847000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55adacd99da1]
mongod(+0x2426785) [0x55adacd99785]
mongod(ZN10cxxabiv111_terminateEPFvvE+0x6) [0x55adace8e7c6]
mongod(+0x251B811) [0x55adace8e811]
mongod(+0x251B918) [0x55adace8e918]
mongod(ZSt20_throw_length_errorPKc+0x3F) [0x55adacea767f]
mongod(+0x1A2EEE4) [0x55adac3a1ee4]
mongod(_ZN5mongo5Value20deserializeForSorterERNS_9BufReaderERKNS0_25SorterDeserializeSettingsE+0x62C) [0x55adacb8687c]
mongod(_ZN5mongo6sorter12FileIteratorINS_5ValueES2_E4nextEv+0x5B) [0x55adac3596eb]
mongod(_ZN5mongo19DocumentSourceGroup10initializeEv+0x618) [0x55adac355418]
mongod(_ZN5mongo19DocumentSourceGroup7getNextEv+0xBC) [0x55adac355fbc]
mongod(_ZN5mongo19DocumentSourceMatch7getNextEv+0x49) [0x55adac36ff29]
mongod(_ZN5mongo8Pipeline7getNextEv+0x3D) [0x55adac3b375d]
mongod(_ZN5mongo18PipelineProxyStage11getNextBsonEv+0x35) [0x55adabc6f265]
mongod(_ZN5mongo18PipelineProxyStage6doWorkEPm+0xFD) [0x55adabc6f18d]
mongod(_ZN5mongo9PlanStage4workEPm+0x6B) [0x55adabc6f95b]
mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x50D) [0x55adabcc07ed]
mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x55adabcc117b]
mongod(_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_7BSONObjERNS_14BSONObjBuilderE+0x1EE7) [0x55adaba64637]
mongod(+0x10E729A) [0x55adaba5a29a]
mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x55adac7fdfb9]
mongod(+0xACF088) [0x55adab442088]
mongod(+0xAD119E) [0x55adab44419e]
mongod(+0xAD2AB9) [0x55adab445ab9]
mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x3D1) [0x55adab446a01]
mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3A) [0x55adab43209a]
mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55adab43ec3a]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55adab4399a7]
mongod(+0xACA101) [0x55adab43d101]
mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x55adac5ec892]
mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x150) [0x55adab437b90]
mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB05) [0x55adab43ac85]
mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x55adab4390e7]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55adab439a2d]
mongod(+0xACA101) [0x55adab43d101]
mongod(+0x1C79DF5) [0x55adac5ecdf5]
mongod(+0x237DE24) [0x55adaccf0e24]
libpthread.so.0(+0x7AA1) [0x7fb432cbeaa1]
libc.so.6(clone+0x6D) [0x7fb432a0bbcd]
----- END BACKTRACE -----

Comment by Carl Champain (Inactive) [ 25/Mar/20 ]

kinwai.cheung@clarivate.com,

You could be experiencing SERVER-41918 (take a look for more details). Can you please upgrade to the latest 4.0.17?

Let us know if it resolves your issue!
Carl

Comment by Kin Wai Cheung [ 24/Mar/20 ]

mongod.log when crash occured on primary node have been attached

Comment by Carl Champain (Inactive) [ 24/Mar/20 ]

Hi kinwai.cheung@clarivate.com,

Thank you for the report.
Can you please provide the mongod.log file for this node?

Generated at Thu Feb 08 05:13:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.