[SERVER-21536] Invariant failure in executor/connection_pool.cpp Created: 18/Nov/15  Updated: 16/Nov/16  Resolved: 17/Dec/15

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 3.2.0-rc3
Fix Version/s: 3.2.1, 3.3.0

Type: Bug Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Adam Midvidy
Resolution: Done Votes: 0
Labels: code-only, fuzzer-blocker
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-23459 Mongos 3.2.0 crash after this error -... Closed
Related
is related to SERVER-21983 3.2.0 mongos crash - Invariant failur... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: Platforms E (01/08/16)
Participants:
Linked BF Score: 0

 Description   

task: https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_jstestfuzz_sharded_WT_bd58ea2ba5d17b960981990bb97cab133d7e90ed_15_11_17_20_13_28

logs: https://logkeeper.mongodb.org/build/564b9554be07c42d836c8e0b/test/564b9b1ebe07c42d836cca76#L10850

[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.655+0000 D EXECUTOR [NetworkInterfaceASIO] Received remote response: RemoteResponse --  cmd:{ lastErrorObject: { updatedExisting: true, n: 1 }, value: { _id: "ip-10-31-116-34:20004:1447794024:980765113", ping: new Date(1447795494619) }, writeConcernError: { code: 11600, errmsg: "interrupted at shutdown" }, ok: 1.0 }
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.655+0000 D EXECUTOR [ShardWorkFixed-0] Executing a task on behalf of pool ShardWorkFixed
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.655+0000 D EXECUTOR [ShardWorkFixed-0] waiting for work; I am one of 1 thread(s); the minimum number of threads is 1
[ShardedClusterFixture:job0:configsvr:primary] 2015-11-17T21:25:24.655+0000 I COMMAND  [conn21] command config.$cmd command: findAndModify { findAndModify: "lockpings", query: { _id: "ip-10-31-116-34:20004:1447794024:980765113" }, update: { $set: { ping: new Date(1447795524621) } }, upsert: true, writeConcern: { w: "majority", wtimeout: 5000 }, maxTimeMS: 30000 } update: { $set: { ping: new Date(1447795524621) } } ntoreturn:1 ntoskip:0 keyUpdates:1 writeConflicts:0 numYields:0 reslen:349 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } protocol:op_command 33ms
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.655+0000 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: interrupted at shutdown
[ShardedClusterFixture:job0:configsvr:primary] 2015-11-17T21:25:24.656+0000 I REPL     [signalProcessingThread] Stopping replication applier threads
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.712+0000 D EXECUTOR [UserCacheInvalidator] Scheduling remote command request: RemoteCommand -- target:localhost:20000 db:admin expDate:2015-11-17T21:25:54.712+0000 cmd:{ _getUserCacheGeneration: 1, maxTimeMS: 30000 }
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.712+0000 I ASIO     [UserCacheInvalidator] dropping unhealthy pooled connection to localhost:20000
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.713+0000 I -        [NetworkInterfaceASIO] Invariant failure iter != pool.end() src/mongo/executor/connection_pool.cpp 566
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.713+0000 I -        [NetworkInterfaceASIO]
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos] ***aborting after invariant() failure
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.715+0000 D COMMAND  [PeriodicTaskRunner] task: DBConnectionPool-cleaner took: 0ms
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.715+0000 D -        [PeriodicTaskRunner] cleaning up unused lock buckets of the global lock manager
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.715+0000 D COMMAND  [PeriodicTaskRunner] task: UnusedLockCleaner took: 0ms
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.715+0000 I NETWORK  [PeriodicTaskRunner] Socket closed remotely, no longer connected (idle 9 secs, remote host 127.0.0.1:20001)
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.715+0000 I NETWORK  [PeriodicTaskRunner] Socket closed remotely, no longer connected (idle 9 secs, remote host 127.0.0.1:20002)
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.715+0000 D COMMAND  [PeriodicTaskRunner] task: DBConnectionPool-cleaner took: 0ms
[ShardedClusterFixture:job0:mongos] 2015-11-17T21:25:24.717+0000 F -        [NetworkInterfaceASIO] Got signal: 6 (Aborted).
[ShardedClusterFixture:job0:mongos] 
[ShardedClusterFixture:job0:mongos]  0xc732e2 0xc72229 0xc72a32 0x320160f790 0x3201232625 0x3201233e05 0xbf98cb 0xa0b7a0 0xa0d201 0xa1059c 0xa108d7 0xa11969 0xc8fd31 0xc8ff51 0xc940ef 0xa1aa55 0xe4a7e0 0x3201607a51 0x32012e893d
[ShardedClusterFixture:job0:mongos] ----- BEGIN BACKTRACE -----
[ShardedClusterFixture:job0:mongos] {"backtrace":[{"b":"400000","o":"8732E2"},{"b":"400000","o":"872229"},{"b":"400000","o":"872A32"},{"b":"3201600000","o":"F790"},{"b":"3201200000","o":"32625"},{"b":"3201200000","o":"33E05"},{"b":"400000","o":"7F98CB"},{"b":"400000","o":"60B7A0"},{"b":"400000","o":"60D201"},{"b":"400000","o":"61059C"},{"b":"400000","o":"6108D7"},{"b":"400000","o":"611969"},{"b":"400000","o":"88FD31"},{"b":"400000","o":"88FF51"},{"b":"400000","o":"8940EF"},{"b":"400000","o":"61AA55"},{"b":"400000","o":"A4A7E0"},{"b":"3201600000","o":"7A51"},{"b":"3201200000","o":"E893D"}],"processInfo":{ "mongodbVersion" : "3.2.0-rc3", "gitVersion" : "bd58ea2ba5d17b960981990bb97cab133d7e90ed", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-220.el6.x86_64", "version" : "#1 SMP Wed Nov 9 08:03:13 EST 2011", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7C3205CC858F69DC8E3C6E602ED085F880344116" }, { "b" : "7FFF858CB000", "elfType" : 3, "buildId" : "08F634A1D22DEFF00461D50A7699DACDC97657BF" }, { "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "E0AEE889D5BF1373F2F9EE0D448DBF3F5B5113F0" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0C72521270790A1BD52C8F6B989EEA5A575085BF" }, { "b" : "7F22BF775000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "93610457BCF424BEBBF1F3FB44E51B51B50F2B55" }, { "b" : "7F22BF392000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "06DDBB192AF74F99DB58F2150BFB83F42F5EBAD3" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "58C5A5FF5C82D7BE3113BE36DD87C7004E3C4DB1" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "B5AE05CEDC0CE917F50A3A468CFA2ACD8592E8F6" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "28AF9321EBEA9D172CA43E11A60E02D0F7014870" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "2AC15B051D1B8B53937E3341EA931D0E96F745D9" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D467973C46E563CDCF64B5F12B2D6A50C7A25BA1" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A6D15926E61580E250ED91F84FF7517F3970CD83" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "04202A4A8BE624D2193E812A25589E2DD02D5B5C" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F704FA7D21D05EF31E90FB4890FCA7F3D91DA138" }, { "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "128802B73016BE233837EA9F2DCBC2153ACC2D6A" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "DC11D5D89BDC77FF242481122D51E5A08DB60DA8" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "15782495E3AF093E67DDAE9A86436FFC6B3CC4D3" }, { "b" : "7EF0BC38A000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "13FFCD68952B7715DDF34C9321D82E3041EA9006" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "44A3A1C1891B4C8170C3DB80E7117A022E5EECD0" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "58BAC04A1DB3964A8F594EFFBE4838AD01214EDC" }, { "b" : "7EF0BC969000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "2D0F26E648D9661ABD83ED8B4BBE8F2CFA50393B" }, { "b" : "7F22BED5E000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "4514552B5354286A143770420B38F2D5985D7FA1" }, { "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "437CA0AB593A7383FF1A1700D14AF4998FE93CF3" }, { "b" : "7F22BEB59000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "EEAA33A75735D35F4BF25C3C2830B8C90ABDD8B5" }, { "b" : "7F22BE94C000", "path" : "/usr/lib64/sasl2/libdigestmd5.so", "elfType" : 3, "buildId" : "34D8E3E2565DEF4A685D6976831B0372AD456993" }, { "b" : "7F22BE746000", "path" : "/usr/lib64/sasl2/libcrammd5.so", "elfType" : 3, "buildId" : "4CC7E695963F5C8B772EDFF456DB67F89E58FBD6" }, { "b" : "7F22BE541000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "F8DDC7A3CA1CE5B75719AE0DC821647B609D17B6" }, { "b" : "7F22BE33C000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "9D19F93E342AA4EE2D646E64642625F365056E5C" }, { "b" : "7F22BE134000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "F7BCE9C6BFF4EAF0CB3142B299CF22D094CE4F04" } ] }}
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc732e2]
[ShardedClusterFixture:job0:mongos]  mongos(+0x872229) [0xc72229]
[ShardedClusterFixture:job0:mongos]  mongos(+0x872A32) [0xc72a32]
[ShardedClusterFixture:job0:mongos]  libpthread.so.0(+0xF790) [0x320160f790]
[ShardedClusterFixture:job0:mongos]  libc.so.6(gsignal+0x35) [0x3201232625]
[ShardedClusterFixture:job0:mongos]  libc.so.6(abort+0x175) [0x3201233e05]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0xbf98cb]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN5mongo8executor14ConnectionPool12SpecificPool22takeFromProcessingPoolEPNS1_19ConnectionInterfaceE+0x0) [0xa0b7a0]
[ShardedClusterFixture:job0:mongos]  mongos(+0x60D201) [0xa0d201]
[ShardedClusterFixture:job0:mongos]  mongos(+0x61059C) [0xa1059c]
[ShardedClusterFixture:job0:mongos]  mongos(+0x6108D7) [0xa108d7]
[ShardedClusterFixture:job0:mongos]  mongos(+0x611969) [0xa11969]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x2F1) [0xc8fd31]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xC1) [0xc8ff51]
[ShardedClusterFixture:job0:mongos]  mongos(_ZN4asio10io_service3runEv+0x2F) [0xc940ef]
[ShardedClusterFixture:job0:mongos]  mongos(+0x61AA55) [0xa1aa55]
[ShardedClusterFixture:job0:mongos]  mongos(+0xA4A7E0) [0xe4a7e0]
[ShardedClusterFixture:job0:mongos]  libpthread.so.0(+0x7A51) [0x3201607a51]
[ShardedClusterFixture:job0:mongos]  libc.so.6(clone+0x6D) [0x32012e893d]
[ShardedClusterFixture:job0:mongos] -----  END BACKTRACE  -----



 Comments   
Comment by Githook User [ 22/Dec/15 ]

Author:

{u'username': u'amidvidy', u'name': u'Adam Midvidy', u'email': u'amidvidy@gmail.com'}

Message: SERVER-21536 don't refresh connections that have already been checked out

(cherry picked from commit a8eb2e709e826cfdb44af619b14b5d4e2f12a2e1)
Branch: v3.2
https://github.com/mongodb/mongo/commit/e0f04ca05bc38ec6bf69616609cc35f341fc6f57

Comment by Githook User [ 17/Dec/15 ]

Author:

{u'username': u'amidvidy', u'name': u'Adam Midvidy', u'email': u'amidvidy@gmail.com'}

Message: SERVER-21536 don't refresh connections that have already been checked out
Branch: master
https://github.com/mongodb/mongo/commit/a8eb2e709e826cfdb44af619b14b5d4e2f12a2e1

Comment by Kamran K. [ 16/Dec/15 ]

I saw a similar invariant failure on a mongod node: https://evergreen.mongodb.com/task/mongodb_mongo_master_windows_64_2k8_debug_concurrency_sharded_WT_f0e044838d6a1093c59626e90e7574d6e428bc86_15_12_15_21_51_57

-        [NetworkInterfaceASIO-0] Invariant failure iter != pool.end() src\mongo\executor\connection_pool.cpp 549
CONTROL  [NetworkInterfaceASIO-0] *** unhandled exception 0x80000003 at 0x000007FEFD1C33B2, terminating
CONTROL  [NetworkInterfaceASIO-0] *** stack trace for unhandled exception:
WRITE    [conn123] update db61.coll61 update: { $inc: { n: 1.0 } } keysExamined:0 docsExamined:1839 nMatched:1839 nModified:1839 keyUpdates:0 writeConflicts:13 numYields:498 locks:{ Global: { acquireCount: { r: 2338, w: 2338 } }, Database: { acquireCount: { w: 2338 } }, Collection: { acquireCount: { w: 499 } }, Metadata: { acquireCount: { w: 1839 } }, oplog: { acquireCount: { w: 1839 } } } 12596ms
COMMAND  [conn123] command db61.$cmd command: update { update: "coll61", updates: [ { q: {}, u: { $inc: { n: 1.0 } }, multi: true, upsert: false } ], ordered: true, shardVersion: [ Timestamp 0|0, ObjectId('00000000ffffffffffffffff') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:232 locks:{ Global: { acquireCount: { r: 2338, w: 2338 } }, Database: { acquireCount: { w: 2338 } }, Collection: { acquireCount: { w: 499 } }, Metadata: { acquireCount: { w: 1839 } }, oplog: { acquireCount: { w: 1839 } } } protocol:op_command 12609ms
WRITE    [conn82] update db61.coll61 update: { $inc: { n: 1.0 } } keysExamined:0 docsExamined:1961 nMatched:1961 nModified:1961 keyUpdates:0 writeConflicts:0 numYields:30 locks:{ Global: { acquireCount: { r: 1992, w: 1992 } }, Database: { acquireCount: { w: 1992 } }, Collection: { acquireCount: { w: 31 } }, Metadata: { acquireCount: { w: 1961 } }, oplog: { acquireCount: { w: 1961 } } } 631ms
COMMAND  [conn82] command db61.$cmd command: update { update: "coll61", updates: [ { q: {}, u: { $inc: { n: 1.0 } }, multi: true, upsert: false } ], ordered: true, shardVersion: [ Timestamp 0|0, ObjectId('00000000ffffffffffffffff') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:232 locks:{ Global: { acquireCount: { r: 1992, w: 1992 } }, Database: { acquireCount: { w: 1992 } }, Collection: { acquireCount: { w: 31 } }, Metadata: { acquireCount: { w: 1961 } }, oplog: { acquireCount: { w: 1961 } } } protocol:op_command 686ms
WRITE    [conn80] update db61.coll61 update: { $inc: { n: 1.0 } } keysExamined:0 docsExamined:1839 nMatched:1839 nModified:1839 keyUpdates:0 writeConflicts:33 numYields:510 locks:{ Global: { acquireCount: { r: 2350, w: 2350 } }, Database: { acquireCount: { w: 2350 } }, Collection: { acquireCount: { w: 511 } }, Metadata: { acquireCount: { w: 1839 } }, oplog: { acquireCount: { w: 1839 } } } 12259ms
COMMAND  [conn80] command db61.$cmd command: update { update: "coll61", updates: [ { q: {}, u: { $inc: { n: 1.0 } }, multi: true, upsert: false } ], ordered: true, shardVersion: [ Timestamp 0|0, ObjectId('00000000ffffffffffffffff') ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:232 locks:{ Global: { acquireCount: { r: 2350, w: 2350 } }, Database: { acquireCount: { w: 2350 } }, Collection: { acquireCount: { w: 511 } }, Metadata: { acquireCount: { w: 1839 } }, oplog: { acquireCount: { w: 1839 } } } protocol:op_command 12260ms
CONTROL  [NetworkInterfaceASIO-0] KERNELBASE.dll                                                                                                   DebugBreak+0x2
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\mongo\util\assert_util.cpp(152)                                                          mongo::invariantFailed+0x19a
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\mongo\executor\connection_pool.cpp(549)                                                  mongo::executor::ConnectionPool::SpecificPool::takeFromPool+0x187
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\mongo\executor\connection_pool.cpp(371)                                                  <lambda_dc0c64eb9bf7590ba921f557097b8044>::operator()+0x63
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\mongo\executor\connection_pool_asio.cpp(87)                                              <lambda_7724e334dbb065140334c2272bfff8f7>::operator()+0xfe
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\detail\impl\strand_service.hpp(61)        asio::detail::strand_service::dispatch<asio::detail::binder1<<lambda_7724e334dbb065140334c2272bfff8f7> & __ptr64,std::error_code> >+0x6b
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\handler_invoke_hook.hpp(69)               asio::asio_handler_invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_service::strand,<lambda_7724e334dbb065140334c2272bfff8f7>,asio::detail::is_continuation_if_running>,std::error_code>,<lambda_7724e334dbb065140334c2272bfff8f7> > >+0x37
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\detail\completion_handler.hpp(69)         asio::detail::completion_handler<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_service::strand,<lambda_7724e334dbb065140334c2272bfff8f7>,asio::detail::is_continuation_if_running>,std::error_code>,<lambda_7724e334dbb065140334c2272bfff8f7> > >::do_complete+0xf1
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\detail\impl\strand_service.hpp(87)        asio::detail::strand_service::dispatch<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_service::strand,<lambda_7724e334dbb065140334c2272bfff8f7>,asio::detail::is_continuation_if_running>,std::error_code>,<lambda_7724e334dbb065140334c2272bfff8f7> > >+0x14b
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\detail\wrapped_handler.hpp(233)           asio::detail::asio_handler_invoke<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_service::strand,<lambda_7724e334dbb065140334c2272bfff8f7>,asio::detail::is_continuation_if_running>,std::error_code>,asio::io_service::strand,<lambda_7724e334dbb065140334c2272bfff8f7>,asio::detail::is_continuation_if_running>+0xac
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\detail\wait_handler.hpp(71)               asio::detail::wait_handler<asio::detail::wrapped_handler<asio::io_service::strand,<lambda_7724e334dbb065140334c2272bfff8f7>,asio::detail::is_continuation_if_running> >::do_complete+0xdb
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\detail\impl\win_iocp_io_service.ipp(404)  asio::detail::win_iocp_io_service::do_one+0x2be
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\detail\impl\win_iocp_io_service.ipp(162)  asio::detail::win_iocp_io_service::run+0xbb
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\third_party\asio-asio-1-11-0\asio\include\asio\impl\io_service.ipp(61)                   asio::io_service::run+0x32
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      ...\src\mongo\executor\network_interface_asio.cpp(116)                                           <lambda_c44aeeec9ed6b3fe32a46d4f069876e0>::operator()+0x361
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      c:\program files (x86)\microsoft visual studio 12.0\vc\include\thr\xthread(187)                  std::_LaunchPad<std::_Bind<0,void,<lambda_c44aeeec9ed6b3fe32a46d4f069876e0> > >::_Go+0x1c
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      f:\dd\vctools\crt\crtw32\stdcpp\thr\threadcall.cpp(28)                                           _Call_func+0x29
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      f:\dd\vctools\crt\crtw32\startup\threadex.c(376)                                                 _callthreadstartex+0x25
CONTROL  [NetworkInterfaceASIO-0] mongod.exe      f:\dd\vctools\crt\crtw32\startup\threadex.c(359)                                                 _threadstartex+0xe7
CONTROL  [NetworkInterfaceASIO-0] kernel32.dll                                                                                                     BaseThreadInitThunk+0xd
-        [NetworkInterfaceASIO-0]
CONTROL  [NetworkInterfaceASIO-0] writing minidump diagnostic file C:\data\mci\src\mongod.2015-12-15T23-14-41.mdmp

The minidump is on the task, in case it's helpful.

Comment by Kaloian Manassiev [ 11/Dec/15 ]

Here is a link to the task itself. It happened on a private patch with some logging changes to the ASIO interface. I am pretty sure the logging changes are unrelated, but just FYI in case the line numbers differ.

Comment by Adam Midvidy [ 11/Dec/15 ]

kaloian.manassiev can you also add a link to the task (as opposed to just the logs, which will go away eventually)

Comment by Kaloian Manassiev [ 11/Dec/15 ]

Another occurrence: https://logkeeper.mongodb.org/build/566a1a13be07c469768fd807/test/566a1cdc9041306f248ffcf0#L2890

Comment by Adam Midvidy [ 19/Nov/15 ]

As the failure appears to only happen during mongos shutdown, we don't believe this to be a release blocker. Moving from 3.2.0-rc4 to 3.1 required.

Generated at Thu Feb 08 03:57:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.