2018-05-15T11:42:28.711+0900 W SHARDING [replSetDistLockPinger] Failed to unlock lock with ts: 5afa45ae1d8f573b50374a9b and _id: config-movePrimary :: caused by :: NotMaster: Not primary while running findAndModify command on collection config.locks 2018-05-15T11:42:28.711+0900 W SHARDING [replSetDistLockPinger] Failed to unlock lock with ts: 5afa43561d8f573b503726eb and _id: config-movePrimary :: caused by :: NotMaster: Not primary while running findAndModify command on collection config.locks 2018-05-15T11:42:28.711+0900 W SHARDING [replSetDistLockPinger] Failed to unlock lock with ts: 5afa44821d8f573b503738bc and _id: config-movePrimary :: caused by :: NotMaster: Not primary while running findAndModify command on collection config.locks 2018-05-15T11:42:28.711+0900 W SHARDING [replSetDistLockPinger] Failed to unlock lock with ts: 5afa46da1d8f573b50375c67 and _id: config-movePrimary :: caused by :: NotMaster: Not primary while running findAndModify command on collection config.locks 2018-05-15T11:42:28.711+0900 W SHARDING [replSetDistLockPinger] Failed to unlock lock with ts: 5afa48061d8f573b50376e53 and _id: config-movePrimary :: caused by :: NotMaster: Not primary while running findAndModify command on collection config.locks 2018-05-15T11:05:08.583+0900 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Connecting to 10.106.100.192:53301 2018-05-15T11:05:08.654+0900 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-0-0] Successfully connected to 10.106.100.192:53301, took 71ms (1 connections now open to 10.106.100.192:53301) 2018-05-15T11:05:08.673+0900 I SHARDING [thread1912008] Refreshing chunks for collection config.system.sessions based on version 1|0||5a991811b41e8bf3aadbbac8 2018-05-15T11:05:08.894+0900 I SHARDING [ConfigServerCatalogCacheLoader-90] Refresh for collection config.system.sessions took 220 and found the collection is not sharded 2018-05-15T11:05:08.959+0900 F - [thread1912008] Invariant failure Hit a MONGO_UNREACHABLE! src/mongo/s/client/shard_local.cpp 63 2018-05-15T11:05:08.959+0900 F - [thread1912008] ***aborting after invariant() failure 2018-05-15T11:05:10.070+0900 I NETWORK [listener] connection accepted from 127.0.0.1:42128 #1912006 (49 connections now open) 2018-05-15T11:05:10.070+0900 I NETWORK [conn1912006] received client metadata from 127.0.0.1:42128 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.0" }, os: { type: "Linux", name: "CentOS Linux release 7.3.1611 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-514.el7.x86_64" } } 2018-05-15T11:05:10.090+0900 I ACCESS [conn1912006] Successfully authenticated as principal root on admin 2018-05-15T11:05:10.120+0900 I NETWORK [conn1912006] end connection 127.0.0.1:42128 (48 connections now open) 2018-05-15T11:05:10.191+0900 I NETWORK [listener] connection accepted from 127.0.0.1:42130 #1912007 (49 connections now open) 2018-05-15T11:05:10.191+0900 I NETWORK [conn1912007] received client metadata from 127.0.0.1:42130 conn: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.6.0" }, os: { type: "Linux", name: "CentOS Linux release 7.3.1611 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-514.el7.x86_64" } } 2018-05-15T11:05:10.208+0900 I ACCESS [conn1912007] Successfully authenticated as principal root on admin 2018-05-15T11:05:10.238+0900 I NETWORK [conn1912007] end connection 127.0.0.1:42130 (48 connections now open) 2018-05-15T11:05:10.673+0900 F - [thread1912008] Got signal: 6 (Aborted). 0x7f2342f41111 0x7f2342f40329 0x7f2342f4080d 0x7f233fe28370 0x7f233fa8d1d7 0x7f233fa8e8c8 0x7f234170c50c 0x7f2341e77f9c 0x7f2341fbb253 0x7f2341fbbafd 0x7f2341fbc17f 0x7f2341fc0cda 0x7f2341e36c6c 0x7f2341e43016 0x7f2341e28cbb 0x7f2341e290b5 0x7f23428923d3 0x7f2342895a14 0x7f2342895ecd 0x7f2341e27502 0x7f234288b6ef 0x7f234288c4a8 0x7f2341e011b2 0x7f23429563ea 0x7f2342956dac 0x7f2342957184 0x7f2342abb829 0x7f2342abba71 0x7f2341e000bd 0x7f234304f060 0x7f233fe20dc5 0x7f233fb4f76d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"7F2340D96000","o":"21AB111","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F2340D96000","o":"21AA329"},{"b":"7F2340D96000","o":"21AA80D"},{"b":"7F233FE19000","o":"F370"},{"b":"7F233FA58000","o":"351D7","s":"gsignal"},{"b":"7F233FA58000","o":"368C8","s":"abort"},{"b":"7F2340D96000","o":"97650C","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F2340D96000","o":"10E1F9C"},{"b":"7F2340D96000","o":"1225253","s":"_ZN5mongo19AsyncRequestsSender10RemoteData27resolveShardIdToHostAndPortERKNS_21ReadPreferenceSettingE"},{"b":"7F2340D96000","o":"1225AFD","s":"_ZN5mongo19AsyncRequestsSender16_scheduleRequestENS_8WithLockEm"},{"b":"7F2340D96000","o":"122617F","s":"_ZN5mongo19AsyncRequestsSender17_scheduleRequestsENS_8WithLockE"},{"b":"7F2340D96000","o":"122ACDA","s":"_ZN5mongo19AsyncRequestsSenderC2EPNS_16OperationContextEPNS_8executor12TaskExecutorENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorINS0_7RequestESaISD_EERKNS_21ReadPreferenceSettingENS_5Shard11RetryPolicyE"},{"b":"7F2340D96000","o":"10A0C6C","s":"_ZN5mongo14BatchWriteExec12executeBatchEPNS_16OperationContextERNS_10NSTargeterERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseEPNS_19BatchWriteExecStatsE"},{"b":"7F2340D96000","o":"10AD016","s":"_ZN5mongo13ClusterWriter5writeEPNS_16OperationContextERKNS_21BatchedCommandRequestEPNS_19BatchWriteExecStatsEPNS_22BatchedCommandResponseE"},{"b":"7F2340D96000","o":"1092CBB"},{"b":"7F2340D96000","o":"10930B5"},{"b":"7F2340D96000","o":"1AFC3D3"},{"b":"7F2340D96000","o":"1AFFA14"},{"b":"7F2340D96000","o":"1AFFECD","s":"_ZN5mongo18SessionsCollection9doRefreshERKNS_15NamespaceStringERKSt13unordered_setINS_20LogicalSessionRecordENS_24LogicalSessionRecordHashESt8equal_toIS5_ESaIS5_EESt8functionIFNS_6StatusENS_7BSONObjEEE"},{"b":"7F2340D96000","o":"1091502","s":"_ZN5mongo25SessionsCollectionSharded15refreshSessionsEPNS_16OperationContextERKSt13unordered_setINS_20LogicalSessionRecordENS_24LogicalSessionRecordHashESt8equal_toIS4_ESaIS4_EE"},{"b":"7F2340D96000","o":"1AF56EF","s":"_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE"},{"b":"7F2340D96000","o":"1AF64A8","s":"_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE"},{"b":"7F2340D96000","o":"106B1B2"},{"b":"7F2340D96000","o":"1BC03EA","s":"_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_"},{"b":"7F2340D96000","o":"1BC0DAC","s":"_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_"},{"b":"7F2340D96000","o":"1BC1184","s":"_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m"},{"b":"7F2340D96000","o":"1D25829","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"7F2340D96000","o":"1D25A71","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"7F2340D96000","o":"106A0BD"},{"b":"7F2340D96000","o":"22B9060"},{"b":"7F233FE19000","o":"7DC5"},{"b":"7F233FA58000","o":"F776D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.0", "gitVersion" : "a57d8e71e6998a2d0afde7edc11bd23e5661c915", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-514.el7.x86_64", "version" : "#1 SMP Tue Nov 22 16:42:41 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "7F2340D96000", "elfType" : 3, "buildId" : "0900573D611BFAA62156892DB8AAF9BC4331496B" }, { "b" : "7FFC834ED000", "elfType" : 3, "buildId" : "BF6B0C931C67B5BB2B0E80E07CBF73BAA5A466D2" }, { "b" : "7F2340959000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "FE7AE845A123A3DFC0FDC2408BCBC2BA8B61B158" }, { "b" : "7F2340751000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "82E77ADE22BC9FFF8D3458BD37331E7EDF174C28" }, { "b" : "7F234054D000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "C5F560504E1AF52E29679C3B52FF11121015D6BB" }, { "b" : "7F234024B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "721C7CC9488EFA25F83B48AF713AB27DBE48EF3E" }, { "b" : "7F2340035000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "408B46E291B2D4C9612E27C0509D165D7E186D40" }, { "b" : "7F233FE19000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C3DEB1FA27CD0C1C3CC575B944ABACBA0698B0F2" }, { "b" : "7F233FA58000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "1CC0E171FD5E7E28A6BB49667AFBD730CDBF22A0" }, { "b" : "7F2340B73000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "0874508AA13D28E3F48637C1D5BF067BA8D9FD3A" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f2342f41111] mongod(+0x21AA329) [0x7f2342f40329] mongod(+0x21AA80D) [0x7f2342f4080d] libpthread.so.0(+0xF370) [0x7f233fe28370]