2016-01-29T20:38:15.122+0400 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-3-0] Successfully connected to dj-mrs-master.tumo.lab:27017
2016-01-29T20:38:18.724+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:38:26.725+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:38:34.725+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:38:42.724+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:38:50.727+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:38:58.725+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:06.724+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:14.732+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:16.084+0400 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to gm-central-mrs-slave:27017
2016-01-29T20:39:16.198+0400 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to sk-central-mrs-slave:27017
2016-01-29T20:39:22.725+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:30.725+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:32.827+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:38.729+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:46.724+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:39:54.724+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:40:02.725+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:40:10.726+0400 I ACCESS   [conn79] Successfully authenticated as principal root on admin
2016-01-29T20:40:15.132+0400 I -        [NetworkInterfaceASIO-TaskExecutorPool-2-0] Invariant failure _checkedOutPool.empty() src/mongo/executor/connection_pool.cpp 570
2016-01-29T20:40:15.132+0400 I -        [NetworkInterfaceASIO-TaskExecutorPool-2-0] 

***aborting after invariant() failure


2016-01-29T20:40:15.182+0400 F -        [NetworkInterfaceASIO-TaskExecutorPool-2-0] Got signal: 6 (Aborted).

 0xc31fc2 0xc31129 0xc31932 0x7efda9d56340 0x7efda99b7cc9 0x7efda99bb0d8 0xbbb1db 0x9d5783 0x9d7f9c 0x9d8b17 0x9d9281 0xc4dc61 0xc4de81 0xc5201f 0x9e2445 0x7efdaa531a40 0x7efda9d4e182 0x7efda9a7b47d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"831FC2"},{"b":"400000","o":"831129"},{"b":"400000","o":"831932"},{"b":"7EFDA9D46000","o":"10340"},{"b":"7EFDA9981000","o":"36CC9"},{"b":"7EFDA9981000","o":"3A0D8"},{"b":"400000","o":"7BB1DB"},{"b":"400000","o":"5D5783"},{"b":"400000","o":"5D7F9C"},{"b":"400000","o":"5D8B17"},{"b":"400000","o":"5D9281"},{"b":"400000","o":"84DC61"},{"b":"400000","o":"84DE81"},{"b":"400000","o":"85201F"},{"b":"400000","o":"5E2445"},{"b":"7EFDAA480000","o":"B1A40"},{"b":"7EFDA9D46000","o":"8182"},{"b":"7EFDA9981000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.2.1", "gitVersion" : "a14d55980c2cdc565d4704a7e3ad37e4e535c1b2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.19.0-47-generic", "version" : "#53~14.04.1-Ubuntu SMP Mon Jan 18 16:09:14 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "13702C8A170E2A8D9CB29118516D8D3C99083030" }, { "b" : "7FFC8C955000", "elfType" : 3, "buildId" : "A7B0B18F330161D11A1B8E085900D9E8244369FF" }, { "b" : "7EFDAAF6B000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7EFDAAB90000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7EFDAA988000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7EFDAA784000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7EFDAA480000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }, { "b" : "7EFDAA17A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7EFDA9F64000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7EFDA9D46000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7EFDA9981000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7EFDAB1CA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc31fc2]
 mongos(+0x831129) [0xc31129]
 mongos(+0x831932) [0xc31932]
 libpthread.so.0(+0x10340) [0x7efda9d56340]
 libc.so.6(gsignal+0x39) [0x7efda99b7cc9]
 libc.so.6(abort+0x148) [0x7efda99bb0d8]
 mongos(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0xbbb1db]
 mongos(_ZN5mongo8executor14ConnectionPool12SpecificPool8shutdownEv+0x2A3) [0x9d5783]
 mongos(+0x5D7F9C) [0x9d7f9c]
 mongos(+0x5D8B17) [0x9d8b17]
 mongos(+0x5D9281) [0x9d9281]
 mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x2F1) [0xc4dc61]
 mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xC1) [0xc4de81]
 mongos(_ZN4asio10io_service3runEv+0x2F) [0xc5201f]
 mongos(+0x5E2445) [0x9e2445]
 libstdc++.so.6(+0xB1A40) [0x7efdaa531a40]
 libpthread.so.0(+0x8182) [0x7efda9d4e182]
 libc.so.6(clone+0x6D) [0x7efda9a7b47d]
-----  END BACKTRACE  -----
2016-02-01T10:28:59.408+0400 I CONTROL  [main] ***** SERVER RESTARTED *****
2016-02-01T10:28:59.420+0400 I CONTROL  [main] ** WARNING: You are running this process as the root user, which is not recommended.
2016-02-01T10:28:59.420+0400 I CONTROL  [main] 
2016-02-01T10:28:59.451+0400 I SHARDING [mongosMain] MongoS version 3.2.1 starting: pid=21899 port=27017 64-bit host=dj-web02.tumo.lab (--help for usage)
2016-02-01T10:28:59.451+0400 I CONTROL  [mongosMain] db version v3.2.1
2016-02-01T10:28:59.451+0400 I CONTROL  [mongosMain] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b2
2016-02-01T10:28:59.451+0400 I CONTROL  [mongosMain] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-02-01T10:28:59.451+0400 I CONTROL  [mongosMain] allocator: tcmalloc
2016-02-01T10:28:59.451+0400 I CONTROL  [mongosMain] modules: none
2016-02-01T10:28:59.451+0400 I CONTROL  [mongosMain] build environment:
2016-02-01T10:28:59.451+0400 I CONTROL  [mongosMain]     distmod: ubuntu1404
2016-02-01T10:28:59.452+0400 I CONTROL  [mongosMain]     distarch: x86_64
2016-02-01T10:28:59.452+0400 I CONTROL  [mongosMain]     target_arch: x86_64
2016-02-01T10:28:59.452+0400 I CONTROL  [mongosMain] options: { config: "/etc/mongos.conf", net: { port: 27017 }, processManagement: { fork: true }, security: { keyFile: "/etc/mongodb_keyfile" }, sharding: { configDB: "mcfg/dj-cfg.tumo.lab:27019,mcfg01.tumo.lab:27019,mcfg02.tumo.lab:27019,mcfg03.tumo.lab:27019" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongos.log" } }
2016-02-01T10:28:59.452+0400 I SHARDING [mongosMain] Updating config server connection string to: mcfg/dj-cfg.tumo.lab:27019,mcfg01.tumo.lab:27019,mcfg02.tumo.lab:27019,mcfg03.tumo.lab:27019
2016-02-01T10:28:59.452+0400 I NETWORK  [mongosMain] Starting new replica set monitor for mcfg/dj-cfg.tumo.lab:27019,mcfg01.tumo.lab:27019,mcfg02.tumo.lab:27019,mcfg03.tumo.lab:27019
2016-02-01T10:28:59.452+0400 I NETWORK  [ReplicaSetMonitorWatcher] starting
2016-02-01T10:28:59.455+0400 I SHARDING [thread1] creating distributed lock ping thread for process dj-web02.tumo.lab:27017:1454308139:1283860503 (sleeping for 30000ms)
2016-02-01T10:28:59.512+0400 I NETWORK  [replSetDistLockPinger] changing hosts to mcfg/dj-cfg:27019,gm-cfg:27019,mcfg01:27019,mcfg02:27019,mcfg03:27019,sk-cfg:27019 from mcfg/dj-cfg.tumo.lab:27019,mcfg01.tumo.lab:27019,mcfg02.tumo.lab:27019,mcfg03.tumo.lab:27019
2016-02-01T10:28:59.512+0400 I SHARDING [replSetDistLockPinger] Updating config server connection string to: mcfg/dj-cfg:27019,gm-cfg:27019,mcfg01:27019,mcfg02:27019,mcfg03:27019,sk-cfg:27019
2016-02-01T10:28:59.626+0400 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to mcfg01:27019
2016-02-01T10:28:59.629+0400 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to mcfg01:27019
2016-02-01T10:28:59.650+0400 I NETWORK  [mongosMain] Starting new replica set monitor for central/central-mrs-master:27017,central-mrs-slave:27017,dj-central-mrs-slave:27017,gm-central-mrs-slave:27017,sk-central-mrs-slave:27017
2016-02-01T10:28:59.650+0400 I NETWORK  [mongosMain] Starting new replica set monitor for dilijan/dj-mrs-master.tumo.lab:27017,evn-dj-mrs-slave:27017
2016-02-01T10:28:59.651+0400 I NETWORK  [mongosMain] Starting new replica set monitor for gyumri/gm-mrs-master:27017,gm-mrs-slave:27017
2016-02-01T10:28:59.651+0400 I NETWORK  [mongosMain] Starting new replica set monitor for stepanakert/evn-sk-mrs-slave:27017,sk-mrs-master:27017
2016-02-01T10:28:59.651+0400 I NETWORK  [mongosMain] Starting new replica set monitor for yerevan/evn-mrs-master:27017,evn-mrs-slave:27017
2016-02-01T10:28:59.712+0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed findAndModify query predicate didn't match any lock document
2016-02-01T10:28:59.859+0400 I NETWORK  [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-02-01T10:28:59.859+0400 I SHARDING [Balancer] about to contact config servers and shards
2016-02-01T10:28:59.915+0400 I NETWORK  [mongosMain] waiting for connections on port 27017
2016-02-01T10:28:59.968+0400 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to central-mrs-master:27017
2016-02-01T10:29:00.047+0400 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to dj-mrs-master.tumo.lab:27017
2016-02-01T10:29:00.192+0400 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to gm-mrs-master:27017
2016-02-01T10:29:00.276+0400 I NETWORK  [mongosMain] connection accepted from 10.44.20.221:33625 #1 (1 connection now open)
2016-02-01T10:29:00.437+0400 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to sk-mrs-master:27017
2016-02-01T10:29:00.551+0400 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to evn-mrs-master:27017
2016-02-01T10:29:00.555+0400 I SHARDING [Balancer] config servers and shards contacted successfully