[SERVER-38271] mongo db is crashing Created: 27/Nov/18  Updated: 08/Dec/18  Resolved: 07/Dec/18

Status: Closed
Project: Core Server
Component/s: GridFS, Querying, Storage
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Mamta Kumari Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Image20181127194805.png    
Operating System: ALL
Participants:

 Description   

I am uploading 9-11 mb file and storing in mongo db in fs.chunks through api test case. in first 5-7 attempts it will work fine after that mongo db get shut down.

 

err is comming while running this command 
18-11-27T10:57:08.278+0000 I COMMAND [conn46] command abc_itt.fs.chunks command: insert { insert: "fs.chunks", ordered: true, lsid:

{ id: UUID("04d57a84-386e-4331-8ede-7f0474a3bdb5") }

, $db: "abc_itt" } ninserted:1 keysInserted:2 numYields:0 reslen:44 locks:{ Global: { acquireCount:

{ r: 1, w: 1 }

}, Database: { acquireCount:

{ w: 1 }

}, Collection: { acquireCount:

{ w: 1 }

} } protocol:op_query 427ms

in console i m getting "{"message":"getClientDb: Error - ","name":"MongoNetworkError","stack":"MongoNetworkError: failed to reconnect after 30 attempts with interval 1000 ms\n at Connection.<anonymous> (/opt/mount1/cartos/main/node_modules/mongodb-core/lib/connection/pool.js:363:13)\n at emitTwo (events.js:106:13)\n at Connection.emit (events.js:191:7)\n at Socket.<anonymous> (/opt/mount1/cartos/main/node_modules/mongodb-core/lib/connection/connection.js:245:50)\n at Socket.g (events.js:292:16)\n at emitOne (events.js:96:13)\n at Socket.emit (events.js:188:7)\n at emitErrorNT (net.js:1281:8)\n at combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickDomainCallback (internal/process/next_tick.js:128:9)","errorLabels":["TransientTransactionError"],"_error_callsites":[{},{},{},{},{},{},{},{},{},{}],"level":"error","timestamp":"2018-11-27T09:15:00.566Z"}"

 

then "{"message":"getClientDb: Error - ","name":"MongoError","stack":"MongoError: Topology was destroyed\n at nextFunction (/opt/mount1/cartos/main/node_modules/mongodb-core/lib/cursor.js:603:27)\n at Cursor.next (/opt/mount1/cartos/main/node_modules/mongodb-core/lib/cursor.js:824:3)\n at Cursor.next (/opt/mount1/cartos/main/node_modules/mongodb/lib/cursor.js:211:36)\n at fetchDocs (/opt/mount1/cartos/main/node_modules/mongodb/lib/operations/cursor_ops.js:211:12)\n at toArray (/opt/mount1/cartos/main/node_modules/mongodb/lib/operations/cursor_ops.js:241:3)\n at executeOperation (/opt/mount1/cartos/main/node_modules/mongodb/lib/utils.js:420:24)\n at Cursor.toArray (/opt/mount1/cartos/main/node_modules/mongodb/lib/cursor.js:818:10)\n at MongoNativeClientManager.getConnection (/opt/mount1/cartos/main/dbLayer/connectionManager/mongoNativeClientManager.js:22:88)\n at BucketApiStorageService.save (/opt/mount1/cartos/main/server/documentStorage/bucketApiStorageService.js:19:29)\n at Object.savedActualImg (/opt/mount1/cartos/main/server/media/MediaHelper.js:58:38)\n at /opt/mount1/cartos/main/node_modules/async/lib/async.js:580:25\n at /opt/mount1/cartos/main/node_modules/async/lib/async.js:122:13\n at _each (/opt/mount1/cartos/main/node_modules/async/lib/async.js:46:13)\n at Object.async.each (/opt/mount1/cartos/main/node_modules/async/lib/async.js:121:9)\n at _parallel (/opt/mount1/cartos/main/node_modules/async/lib/async.js:579:20)\n at Object.async.parallel (/opt/mount1/cartos/main/node_modules/async/lib/async.js:595:9)","_error_callsites":[{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{}],"level":"error","timestamp":"2018-11-27T09:15:18.207Z"}"

 

and mongo db shuts down, need to again restart then same thing happening

 

I have 1 tier architecture on server with 8 GB ram 4 cpu

 



 Comments   
Comment by Mamta Kumari [ 08/Dec/18 ]

Hello Danny
Thanks for your all support.

Comment by Danny Hatcher (Inactive) [ 07/Dec/18 ]

Hello Mamta,

In the syslog you provided I can see two different mongod processes, a MySQL process, and an expensive Java process. I also see at one point that there was a Vi process that had to be OOM-killed for taking up too much memory. As it seems that that this server has many different programs contending for memory, all getting OOM-killed at various points, I have to conclude that there is no problem with mongod in this instance and that the servers are simply fully-loaded. We strongly recommend reserving an appropriate amount of resources for your mongod process via dedicated servers or methods like cgroups.

I do not see anything to indicate a bug in the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

See also our Technical Support page for additional support resources.

Thank you,

Danny

Comment by Danny Hatcher (Inactive) [ 06/Dec/18 ]

Hello Mamta,

From the information that you have provided thus far, we can see that the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.

If you are experiencing the problem on another server, could you provide the mongod logs, "diagnostic.data" folder, and syslog from that server to the Secure Uploader?

Thank you,

Danny

Comment by Mamta Kumari [ 06/Dec/18 ]

Hello Danny
this is one tier architecture but we are facing the same issue on another server too. there we are running only one mongod process per server.

and I checked there are not  any other intensive processes running on this server
Last log which we are getting is this

warning: log line attempted (2464kB) over max size (10kB), printing beginning and end ... command abc_itt.asset command: getMore { getMore: 17479308266895, collection: "asset", batchSize: 1000, lsid: { id: UUID("5ba1ed6d-9398-4200-8b83-a76ee95f77c6") }, $db: "abc_itt" } originatingCommand: { find: "asset", filter: { _id: { $in: [ ObjectId('5b7bb5b7c055ba33a62bc3fa'), ObjectId('5b7bb5b7c055ba33a62bc3fb'), ObjectId('5b7bb5b7c055ba33a62bc3fc'), ObjectId('5b7bb5b7c055ba33a62bc3fd'), ObjectId('5b7bb5b7c055ba33a62bc3fe'), ObjectId('5b7bb5b7c055ba33a62bc3ff'), ObjectId('5b7bb5b7c055ba33a62bc400'), ObjectId('5b7bb5b7c055ba33a62bc401'), ObjectId('5b7bb5b7c055ba33a62bc402'), ObjectId('5b7bb5b7c055ba33a62bc403'), ObjectId('5b7bb5b7c055ba33a62bc404'), ObjectId('5b7bb5b7c055ba33a62bc405'), ObjectId('5b7bb5b7c055ba33a62bc406'), ObjectId('5b7bb5b7c055ba33a62bc407'), ObjectId('5b7bb5b7c055ba33a62bc408'), ObjectId('5b7bb5b7c055ba33a62bc409'), ObjectId('5b7bb5b7c055ba33a62bc40a'), ObjectId('5b7bb5b7c055ba33a62bc40b'), ObjectId('5b7bb5b7c055ba33a62bc40c'), ObjectId('5b7bb5b7c055ba33a62bc40d'), ObjectId('5b7bb5b7c055ba33a62bc40e'), ObjectId('5b7bb5b7c055ba33a62bc40f'), ObjectId('5b7bb5b7c055ba33a62bc410'), ObjectId('5b7bb5b7c055ba33a62bc411'), ObjectId('5b7bb5b7c055ba33a62bc412'), ObjectId('5b7bb5b7c055ba33a62bc413'), ObjectId('5b7bb5b7c055ba33a62bc414'), ObjectId('5b7bb5b7c055ba33a62bc415'), ObjectId('5b7bb5b7c055ba33a62bc416'), ObjectId('5b7bb5b7c055ba33a62bc417'), ObjectId('5b7bb5b7c055ba33a62bc418'), ObjectId('5b7bb5b7c055ba33a62bc419'), ObjectId('5b7bb5b7c055ba33a62bc41a'), ObjectId('5b7bb5b7c055ba33a62bc41b'), ObjectId('5b7bb5b7c055ba33a62bc41c'), ObjectId('5b7bb5b7c055ba33a62bc41d'), ObjectId('5b7bb5b7c055ba33a62bc41e'), ObjectId('5b7bb5b7c055ba33a62bc41f'), ObjectId('5b7bb5b7c055ba33a62bc420'), ObjectId('5b7bb5b7c055ba33a62bc421'), ObjectId('5b7bb5b7c055ba33a62bc422'), ObjectId('5b7bb5b7c055ba33a62bc423'), ObjectId('5b7bb5b7c055ba33a62bc424'), ObjectId('5b7bb5b7c055ba33a62bc425'), ObjectId('5b7bb5b7c055ba33a62bc426'), ObjectId('5b7bb5b7c055ba33a62bc427'), ObjectId('5b7bb5b7c055ba33a62bc428'), ObjectId('5b7bb5b7c055ba33a62bc429'), ObjectId('5b7bb5b7c055ba33a62bc42a'), ObjectId('5b7bb5b7c055ba33a62bc42b'), ObjectId('5b7bb5b7c055ba33a62bc42c'), ObjectId('5b7bb5b7c055ba33a62bc42d'), ObjectId('5b7bb5b7c055ba33a62bc42e'), ObjectId('5b7bb5b7c055ba33a62bc42f'), ObjectId('5b7bb5b7c055ba33a62bc430'), ObjectId('5b7bb5b7c055ba33a62bc431'), ObjectId('5b7bb5b7c055ba33a62bc432'), ObjectId('5b7bb5b7c055ba33a62bc433'), ObjectId('5b7bb5b7c055ba33a62bc434'), ObjectId('5b7bb5b7c055ba33a62bc435'), ObjectId('5b7bb5b7c055ba33a62bc436'), ObjectId('5b7bb5b7c055ba33a62bc437'), ObjectId('5b7bb5b7c055ba33a62bc438'), ObjectId('5b7bb5b7c055ba33a62bc439'), ObjectId('5b7bb5b7c055ba33a62bc43a'), ObjectId('5b7bb5b7c055ba33a62bc43b'), ObjectId('5b7bb5b7c055ba33a62bc43c'), ObjectId('5b7bb5b7c055ba33a62bc43d'), ObjectId('5b7bb5b7c055ba33a62bc43e'), ObjectId('5b7bb5b7c055ba33a62bc43f'), ObjectId('5b7bb5b7c055ba33a62bc440'), ObjectId('5b7bb5b7c055ba33a62bc441'), ObjectId('5b7bb5b7c055ba33a62bc442'), ObjectId('5b7bb5b7c055ba33a62bc443'), ObjectId('5b7bb5b7c055ba33a62bc444'), ObjectId('5b7bb5b7c055ba33a62bc445'), ObjectId('5b7bb5b7c055ba33a62bc446'), ObjectId('5b7bb5b7c055ba33a62bc447'), ObjectId('5b7bb5b7c055ba33a62bc448'), ObjectId('5b7bb5b7c055ba33a62bc449'), ObjectId('5b7bb5b7c055ba33a62bc44a'), ObjectId('5b7bb5b7c055ba33a62bc44b'), ObjectId('5b7bb5b7c055ba33a62bc44c'), ObjectId('5 .......... Id('5b7bb67f23df0333f580ce9f'), ObjectId('5b7bb67f23df0333f580cea0'), ObjectId('5b7bb67f23df0333f580cea1'), ObjectId('5b7bb67f23df0333f580cea2'), ObjectId('5b7bb67f23df0333f580cea3'), ObjectId('5b7bb67f23df0333f580cea4'), ObjectId('5b7bb67f23df0333f580cea5'), ObjectId('5b7bb67f23df0333f580cea6'), ObjectId('5b7bb67f23df0333f580cea7'), ObjectId('5b7bb67f23df0333f580cea8'), ObjectId('5b7bb67f23df0333f580cea9'), ObjectId('5b7bb68023df0333f580ceaa'), ObjectId('5b7bb68023df0333f580ceab'), ObjectId('5b7bb68023df0333f580ceac'), ObjectId('5b7bb68023df0333f580cead'), ObjectId('5b7bb68023df0333f580ceae'), ObjectId('5b7bb68023df0333f580ceaf'), ObjectId('5b7bb68023df0333f580ceb0'), ObjectId('5b7bb68023df0333f580ceb1'), ObjectId('5b7bb68023df0333f580ceb2'), ObjectId('5b7bb68023df0333f580ceb3'), ObjectId('5b7bb68023df0333f580ceb7'), ObjectId('5b7bb68023df0333f580ceb8'), ObjectId('5b7bb68023df0333f580ceb9'), ObjectId('5b7bb68023df0333f580ceba'), ObjectId('5b7bb68023df0333f580cebb'), ObjectId('5b7bb68023df0333f580cedc'), ObjectId('5b7bb68023df0333f580cee2'), ObjectId('5b7bb68023df0333f580cee5'), ObjectId('5b7bb68023df0333f580cee6'), ObjectId('5b7bb68023df0333f580cee7'), ObjectId('5b7bb68023df0333f580cfcc'), ObjectId('5b7bb68023df0333f580cfcf'), ObjectId('5b7bb68023df0333f580d282'), ObjectId('5b7bb68023df0333f580d285'), ObjectId('5b7bb68023df0333f580d286'), ObjectId('5b7bb68023df0333f580d287'), ObjectId('5b7bb68023df0333f580d6f2'), ObjectId('5b7bb68023df0333f580d6f5'), ObjectId('5b7bb68023df0333f580d6f6'), ObjectId('5b7bb68023df0333f580d6f9'), ObjectId('5b7bb68023df0333f580d6fa'), ObjectId('5b7bb68023df0333f580d736'), ObjectId('5b7bb68023df0333f580d739'), ObjectId('5b7bb68023df0333f580d73a'), ObjectId('5b7bb68023df0333f580d73b'), ObjectId('5b7bb68023df0333f580d73e'), ObjectId('5b7bb68023df0333f580d73f'), ObjectId('5b7bb68023df0333f580d7a7'), ObjectId('5b7bb68023df0333f580d7aa'), ObjectId('5b7bb68023df0333f580d7ab'), ObjectId('5b7bb68023df0333f580d7af'), ObjectId('5b7bb68023df0333f580d7b2'), ObjectId('5b7bb68023df0333f580d7b3'), ObjectId('5b7bb68023df0333f580d7b4'), ObjectId('5b7bb68023df0333f580d7b8'), ObjectId('5b7bb68023df0333f580d7bb'), ObjectId('5b7bb68023df0333f580daa4'), ObjectId('5b7bb68023df0333f580daa7'), ObjectId('5b7bb68023df0333f580daa8'), ObjectId('5b7bb68023df0333f580daa9'), ObjectId('5b7bb68023df0333f580daaa'), ObjectId('5b7bb68023df0333f580daab'), ObjectId('5b7bb68023df0333f580daac'), ObjectId('5b7bb68023df0333f580daad'), ObjectId('5b7bb68023df0333f580daae'), ObjectId('5b7bb68023df0333f580daaf'), ObjectId('5b7bb68023df0333f580dab0'), ObjectId('5b7bb68023df0333f580dab1'), ObjectId('5b7bb68023df0333f580dac2'), ObjectId('5b7bb68023df0333f580dac5'), ObjectId('5b7bb68023df0333f580dac6'), ObjectId('5b7bb68023df0333f580dac7'), ObjectId('5b7bb68023df0333f580dac8'), ObjectId('5b7bb68023df0333f580dac9'), ObjectId('5b7bb68023df0333f580daca'), ObjectId('5b7bb68023df0333f580dacb') ] },


$and: [ { inactive: { $exists: false } } ] }, returnKey: false, showRecordId: false, lsid: { id: UUID("5ba1ed6d-9398-4200-8b83-a76ee95f77c6") }, $db: "abc_itt" } planSummary: IXSCAN { _id: 1, originalMake: 1, originalModel: 1 } cursorid:17479308266895 keysExamined:22522 docsExamined:22521 numYields:176 nreturned:1000 reslen:2121482 locks:{ Global: { acquireCount: { r: 354 } }, Database: { acquireCount: { r: 177 } }, Collection: { acquireCount: { r: 177 } } } protocol:op_query 153ms

Can we do something about this

Comment by Danny Hatcher (Inactive) [ 05/Dec/18 ]

Hello Mamta,

From an analysis of the data, it appears that MongoDB's resident memory usage is relatively low and the majority of memory usage on the system is anon active. Can you confirm if there are any other intensive processes running on this server? We recommend reserving servers for just MongoDB and allocating only one mongod process per server.

Thank you,

Danny

Comment by Mamta Kumari [ 05/Dec/18 ]

Hello Danny
did you find something there?
Issue is increasing day by day

Comment by Mamta Kumari [ 04/Dec/18 ]

Hello Danny
I have uploaded the diagnostic.data folder
let me know if you find something there

thanks for your support

Comment by Danny Hatcher (Inactive) [ 03/Dec/18 ]

Hello Mamta,

It appears that the diagnostic.data folder is empty. Can you please try to upload the contents of that folder again using our Secure Uploader?

Thank you,

Danny

Comment by Danny Hatcher (Inactive) [ 03/Dec/18 ]

Hello Mamta,

I have created a Secure Uploader for any files that need to be uploaded as a result of this investigation. Please rest assured that these files can only be accessed by MongoDB engineers and will be deleted after some time.

I have uploaded what you sent me via email and will look through them to continue my analysis.

Thank you,

Danny

Comment by Mamta Kumari [ 03/Dec/18 ]

Hi Denny
both files sent you over mail

Thanks
Mamta

Comment by Danny Hatcher (Inactive) [ 03/Dec/18 ]

Hello Mamta,

However, if you would like us to look into it further, please provide the mongod log files covering the incident and the "diagnostic.data" folder located under the $dbpath.

Thank you,

Danny

Comment by Danny Hatcher (Inactive) [ 03/Dec/18 ]

Hello Mamta,

Unfortunately, I do not see any problems with MongoDB itself from what you've posted. If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server. I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.

Thus, I do not see anything to indicate a bug in the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

See also our Technical Support page for additional support resources.

Thank you,

Danny

Comment by Mamta Kumari [ 02/Dec/18 ]

hello Denny,

At first, when mongo db gets shut down I guess it shuts down hard, it is not printing anything.

when I restart... it is printing this

ommand config.$cmd command: update \{ update: "system.sessions", ordered: false, allowImplicitCollectionCreation: false, $db: "config" } numYields:0 reslen:97949 locks:{ Global: { acquireCount:
 
{ r: 13001, w: 13001 }
 
}, Database: { acquireCount:
 
{ w: 13000, W: 1 }
 
}, Collection: { acquireCount:
 
{ w: 13000 }
 
} } protocol:op_msg 230ms
 2018-08-21T07:23:30.591+0000 I COMMAND [conn112249] command abc_itt.levelEntityMap command: aggregate { aggregate: "levelEntityMap", pipeline: [ { $match: { $and: [ { levelId:
 
{ $in: [ ObjectId('5b7bb16572aa981c9c60faa4'), ObjectId('5b7bb16572aa981c9c60faa8'), ObjectId('5b7bb3d4e41c1e1ce30c68a6'), ObjectId('5b7bb43ffee7191ce9d853ae'), ObjectId('5b7bb49ffee7191ce9d853b3'), ObjectId('5b7bb67f23df0333f580cc2a'), ObjectId('5b7bb67f23df0333f580cc2f'), ObjectId('5b7bb67f23df0333f580cc33'), ObjectId('5b7bb67f23df0333f580ce7e'), ObjectId('5b7bb67f23df0333f580cc3b'), ObjectId('5b7bb68023df0333f580daca'), ObjectId('5b7bb68023df0333f580dacb') ] }
 
, transitionName: "transition1" }, returnKey: false, showRecordId: false, lsid: \{ id: UUID("bce2404e-0786-4b1d-9d54-88a8f8dcb7d7") }, $db: "abc_itt" } planSummary: IXSCAN \{ fromAssetId: 1 } keysExamined:0 docsExamined:0 cursorExhausted:1 numYields:0 nreturned:0 reslen:117 locks:{ Global: { acquireCount:
 
{ r: 2 }
 
}, Database: { acquireCount:
 
{ r: 1 }
 
}, Collection: { acquireCount:
 
{ r: 1 }
 
} } protocol:op_query 256ms
 2018-08-21T07:24:06.329+0000 I COMMAND [conn112196] {color:#4c9aff}warning: log line attempted (2464kB) over max size (10kB), printing beginning and end ... command cartos_itt.asset command: aggregate { aggregate: "asset", pipeline: [ { $match: { _id: { $in: [ ObjectId('5b7bb5b7c055ba33a62bc3fa'),{color} ObjectId('5b7bb5b7c055ba33a62bc3fb'), ObjectId('5b7bb5b7c055ba33a62bc3fc'), ObjectId('5b7bb5b7c055ba33a62bc3fd'), ObjectId('5b7bb5b7c055ba33a62bc3fe'), ObjectId('5b7bb5b7c055ba33a62bc3ff'), ObjectId('5b7bb5b7c055ba33a62bc400'), ObjectId('5b7bb5b7c055ba33a62bc401'), ObjectId('5b7bb5b7c055ba33a62bc402'), ObjectId('5b7bb5b7c055ba33a62bc403'), ObjectId('5b7bb5b7c055ba33a62bc404'), ObjectId('5b7bb5b7c055ba33a62bc405'), ObjectId('5b7bb5b7c055ba33a62bc406'), ObjectId('5b7bb5b7c055ba33a62bc407'), ObjectId('5b7bb5b7c055ba33a62bc408'), ObjectId('5b7bb5b7c055ba33a62bc409'), ObjectId('5b7bb5b7c055ba33a62bc40a'), ObjectId('5b7bb5b7c055ba33a62bc40b'), ObjectId('5b7bb5b7c055ba33a62bc40c'), ObjectId('5b7bb5b7c055ba33a62bc40d'), ObjectId('5b7bb5b7c055ba33a62bc40e'), ObjectId('5b7bb5b7c055ba33a62bc40f'), ObjectId('5b7bb5b7c055ba33a62bc410'), ObjectId('5b7bb5b7c055ba33a62bc411'), ObjectId('5b7bb5b7c055ba33a62bc412'), ObjectId('5b7bb5b7c055ba33a62bc413'), ObjectId('5b7bb5b7c055ba33a62bc414'), ObjectId('5b7bb5b7c055ba33a62bc415'), ObjectId('5b7bb5b7c055ba33a62bc416'), ObjectId('5b7bb5b7c055ba33a62bc417'), ObjectId('5b7bb5b7c055ba33a62bc418'), ObjectId('5b7bb5b7c055ba33a62bc419'), ObjectId('5b7bb5b7c055ba33a62bc41a'), ObjectId('5b7bb5b7c055ba33a62bc41b'), ObjectId('5b7bb5b7c055ba33a62bc41c'), ObjectId('5b7bb5b7c055ba33a62bc41d'), ObjectId('5b7bb5b7c055ba33a62bc41e'), ObjectId('5b7bb5b7c055ba33a62bc41f'), ObjectId('5b7bb5b7c055ba33a62bc420'), ObjectId('5b7bb5b7c055ba33a62bc421'), ObjectId('5b7bb5b7c055ba33a62bc422'), ObjectId('5b7bb5b7c055ba33a62bc423'), ObjectId('5b7bb5b7c055ba33a62bc424'), ObjectId('5b7bb5b7c055ba33a62bc425'), ObjectId('5b7bb5b7c055ba33a62bc426'), ObjectId('5b7bb5b7c055ba33a62bc427'), ObjectId('5b7bb5b7c055ba33a62bc428'), ObjectId('5b7bb5b7c055ba33a62bc429'), ObjectId('5b7bb5b7c055ba33a62bc42a'), ObjectId('5b7bb5b7c055ba33a62bc42b'), ObjectId('5b7bb5b7c055ba33a62bc42c'), ObjectId('5b7bb5b7c055ba33a62bc42d'), ObjectId('5b7bb5b7c055ba33a62bc42e'), ObjectId('5b7bb5b7c055ba33a62bc42f'), ObjectId('5b7bb5b7c055ba33a62bc430'), ObjectId('5b7bb5b7c055ba33a62bc431'), ObjectId('5b7bb5b7c055ba33a62bc432'), ObjectId('5b7bb5b7c055ba33a62bc433'), ObjectId('5b7bb5b7c055ba33a62bc434'), ObjectId('5b7bb5b7c055ba33a62bc435'), ObjectId('5b7bb5b7c055ba33a62bc436'), ObjectId('5b7bb5b7c055ba33a62bc437'), ObjectId('5b7bb5b7c055ba33a62bc438'), ObjectId('5b7bb5b7c055ba33a62bc439'), ObjectId('5b7bb5b7c055ba33a62bc43a'), ObjectId('5b7bb5b7c055ba33a62bc43b'), ObjectId('5b7bb5b7c055ba33a62bc43c'), ObjectId('5b7bb5b7c055ba33a62bc43d'), ObjectId('5b7bb5b7c055ba33a62bc43e'), ObjectId('5b7bb5b7c055ba33a62bc43f'), ObjectId('5b7bb5b7c055ba33a62bc440'), ObjectId('5b7bb5b7c055ba33a62bc441'), ObjectId('5b7bb5b7c055ba33a62bc442'), ObjectId('5b7bb5b7c055ba33a62bc443'), ObjectId('5b7bb5b7c055ba33a62bc444'), ObjectId('5b7bb5b7c055ba33a62bc445'), ObjectId('5b7bb5b7c055ba33a62bc446'), ObjectId('5b7bb5b7c055ba33a62bc447'), ObjectId('5b7bb5b7c055ba33a62bc448'), ObjectId('5b7bb5b7c055ba33a62bc449'), ObjectId('5b7bb5b7c055ba33a62bc44a'), ObjectId('5b7bb5b7c055ba33a62bc44b'), ObjectId('5b7bb5b7c055ba33a62bc44c'), ObjectId('5b7bb5b7c055ba33a62bc44d'), ObjectId('5b7bb5b7c055ba33a62bc44e'), ObjectId('5b7bb5b7c055ba33a62bc44f'), ObjectId('5b7bb5b7c055ba33a62bc450'), Obj .......... 5b7bb67f23df0333f580ce9e'), ObjectId('5b7bb67f23df0333f580ce9f'), ObjectId('5b7bb67f23df0333f580cea0'), ObjectId('5b7bb67f23df0333f580cea1'), ObjectId('5b7bb67f23df0333f580cea2'), ObjectId('5b7bb67f23df0333f580cea3'), ObjectId('5b7bb67f23df0333f580cea4'), ObjectId('5b7bb67f23df0333f580cea5'), ObjectId('5b7bb67f23df0333f580cea6'), ObjectId('5b7bb67f23df0333f580cea7'), ObjectId('5b7bb67f23df0333f580cea8'), ObjectId('5b7bb67f23df0333f580cea9'), ObjectId('5b7bb68023df0333f580ceaa'), ObjectId('5b7bb68023df0333f580ceab'), ObjectId('5b7bb68023df0333f580ceac'), ObjectId('5b7bb68023df0333f580cead'), ObjectId('5b7bb68023df0333f580ceae'), ObjectId('5b7bb68023df0333f580ceaf'), ObjectId('5b7bb68023df0333f580ceb0'), ObjectId('5b7bb68023df0333f580ceb1'), ObjectId('5b7bb68023df0333f580ceb2'), ObjectId('5b7bb68023df0333f580ceb3'), ObjectId('5b7bb68023df0333f580ceb7'), ObjectId('5b7bb68023df0333f580ceb8'), ObjectId('5b7bb68023df0333f580ceb9'), ObjectId('5b7bb68023df0333f580ceba'), ObjectId('5b7bb68023df0333f580cebb'), ObjectId('5b7bb68023df0333f580cedc'), ObjectId('5b7bb68023df0333f580cee2'), ObjectId('5b7bb68023df0333f580cee5'), ObjectId('5b7bb68023df0333f580cee6'), ObjectId('5b7bb68023df0333f580cee7'), ObjectId('5b7bb68023df0333f580cfcc'), ObjectId('5b7bb68023df0333f580cfcf'), ObjectId('5b7bb68023df0333f580d282'), ObjectId('5b7bb68023df0333f580d285'), ObjectId('5b7bb68023df0333f580d286'), ObjectId('5b7bb68023df0333f580d287'), ObjectId('5b7bb68023df0333f580d6f2'), ObjectId('5b7bb68023df0333f580d6f5'), ObjectId('5b7bb68023df0333f580d6f6'), ObjectId('5b7bb68023df0333f580d6f9'), ObjectId('5b7bb68023df0333f580d6fa'), ObjectId('5b7bb68023df0333f580d736'), ObjectId('5b7bb68023df0333f580d739'), ObjectId('5b7bb68023df0333f580d73a'), ObjectId('5b7bb68023df0333f580d73b'), ObjectId('5b7bb68023df0333f580d73e'), ObjectId('5b7bb68023df0333f580d73f'), ObjectId('5b7bb68023df0333f580d7a7'), ObjectId('5b7bb68023df0333f580d7aa'), ObjectId('5b7bb68023df0333f580d7ab'), ObjectId('5b7bb68023df0333f580d7af'), ObjectId('5b7bb68023df0333f580d7b2'), ObjectId('5b7bb68023df0333f580d7b3'), ObjectId('5b7bb68023df0333f580d7b4'), ObjectId('5b7bb68023df0333f580d7b8'), ObjectId('5b7bb68023df0333f580d7bb'), ObjectId('5b7bb68023df0333f580daa4'), ObjectId('5b7bb68023df0333f580daa7'), ObjectId('5b7bb68023df0333f580daa8'), ObjectId('5b7bb68023df0333f580daa9'), ObjectId('5b7bb68023df0333f580daaa'), ObjectId('5b7bb68023df0333f580daab'), ObjectId('5b7bb68023df0333f580daac'), ObjectId('5b7bb68023df0333f580daad'), ObjectId('5b7bb68023df0333f580daae'), ObjectId('5b7bb68023df0333f580daaf'), ObjectId('5b7bb68023df0333f580dab0'), ObjectId('5b7bb68023df0333f580dab1'), ObjectId('5b7bb68023df0333f580dac2'), ObjectId('5b7bb68023df0333f580dac5'), ObjectId('5b7bb68023df0333f580dac6'), ObjectId('5b7bb68023df0333f580dac7'), ObjectId('5b7bb68023df0333f580dac8'), ObjectId('5b7bb68023df0333f580dac9'), ObjectId('5b7bb68023df0333f580daca'), ObjectId('5b7bb68023df0333f580dacb') ] }, inactive: \{ $exists: false } } } ], cursor: {}, lsid: \{ id: UUID("4632b596-3c9a-46d1-aa7a-fc268d120493") }, $db: "cartos_itt" } planSummary: IXSCAN \{ _id: 1, originalMake: 1, originalModel: 1 } cursorid:7171132554142274859 keysExamined:33640 docsExamined:33640 numYields:264 nreturned:101 reslen:152458 locks:{ Global: { acquireCount:
 
{ r: 532 }
 
}, Database: { acquireCount:
 
{ r: 266 }
 
}, Collection: { acquireCount:
 
{ r: 266 }
 
} } protocol:op_query 609ms
 2018-08-21T07:24:06.723+0000 I COMMAND [conn112201] warning: log line attempted (2464kB) over max size (10kB), printing beginning and end ... command cartos_itt.levelEntityMap command: aggregate { aggregate: "levelEntityMap", pipeline: [ { $match: { entityId:
 
{ $in: [ ObjectId('5b7bb5b7c055ba33a62bc3fa'), ObjectId('5b7bb5b7c055ba33a62bc3fb'), ObjectId('5b7bb68023df0333f580dac9'), ObjectId('5b7bb68023df0333f580daca'), ObjectId('5b7bb68023df0333f580dacb') ] }
 
} }, { $unwind:
 
{ path: "$zoneIds" }
 
}, { $lookup:
 
{ from: "level", localField: "zoneIds", foreignField: "_id", as: "zone" }
 
}, { $unwind:
 
{ path: "$zone" }
 
}, { $group: { _id: "$entityId", tagId:
 
{ $push: "$zone.zoneTagId" }
 
} }, { $unwind:
 
{ path: "$tagId" }
 
}, { $lookup:
 
{ from: "tag", localField: "tagId", foreignField: "_id", as: "tag" }
 
}, { $unwind:
 
{ path: "$tag" }
 
} ], cursor: {}, lsid: \{ id: UUID("a231aa97-36f0-46f8-a42a-0942850ce1ab") }, $db: "cartos_itt" } planSummary: COLLSCAN keysExamined:0 docsExamined:87895 cursorExhausted:1 numYields:686 nreturned:0 reslen:113 locks:{ Global: { acquireCount:
 
{ r: 1384 }
 
}, Database: { acquireCount:
 
{ r: 692 }
 
}, Collection: { acquireCount:
 
{ r: 691 }
 
} } protocol:op_query 581ms
 2018-08-21T07:24:07.619+0000 I COMMAND [conn112198] warning: log line attempted (2464kB) over max size (10kB), printing beginning and end ... command cartos_itt.asset command: getMore \{ getMore:ObjectId('5b7bb68023df0333f580daca'), ObjectId('5b7bb68023df0333f580dacb') ] }, inactive: \{ $exists: false } } } ], cursor: {}, lsid: \{ id: UUID("4632b596-3c9a-46d1-aa7a-fc268d120493") }, $db: "cartos_itt" } planSummary: IXSCAN \{ _id: 1, originalMake: 1, originalModel: 1 } cursorid:7171132554142274859 keysExamined:22276 docsExamined:22275 numYields:174 nreturned:1000 reslen:2420869 locks:{ Global: { acquireCount:
 
{ r: 354 }
 
}, Database: { acquireCount:
 
{ r: 177 }
 
}, Collection: { acquireCount:
 
{ r: 176 }
 
} } protocol:op_query 174ms
 2018-08-21T07:24:25.123+0000 I COMMAND [conn112200] warning: log line attempted (2464kB) over max size (10kB), printing beginning and end ... command cartos_itt.asset command: find { find: "asset", filter: { _id:
 
{ $in: [ ObjectId('5b7bb5b7c055ba33a62bc3fa'), ObjectId('5b7bb5b7c055ba33a62bc3fb'), ObjectId('5b7bb5b7c055ba33a62bc3fc'), ObjectId('5b7bb5b7c055ba33a62bc3fd'), ObjectId('5b7bb5b7c055ba33a62bc3fe'), ObjectId('5b7bb5b7c055ba33a62bc3ff'), ObjectId('5b7bb5b7c055ba33a62bc400'), ObjectId('5b7bb5b7c055ba33a62bc401'), ObjectId('5b7bb5b7c055ba33a62bc402'), ObjectId('5b7bb5b7c055ba33a62bc403'), ObjectId('5b7bb5b7c055ba33a62bc404'), ObjectId('5b7bb5b7c055ba33a62bc405'), ObjectId('5b7bb5b7c055ba33a62bc406'), ObjectId('5b7bb5b7c055ba33a62bc407'), ObjectId('5b7bb5b7c055ba33a62bc408'), ObjectId('5b7bb5b7c055ba33a62bc409'), ObjectId('5b7bb5b7c055ba33a62bc40a'), ObjectId('5b7bb5b7c055ba33a62bc40b'), ObjectId('5b7bb5b7c055ba33a62bc40c'), ObjectId('5b7bb5b7c055ba33a62bc40d'), ObjectId('5b7bb5b7c055ba33a62bc40e'), ObjectId('5b7bb5b7c055ba33a62bc40f'), ObjectId('5b7bb5b7c055ba33a62bc410'), ObjectId('5b7bb5b7c055ba33a62bc411'), ObjectId('5b7bb5b7c055ba33a62bc412'), ObjectId('5b7bb5b7c055ba33a62bc413'), ObjectId('5b7bb5b7c055ba33a62bc414'), ObjectId('5b7bb5b7c055ba33a62bc415'), ObjectId('5b7bb5b7c055ba33a62bc416'), ObjectId('5b7bb5b7c055ba33a62bc417'), ObjectId('5b7bb5b7c055ba33a62bc418'), ObjectId('5b7bb5b7c055ba33a62bc419'), ObjectId('5b7bb5b7c055ba33a62bc41a'), ObjectId('5b7bb5b7c055ba33a62bc41b'), ObjectId('5b7bb5b7c055ba33a62bc41c'), ObjectId('5b7bb5b7c055ba33a62bc41d'), ObjectId('5b7bb5b7c055ba33a62bc41e'), ObjectId('5b7bb5b7c055ba33a62bc41f'), ObjectId('5b7bb5b7c055ba33a62bc420'), ObjectId('5b7bb5b7c055ba33a62bc421'), ObjectId('5b7bb5b7c055ba33a62bc422'), ObjectId('5b7bb5b7c055ba33a62bc423'), ObjectId('5b7bb5b7c055ba33a62bc424'), ObjectId('5b7bb5b7c055ba33a62bc425'), ObjectId('5b7bb5b7c055ba33a62bc426'), ObjectId('5b7bb5b7c055ba33a62bc427'), ObjectId('5b7bb5b7c055ba33a62bc428'), ObjectId('5b7bb5b7c055ba33a62bc429'), ObjectId('5b7bb5b7c055ba33a62bc42a'), ObjectId('5b7bb5b7c055ba33a62bc42b'), ObjectId('5b7bb5b7c055ba33a62bc42c'), ObjectId('5b7bb5b7c055ba33a62bc42d'), ObjectId('5b7bb5b7c055ba33a62bc42e'), ObjectId('5b7bb5b7c055ba33a62bc42f'), ObjectId('5b7bb5b7c055ba33a62bc430'), ObjectId('5b7bb5b7c055ba33a62bc431'), ObjectId('5b7bb5b7c055ba33a62bc432'), ObjectId('5b7bb5b7c055ba33a62bc433'), ObjectId('5b7bb5b7c055ba33a62bc434'), ObjectId('5b7bb5b7c055ba33a62bc435'), ObjectId('5b7bb5b7c055ba33a62bc436'), ObjectId('5b7bb5b7c055ba33a62bc437'), ObjectId('5b7bb5b7c055ba33a62bc438'), ObjectId('5b7bb5b7c055ba33a62bc439'), ObjectId('5b7bb5b7c055ba33a62bc43a'), ObjectId('5b7bb5b7c055ba33a62bc43b'), ObjectId('5b7bb5b7c055ba33a62bc43c'), ObjectId('5b7bb5b7c055ba33a62bc43d'), ObjectId('5b7bb5b7c055ba33a62bc43e'), ObjectId('5b7bb5b7c055ba33a62bc43f'), ObjectId('5b7bb5b7c055ba33a62bc440'), ObjectId('5b7bb5b7c055ba33a62bc441'), ObjectId('5b7bb5b7c055ba33a62bc442'), ObjectId('5b7bb5b7c055ba33a62bc443'), ObjectId('5b7bb5b7c055ba33a62bc444'), ObjectId('5b7bb5b7c055ba33a62bc445'), ObjectId('5b7bb5b7c055ba33a62bc446'), ObjectId('5b7bb5b7c055ba33a62bc447'), ObjectId('5b7bb5b7c055ba33a62bc448'), ObjectId('5b7bb5b7c055ba33a62bc449'), ObjectId('5b7bb5b7c055ba33a62bc44a'), ObjectId('5b7bb5b7c055ba33a62bc44b'), ObjectId('5b7bb5b7c055ba33a62bc44c'), ObjectId('5b7bb5b7c055ba33a62bc44d'), ObjectId('5b7bb5b7c055ba33a62bc44e'), ObjectId('5b7bb5b7c055ba33a62bc44f'), ObjectId('5b7bb5b7c055ba33a62bc450'), ObjectId('5b7bb5b7c055ba33a .......... ctId('5b7bb67f23df0333f580ce9f'), ObjectId('5b7bb67f23df0333f580cea0'), ObjectId('5b7bb67f23df0333f580cea1'), ObjectId('5b7bb67f23df0333f580cea2'), ObjectId('5b7bb67f23df0333f580cea3'), ObjectId('5b7bb67f23df0333f580cea4'), ObjectId('5b7bb67f23df0333f580cea5'), ObjectId('5b7bb67f23df0333f580cea6'), ObjectId('5b7bb67f23df0333f580cea7'), ObjectId('5b7bb67f23df0333f580cea8'), ObjectId('5b7bb67f23df0333f580cea9'), ObjectId('5b7bb68023df0333f580ceaa'), ObjectId('5b7bb68023df0333f580ceab'), ObjectId('5b7bb68023df0333f580ceac'), ObjectId('5b7bb68023df0333f580cead'), ObjectId('5b7bb68023df0333f580ceae'), ObjectId('5b7bb68023df0333f580ceaf'), ObjectId('5b7bb68023df0333f580ceb0'), ObjectId('5b7bb68023df0333f580ceb1'), ObjectId('5b7bb68023df0333f580ceb2'), ObjectId('5b7bb68023df0333f580ceb3'), ObjectId('5b7bb68023df0333f580ceb7'), ObjectId('5b7bb68023df0333f580ceb8'), ObjectId('5b7bb68023df0333f580ceb9'), ObjectId('5b7bb68023df0333f580ceba'), ObjectId('5b7bb68023df0333f580cebb'), ObjectId('5b7bb68023df0333f580cedc'), ObjectId('5b7bb68023df0333f580cee2'), ObjectId('5b7bb68023df0333f580cee5'), ObjectId('5b7bb68023df0333f580cee6'), ObjectId('5b7bb68023df0333f580cee7'), ObjectId('5b7bb68023df0333f580cfcc'), ObjectId('5b7bb68023df0333f580cfcf'), ObjectId('5b7bb68023df0333f580d282'), ObjectId('5b7bb68023df0333f580d285'), ObjectId('5b7bb68023df0333f580d286'), ObjectId('5b7bb68023df0333f580d287'), ObjectId('5b7bb68023df0333f580d6f2'), ObjectId('5b7bb68023df0333f580d6f5'), ObjectId('5b7bb68023df0333f580d6f6'), ObjectId('5b7bb68023df0333f580d6f9'), ObjectId('5b7bb68023df0333f580d6fa'), ObjectId('5b7bb68023df0333f580d736'), ObjectId('5b7bb68023df0333f580d739'), ObjectId('5b7bb68023df0333f580d73a'), ObjectId('5b7bb68023df0333f580d73b'), ObjectId('5b7bb68023df0333f580d73e'), ObjectId('5b7bb68023df0333f580d73f'), ObjectId('5b7bb68023df0333f580d7a7'), ObjectId('5b7bb68023df0333f580d7aa'), ObjectId('5b7bb68023df0333f580d7ab'), ObjectId('5b7bb68023df0333f580d7af'), ObjectId('5b7bb68023df0333f580d7b2'), ObjectId('5b7bb68023df0333f580d7b3'), ObjectId('5b7bb68023df0333f580d7b4'), ObjectId('5b7bb68023df0333f580d7b8'), ObjectId('5b7bb68023df0333f580d7bb'), ObjectId('5b7bb68023df0333f580daa4'), ObjectId('5b7bb68023df0333f580daa7'), ObjectId('5b7bb68023df0333f580daa8'), ObjectId('5b7bb68023df0333f580daa9'), ObjectId('5b7bb68023df0333f580daaa'), ObjectId('5b7bb68023df0333f580daab'), ObjectId('5b7bb68023df0333f580daac'), ObjectId('5b7bb68023df0333f580daad'), ObjectId('5b7bb68023df0333f580daae'), ObjectId('5b7bb68023df0333f580daaf'), ObjectId('5b7bb68023df0333f580dab0'), ObjectId('5b7bb68023df0333f580dab1'), ObjectId('5b7bb68023df0333f580dac2'), ObjectId('5b7bb68023df0333f580dac5'), ObjectId('5b7bb68023df0333f580dac6'), ObjectId('5b7bb68023df0333f580dac7'), ObjectId('5b7bb68023df0333f580dac8'), ObjectId('5b7bb68023df0333f580dac9'), ObjectId('5b7bb68023df0333f580daca'), ObjectId('5b7bb68023df0333f580dacb') ] }
 
, $and: [ { inactive:
 
{ $exists: false }
 
} ] }, returnKey: false, showRecordId: false, lsid: \{ id: UUID("5ba1ed6d-9398-4200-8b83-a76ee95f77c6") }, $db: "cartos_itt" } planSummary: IXSCAN \{ _id: 1, originalMake: 1, originalModel: 1 } cursorid:17479308266895 keysExamined:32913 docsExamined:32913 numYields:258 nreturned:101 reslen:152458 locks:{ Global: { acquireCount:
 
{ r: 518 }
 
}, Database: { acquireCount:
 
{ r: 259 }
 
}, Collection: { acquireCount:
 
{ r: 259 }
 
} } protocol:op_query 543ms
 2018-08-21T07:24:27.708+0000 I COMMAND [conn112217] warning: log line attempted (2464kB) over max size (10kB), printing beginning and end ... command cartos_itt.asset command: getMore { getMore: 17479308266895, collection: "asset", batchSize: 1000, lsid:
 
{ id: UUID("5ba1ed6d-9398-4200-8b83-a76ee95f77c6") }
 
, $db: "cartos_itt" } originatingCommand: { find: "asset", filter: { _id:
 
{ $in: [ ObjectId('5b7bb5b7c055ba33a62bc3fa'), ObjectId('5b7bb68023df0333f580daca'), ObjectId('5b7bb68023df0333f580dacb') ] }
 
, $and: [ { inactive:
 
{ $exists: false }
 
} ] }, returnKey: false, showRecordId: false, lsid: \{ id: UUID("5ba1ed6d-9398-4200-8b83-a76ee95f77c6") }, $db: "cartos_itt" } planSummary: IXSCAN \{ _id: 1, originalMake: 1, originalModel: 1 } cursorid:17479308266895 keysExamined:22522 docsExamined:22521 numYields:176 nreturned:1000 reslen:2121482 locks:{ Global: { acquireCount:
 
{ r: 354 }
 
}, Database: { acquireCount:
 
{ r: 177 }
 
}, Collection: { acquireCount:
 
{ r: 177 }
 
} } protocol:op_query 153ms
 2018-08-22T05:03:12.591+0000 I CONTROL [main] ***** SERVER RESTARTED *****
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] MongoDB starting : pid=11514 port=27018 dbpath=/opt/mount1/cartos/mongodb_3.6.3/data/ 64-bit host=itt-auto2-main.novalocal
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] db version v3.6.3
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] allocator: tcmalloc
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] modules: none
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] build environment:
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] distmod: rhel70
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] distarch: x86_64
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] target_arch: x86_64
 2018-08-22T05:03:12.627+0000 I CONTROL [initandlisten] options: { config: "/opt/mount1/cartos/mongodb_3.6.3/mongod.conf", net:
 
{ bindIp: "0.0.0.0", port: 27018 }
 
, processManagement: \{ fork: true, pidFilePath: "/opt/mount1/cartos/mongodb_3.6.3/mongod.pid" }, security: \{ authorization: "enabled" }, storage: { dbPath: "/opt/mount1/cartos/mongodb_3.6.3/data/", journal:
 
{ enabled: true }
 
}, systemLog: \{ destination: "file", logAppend: true, path: "/opt/mount1/cartos/mongodb_3.6.3/logs/mongod.log" } }
 *{color:#00875a}2018-08-22T05:03:12.670+0000 W - [initandlisten] Detected unclean shutdown - /opt/mount1/cartos/mongodb_3.6.3/data/mongod.lock is not empty.{color}*
 *{color:#00875a}2018-08-22T05:03:12.681+0000 I - [initandlisten] Detected data files in{color}* /opt/mount1/cartos/mongodb_3.6.3/data/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
 2018-08-22T05:03:12.681+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
 2018-08-22T05:03:12.681+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3399M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
 2018-08-22T05:03:15.354+0000 I STORAGE [initandlisten] WiredTiger message [1534914195:354271][11514:0x7f0fd737bb00], txn-recover: Main recovery loop: starting at 29/17676544
 2018-08-22T05:03:15.354+0000 I STORAGE [initandlisten] WiredTiger message [1534914195:354848][11514:0x7f0fd737bb00], txn-recover: Recovering log 29 through 30
 2018-08-22T05:03:15.638+0000 I STORAGE [initandlisten] WiredTiger message [1534914195:638344][11514:0x7f0fd737bb00], [file:index-13735-9106049674331342150.wt|file://index-13735-9106049674331342150.wt/], txn-recover: Recovering log 30 through 30
 2018-08-22T05:03:19.265+0000 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
 2018-08-22T05:03:19.265+0000 I CONTROL [initandlisten] 
 2018-08-22T05:03:19.266+0000 I CONTROL [initandlisten] 
 2018-08-22T05:03:19.266+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
 2018-08-22T05:03:19.266+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
 2018-08-22T05:03:19.266+0000 I CONTROL [initandlisten] 
 2018-08-22T05:03:19.266+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
 2018-08-22T05:03:19.266+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
 2018-08-22T05:03:19.266+0000 I CONTROL [initandlisten]

 

Thanks

Comment by Danny Hatcher (Inactive) [ 29/Nov/18 ]

Hello Mamta,

Can you please provide the shutdown message itself? The way MongoDB was stopped should help determine why it stopped in the first place. Please provide a snippet containing the last operation you provided up through the server restart. This snippet should include any shutdown-related messages.

Thank you,

Danny

Comment by Mamta Kumari [ 28/Nov/18 ]

Hi Danny
in mongod log the last operation  is :   delete { delete: "fs.chunks", ordered: true, lsid:

{ id: UUID("ba4c336a-1c6a-4c5a-ab21-857ac75449f1") }

, $db: "cartos_itt" } numYields:0 reslen:44 locks:{ Global: { acquireCount:

{ r: 6, w: 4 }

}, Database: { acquireCount:

{ r: 1, w: 4 }

}, Collection: { acquireCount:

{ r: 1, w: 4 }

} } protocol:op_query 729ms

after this mongod get shut down and printed nothing in the log. getLastError is also giving null. even though I am not able to get that particular error. 

we are inserting 35 mb file through gridfs in fs.chuncks. In the first 5-7 times it is inserting properly. after that, this issue is coming

in server " "{"message":"getClientDb: Error - ","name":"MongoNetworkError","stack":"MongoNetworkError: failed to reconnect after 30 attempts with interval 1000 ms\n at Connection.<anonymous> (/opt/mount1/cartos/main/node_modules/mongodb-core/lib/connection/pool.js:363:13)\n at emitTwo (events.js:106:13)\n at Connection.emit (events.js:191:7)\n at Socket.<anonymous> (/opt/mount1/cartos/main/node_modules/mongodb-core/lib/connection/connection.js:245:50)\n at Socket.g (events.js:292:16)\n at emitOne (events.js:96:13)\n at Socket.emit (events.js:188:7)\n at emitErrorNT (net.js:1281:8)\n at combinedTickCallback (internal/process/next_tick.js:80:11)\n at process._tickDomainCallback (internal/process/next_tick.js:128:9)","errorLabels":["TransientTransactionError"],"_error_callsites":[{},{},{},{},{},{},{},{},{},{}],"level":"error","timestamp":"2018-11-27T09:15:00.566Z"}"

*I am not able to find a way
*

 

Comment by Danny Hatcher (Inactive) [ 27/Nov/18 ]

Hello Mamta,

The screenshot that you posted does not contain the shutdown. Could you please provide any relevant error that occurs in the mongod logs prior to and including the shutdown?

Thank you,

Danny

Generated at Thu Feb 08 04:48:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.