[SERVER-33735] Failing to start mongos (aborting after invariant() failure) Created: 07/Mar/18  Updated: 02/Apr/18  Resolved: 11/Mar/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nikolay Malitsky Assignee: Ramon Fernandez Marina
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Good afternoon,

We are experiencing an issue with the deployment of a MongoDB 3.6 Sharded Cluster. Following the tutorial procedure ( https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/), a single-member Config Server Replica Set and a single-member Shard Replica
Set were successfully started and initiated. However, starting mongos failed (see below
the log file errors).

Thank you in advance,

2018-03-07T13:11:07.960-0500 I ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to xf03id1-mdb01:27019, took 0ms (4 connections now open to xf03id1-mdb01:27019)
2018-03-07T13:11:07.972-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document
2018-03-07T13:11:09.961-0500 I FTDC     [mongosMain] Initializing full-time diagnostic data capture with directory '/var/log/mongodb-1/mongos-conf.diagnostic.data'
2018-03-07T13:11:09.962-0500 F -        [mongosMain] Invariant failure globalInShutdownDeprecated() src/mongo/db/auth/user_cache_invalidator_job.cpp 116
2018-03-07T13:11:09.962-0500 F -        [mongosMain] 
 
***aborting after invariant() failure
 
2018-03-07T13:11:09.970-0500 F -        [mongosMain] Got signal: 6 (Aborted).



 Comments   
Comment by Nikolay Malitsky [ 12/Mar/18 ]

Ramon. Super. Thank you. Nikolay

On Sun, Mar 11, 2018 at 4:51 PM, Ramon Fernandez (JIRA) <jira@mongodb.org>

Comment by Ramon Fernandez Marina [ 11/Mar/18 ]

You can find various articles and videos on our website on using MongoDB for time series data that should help with your use case.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

I'm closing this ticket for the time being. If you have a reliable way to reproduce the original issue please comment here and we'll reopen for further investigation.

Regards,
Ramón.

Comment by Nikolay Malitsky [ 10/Mar/18 ]

Hi Ramon,

Again, thank you for the script, it was very helpful. Your procedure was
successfully deployed

on the three-server cluster.

Therefore, I was wondering if you could advise or provide some reference
addressing our primary

application concern.

In short, we are using MongoDB for storing time series of events and expect
that some collections

can reach the size of 500 M documents. Events are inserted with the bulk
operations. Therefore,

the primary concern is associated with index scalability. Could you suggest
some information

about constraints, limits? This information is important for making a
decision about switching from

flat collection to the chunk-based approach.

Best regards,

Nikolay

On Thu, Mar 8, 2018 at 6:53 PM, Nikolay Malitsky <nmalitsky@gmail.com>

Comment by Nikolay Malitsky [ 08/Mar/18 ]

Hi Ramon,

Thank you for the nice script. It helped resolve a few questions and
successfully run your scenario

on a single server. We need another day (tomorrow) to confirm it on our
multi-server cluster.

Best regards,

Nikolay

On Thu, Mar 8, 2018 at 11:12 AM, Nikolay Malitsky <nmalitsky@gmail.com>

Comment by Nikolay Malitsky [ 08/Mar/18 ]

Hi Ramon,

Thank you for the prompt response. This is important for us.

I will get back to you later today with a detailed description.

Best regards,

Nikolay

On Thu, Mar 8, 2018 at 8:08 AM, Ramon Fernandez (JIRA) <jira@mongodb.org>

Comment by Ramon Fernandez Marina [ 08/Mar/18 ]

I am unable to reproduce the error above. Below is a script to produce the setup you describe and at the end I end up with a on-shard cluster. Can you please provide a detailed description of what you're doing and how to reproduce this error?

Thanks,
Ramón.

#!/bin/bash
 
rm -rf *.log* cfgdb db?
 
mkdir cfgdb
mongod --configsvr --replSet cfg1 --dbpath cfgdb --bind_ip 0.0.0.0 --logpath cfg.log --fork
sleep 2
 
mongo --port 27019 --eval "rs.initiate()"
 
# ---
mkdir db1 db2 db3
mongod --shardsvr --replSet s1  --dbpath db1 --bind_ip 0.0.0.0 --port 30001 --logpath db1.log --fork
mongod --shardsvr --replSet s1  --dbpath db2 --bind_ip 0.0.0.0 --port 30002 --logpath db2.log --fork
mongod --shardsvr --replSet s1  --dbpath db3 --bind_ip 0.0.0.0 --port 30003 --logpath db3.log --fork
 
mongo --port 30001 --eval "rs.initiate()"
sleep 2
host="`hostname`"
mongo --port 30001 --eval "rs.add(\"$host:30002\")"
mongo --port 30001 --eval "rs.add(\"$host:30003\")"
 
# ---
mongos --port 27017 --configdb cfg1/$host:27019 --logpath mongos.log --fork
sleep 1
 
mongo --port 27017 --eval "sh.addShard(\"s1/$host:30001\")"
mongo --port 27017 --eval "sh.status()"

Generated at Thu Feb 08 04:34:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.