[SERVER-9018] We meet mongo db crash several time Created: 19/Mar/13  Updated: 11/Jul/16  Resolved: 05/Apr/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.2.2
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Rui Huang Assignee: Michael Grundy
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File client-error.txt    
Operating System: ALL
Participants:

 Description   

upgrade to version 2.2.2. We have 2 node, 1 is master, the other is slave. Sometimes 1 node is crash, here is the crash log. After the crash, the other node is with high load in os level, thread number of mongo increased to around 2000, so linux is overload in system%.

Os is centos 6.3, with kernel 2.6.32-279.

Amazon Ec2 16G memory, 4 core.

Here is the log when crash:

Sat Mar 9 10:53:01 Got signal: 11 (Segmentation fault).

Sat Mar 9 10:53:01 Backtrace:
0xaffd31 0x558bb9 0x559142 0x7fd4b1428500 0x83d714 0x8420a0 0x844937 0x823598 0x9a34c2 0x9a2d4f 0xad35cd 0xb45ba9 0x7fd4b1420851 0x7fd4b0ce211d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x558bb9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559142]
/lib64/libpthread.so.0(+0xf500) [0x7fd4b1428500]
/usr/bin/mongod(_ZN5mongo11checkNoModsENS_7BSONObjE+0x34) [0x83d714]
/usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x3160) [0x8420a0]
/usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0x844937]
/usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x1228) [0x823598]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x292) [0x9a34c2]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x4f) [0x9a2d4f]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xad35cd]
/usr/bin/mongod() [0xb45ba9]
/lib64/libpthread.so.0(+0x7851) [0x7fd4b1420851]
/lib64/libc.so.6(clone+0x6d) [0x7fd4b0ce211d]



 Comments   
Comment by Michael Grundy [ 05/Apr/13 ]

Rui -

Thanks for the update. I'm going to close this ticket out.

In the future, filing tickets under the SERVER project is not the best way to get visibility for issues you encounter, as we use this project for tracking bugs and feature requests. If you have a support contract with 10gen, then filing commercial support tickets is your best option. If not, the 10gen development and support teams actively monitor posts to stackoverflow (http://stackoverflow.com/questions/tagged/mongodb) and to the mongo google group (https://groups.google.com/forum/?fromgroups#!forum/mongodb-user)

Thanks!
Mike

Comment by Rui Huang [ 04/Apr/13 ]

I attatched the client error data log, seems node.js drivers is filter part of error type, but still some error data forward to mongo server. Will check code with our dev engineer. Thank you so much to figure out it, will monitor to mongo server continually

Comment by Michael Grundy [ 04/Apr/13 ]

Hi Rui -

objcheck is stopping the insert and the replicas no longer run into trouble trying to insert a bad object into their copies. We now need to figure out why this is happening. Are you getting the errors back on the client side? Does the client code have writeconcern turned on? Writeconcern is a DB option

{w: 1}

, example:

new Db(new Server('localhost', 27017),

{w: 1}

)

Please make sure that is set on the client side and the client code is checking error conditions in the insert callback function.

Thanks!
Mike

Comment by Rui Huang [ 04/Apr/13 ]

Hi, Michael:

Thanks for that information. We open the --objcheck, now "Client Error: bad object in message" appear follow each bad type. So does the objcheck will stop the insert operation and avoid crash?

the log is like this, 1 bad type, then an correspondent bad object in message:

Thu Apr 4 13:14:44 [conn403892] Assertion: 13655:BSONElement: bad type 51
0xaffd31 0xac5eb9 0x57175e 0x7bbfc0 0x7bc17f
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x99) [0xac5eb9]
/usr/bin/mongod(_ZNK5mongo11BSONElement4sizeEi+0x21e) [0x57175e]
/usr/bin/mongod(_ZNK5mongo7BSONObj5validEv+0x60) [0x7bbfc0]
/usr/bin/mongod(_ZNK5mongo7BSONObj5validEv+0x21f) [0x7bc17f]
Thu Apr 4 13:14:44 [conn403892] Assertion: 10307:Client Error: bad object in message
0xaffd31 0xac5eb9 0x7aca2d 0x7b4274 0x5703f2 0xaedfc1 0x37fda07851 0x37fd2e811d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x99) [0xac5eb9]
/usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x75d) [0x7aca2d]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xbc4) [0x7b4274]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x5703f2]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xaedfc1]
/lib64/libpthread.so.0() [0x37fda07851]

Comment by Michael Grundy [ 02/Apr/13 ]

Could you restart all mongod processes with the --objcheck option? I suspect that a corrupt object may be causing the crash.

If a corrupt message comes in, mongod will log the message "Client Error: bad object in message", and will return error code 10307 to the client when requested. Seeing these messages in the mongod logs would validate this theory. Note that --objcheck introduces additional CPU overhead, but it will not be noticeable unless the client is transmitting very deeply-nested documents or very long arrays.

Also, please tar up the logs from all the nodes in this replica set and scp them to us via scp -P 722 filename.tgz sina@www.10gen.com: (when prompted for a password just hit enter).

Thanks!
Mike

Comment by Rui Huang [ 01/Apr/13 ]

the primary node crash again this morning, which is a little different place:

Mon Apr 1 03:01:58 Invalid access at address: 0x7e50b9077000 from thread: conn16020355

Mon Apr 1 03:01:58 Got signal: 11 (Segmentation fault).

Mon Apr 1 03:01:58 Backtrace:
0xaffd31 0x558bb9 0x559142 0x37fda0f500 0x7ac169 0x7ac238 0x7ac6e7 0x7b4274 0x5703f2 0xaedfc1 0x37fda07851 0x37fd2e811d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x558bb9]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559142]
/lib64/libpthread.so.0() [0x37fda0f500]
/usr/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x49) [0x7ac169]
/usr/bin/mongod(_ZN5mongo11insertMultiEbPKcRSt6vectorINS_7BSONObjESaIS3_EE+0x28) [0x7ac238]
/usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x417) [0x7ac6e7]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xbc4) [0x7b4274]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x5703f2]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xaedfc1]
/lib64/libpthread.so.0() [0x37fda07851]
/lib64/libc.so.6(clone+0x6d) [0x37fd2e811d]

Comment by Rui Huang [ 27/Mar/13 ]

we have 3 node in replica, 005 run arbiter. 003 and 004 is primary and secondary now, 003 is secondary before crash on Mar/9, so change 004 as secondary, then on Mar/22, 004 crash again. Write request was forward to Primary.

Seems the crash always happened in replica writer, our disk space is 4T, oplog size default to around 200G, and disk usage in /mnt/mongo/data is more than 700G already.

Comment by Michael Grundy [ 24/Mar/13 ]

Are the other nodes you refer to replicas of the crashing nodes we're discussing? The text "BSONElement: bad type <nn>" is generated on the server (or potentially on the client) when parsing a BSON document to determine its correct size). The number <nn> is in decimal. Your value of 109 (0x6D in hexadecimal) is not a legal value for a BSONElement type (as you can see from the documentation at http://bsonspec.org/#/specification ).

There are a couple of possibilities for how the code happened to be expecting a valid BSON type and instead found a 116 (0x74) and then a 109 (0x6D). The data might have been originally valid but was overwritten by a stray pointer. The database could have become corrupt by a disk error or by an unclean shutdown without journaling enabled. The byte could have been corrupted on the network by a broken network component. The 109 of 116 themselves might be valid data, but an earlier BSONElement had its size specified incorrectly leading to finding them where a type byte was expected.

Issues like SERVER-7005 and SERVER-6768 detail times where drivers, or profiling, have let invalid characters be inserted in indexes or field names (nulls and $ or . respectively). Invalid fields in the oplog maybe causing these issues.

Can you run db.coll.validate(true) from the mongo shell on that collection? This will walk through each document and inspect all data structures for corruption and validity. If the collection is large this may take a while and will cause effectively cause the system to wait to do anything (no writes will be allowed) until it is done.

I think it would be helpful to send us a backup of your database files or a bsondump. You can tgz them up and scp them to us via scp -P 722 filename.tgz sina@www.10gen.com: , we could also convert this ticket to a support one so that you can post sensitive data that only you and 10gen support will be able to see.

Thanks!

Comment by Rui Huang [ 23/Mar/13 ]

we use the newest version node-native-mongodb driver http://mongodb.github.com/node-mongodb-native/.
I check the backtrace and source code of mongo,it stop in the checkNoMods function of update.cpp,it is in UpdateResult _updateObjects function, line 199,417,*** call checkNoMods( updateobj ); and line 199's call is: "// this handles repl inserts".

for in the checkNoMods() function, reference BSONObjIterator i( o ), does mongo check the repl log with correct data format? I check another node of mongo cluster, which have frequently warning like:

Fri Mar 22 23:19:57 [conn1050730] Assertion: 10320:BSONElement: bad type 116
0xaffd31 0xac5eb9 0x57105b 0x7ac186 0x7ac238 0x7ac6e7 0x7b4274 0x5703f2 0xaedfc1 0x37fda07851 0x37fd2e811d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x99) [0xac5eb9]
/usr/bin/mongod(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x57105b]
/usr/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x66) [0x7ac186]
/usr/bin/mongod(_ZN5mongo11insertMultiEbPKcRSt6vectorINS_7BSONObjESaIS3_EE+0x28) [0x7ac238]
/usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x417) [0x7ac6e7]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xbc4) [0x7b4274]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x5703f2]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xaedfc1]
/lib64/libpthread.so.0() [0x37fda07851]
/lib64/libc.so.6(clone+0x6d) [0x37fd2e811d]
Fri Mar 22 23:19:57 [conn1050730] insert ios_cc.server_warns keyUpdates:0 exception: BSONElement: bad type 116 code:10320 locks(micros) w:3253 3ms
Fri Mar 22 23:19:57 [conn1050731] Assertion: 10320:BSONElement: bad type 109
0xaffd31 0xac5eb9 0x57105b 0x7ac186 0x7ac238 0x7ac6e7 0x7b4274 0x5703f2 0xaedfc1 0x37fda07851 0x37fd2e811d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x99) [0xac5eb9]
/usr/bin/mongod(_ZNK5mongo11BSONElement4sizeEv+0x1cb) [0x57105b]
/usr/bin/mongod(_ZN5mongo14checkAndInsertEPKcRNS_7BSONObjE+0x66) [0x7ac186]
/usr/bin/mongod(_ZN5mongo11insertMultiEbPKcRSt6vectorINS_7BSONObjESaIS3_EE+0x28) [0x7ac238]
/usr/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x417) [0x7ac6e7]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xbc4) [0x7b4274]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x5703f2]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xaedfc1]
/lib64/libpthread.so.0() [0x37fda07851]
/lib64/libc.so.6(clone+0x6d) [0x37fd2e811d]
Fri Mar 22 23:19:57 [conn1050731] insert ios_mw.server_warns keyUpdates:0 exception: BSONElement: bad type 109 code:10320 locks(micros) w:2487 2ms

Comment by Michael Grundy [ 23/Mar/13 ]

Thanks for the additional information. What driver and driver version is your application using?

Comment by Rui Huang [ 22/Mar/13 ]

analytics:PRIMARY> rs.status()
{
	"set" : "analytics",
	"date" : ISODate("2013-03-22T20:56:24Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "db-mongo-003.int.funzio.com:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 31518,
			"optime" : Timestamp(1363985782000, 49),
			"optimeDate" : ISODate("2013-03-22T20:56:22Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "db-mongo-004.int.funzio.com:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 31479,
			"optime" : Timestamp(1363985782000, 49),
			"optimeDate" : ISODate("2013-03-22T20:56:22Z"),
			"lastHeartbeat" : ISODate("2013-03-22T20:56:23Z"),
			"pingMs" : 0
		},
		{
			"_id" : 2,
			"name" : "db-mongo-005.int.funzio.com:27017",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 31513,
			"lastHeartbeat" : ISODate("2013-03-22T20:56:24Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}

and config file

more mongod.conf
# mongo.conf
 
# where to log
# logpath=/var/log/mongo/mongod.log
logpath=/mnt/log/mongo/mongod.log
 
logappend=true
 
# fork and run in background
fork = true
 
#port = 27017
 
dbpath=/mnt/mongo/data
 
# location of pidfile
pidfilepath = /var/run/mongodb/mongod.pid
 
# Disables write-ahead journaling
# nojournal = true
 
# Enables periodic logging of CPU utilization and I/O wait
#cpu = true
 
# Turn on/off security.  Off is currently the default
#noauth = true
auth = true
keyFile = /mnt/mongo/keyfile
 
# Verbose logging output.
#verbose = true
 
# Inspect all client data for validity on receipt (useful for
# developing drivers)
#objcheck = true
 
# Enable db quota management
#quota = true
 
# Set oplogging level where n is
#   0=off (default)
#   1=W
#   2=R
#   3=both
#   7=W+some reads
#diaglog = 0
 
# Ignore query hints
#nohints = true
 
# Disable the HTTP interface (Defaults to localhost:27018).
#nohttpinterface = true
 
# Turns off server-side scripting.  This will result in greatly limited
# functionality
#noscripting = true
 
# Turns off table scans.  Any query that would do a table scan fails.
#notablescan = true
 
# Disable data file preallocation.
#noprealloc = true
 
# Specify .ns file size for new databases.
# nssize = <size>
 
# Accout token for Mongo monitoring server.
#mms-token = <token>
 
# Server name for Mongo monitoring server.
#mms-name = <server-name>
 
# Ping interval for Mongo monitoring server.
#mms-interval = <seconds>
 
# Replication Options
replSet = analytics
 
# in replicated mongo databases, specify here whether this is a slave or master
#slave = true
#source = master.example.com
# Slave only: specify a single database to replicate
#only = master.example.com
# or
#master = true
#source = slave.example.com

Comment by Michael Grundy [ 22/Mar/13 ]

I'd like to collect more information to help diagnose this issue:

Can you post the config files and complete logs for each server?
Could you paste the output of running rs.status() on the primary?
Are these servers in MMS?

Thanks!
Mike

Comment by Rui Huang [ 22/Mar/13 ]

Here is the more log for the the first crash:

Sat Mar  9 10:52:52 [initandlisten] connection accepted from 10.100.246.58:58923 #519092 (213 connections now open)
Sat Mar  9 10:52:52 [conn519092] end connection 10.100.246.58:58923 (212 connections now open)
Sat Mar  9 10:52:52 [initandlisten] connection accepted from 10.80.226.11:59152 #519093 (213 connections now open)
Sat Mar  9 10:52:52 [conn519093] end connection 10.80.226.11:59152 (212 connections now open)
Sat Mar  9 10:52:53 [initandlisten] connection accepted from 10.76.197.40:40205 #519094 (213 connections now open)
Sat Mar  9 10:52:53 [initandlisten] connection accepted from 10.76.197.40:40206 #519095 (214 connections now open)
Sat Mar  9 10:52:53 [conn519094] end connection 10.76.197.40:40205 (213 connections now open)
Sat Mar  9 10:52:53 [conn519095] end connection 10.76.197.40:40206 (212 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40210 #519096 (213 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40212 #519097 (214 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40214 #519098 (215 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40216 #519099 (216 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40218 #519100 (217 connections now open)
Sat Mar  9 10:52:54 [conn519097] end connection 10.76.197.40:40212 (216 connections now open)
Sat Mar  9 10:52:54 [conn519096] end connection 10.76.197.40:40210 (216 connections now open)
Sat Mar  9 10:52:54 [conn519098] end connection 10.76.197.40:40214 (214 connections now open)
Sat Mar  9 10:52:54 [conn519099] end connection 10.76.197.40:40216 (214 connections now open)
Sat Mar  9 10:52:54 [conn519100] end connection 10.76.197.40:40218 (213 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40219 #519101 (213 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40220 #519102 (214 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40221 #519103 (215 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40222 #519104 (216 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.76.197.40:40223 #519105 (217 connections now open)
Sat Mar  9 10:52:54 [conn519101]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "91b394e272ce270c", key: "ee603a640905bceed8677937ea78a1ff" }
Sat Mar  9 10:52:54 [conn519104]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "fc908199a9a4f684", key: "ebf95b520137b5e67ce2283d93f26efc" }
Sat Mar  9 10:52:54 [conn519102]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "b6fe0286c52cbe6b", key: "2785eedad979e796ab323556b062abac" }
Sat Mar  9 10:52:54 [conn519103]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "a73801a2ff96257d", key: "1236c491d8e2fc1315e5d639a50d23c2" }
Sat Mar  9 10:52:54 [conn519105]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "9503d99a53e8813d", key: "a09e89ff16b029eec90ec95bdb201e80" }
Sat Mar  9 10:52:54 [conn519103] end connection 10.76.197.40:40221 (216 connections now open)
Sat Mar  9 10:52:54 [conn519104] end connection 10.76.197.40:40222 (215 connections now open)
Sat Mar  9 10:52:54 [conn519105] end connection 10.76.197.40:40223 (214 connections now open)
Sat Mar  9 10:52:54 [conn519102] end connection 10.76.197.40:40220 (213 connections now open)
Sat Mar  9 10:52:54 [conn519101] end connection 10.76.197.40:40219 (212 connections now open)
Sat Mar  9 10:52:54 [initandlisten] connection accepted from 10.100.246.58:58925 #519106 (213 connections now open)
Sat Mar  9 10:52:55 [initandlisten] connection accepted from 10.76.197.40:40230 #519107 (214 connections now open)
Sat Mar  9 10:52:55 [initandlisten] connection accepted from 10.76.197.40:40229 #519108 (215 connections now open)
Sat Mar  9 10:52:55 [conn519107] end connection 10.76.197.40:40230 (214 connections now open)
Sat Mar  9 10:52:55 [conn519108] end connection 10.76.197.40:40229 (213 connections now open)
Sat Mar  9 10:52:55 [conn519106] end connection 10.100.246.58:58925 (212 connections now open)
Sat Mar  9 10:52:55 [initandlisten] connection accepted from 10.80.226.11:59154 #519109 (213 connections now open)
Sat Mar  9 10:52:55 [conn519109] end connection 10.80.226.11:59154 (212 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.100.246.58:58927 #519110 (213 connections now open)
Sat Mar  9 10:52:57 [conn519110] end connection 10.100.246.58:58927 (212 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.76.197.40:40233 #519111 (213 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.76.197.40:40234 #519112 (214 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.80.226.11:59156 #519113 (215 connections now open)
Sat Mar  9 10:52:57 [conn519113] end connection 10.80.226.11:59156 (214 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56489 #519114 (215 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56491 #519115 (216 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56493 #519116 (217 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56494 #519117 (218 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56497 #519118 (219 connections now open)
Sat Mar  9 10:52:57 [conn519114] end connection 10.224.26.28:56489 (218 connections now open)
Sat Mar  9 10:52:57 [conn519116] end connection 10.224.26.28:56493 (217 connections now open)
Sat Mar  9 10:52:57 [conn519115] end connection 10.224.26.28:56491 (216 connections now open)
Sat Mar  9 10:52:57 [conn519117] end connection 10.224.26.28:56494 (215 connections now open)
Sat Mar  9 10:52:57 [conn519118] end connection 10.224.26.28:56497 (216 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56498 #519119 (215 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56499 #519120 (216 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56500 #519121 (217 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56501 #519122 (218 connections now open)
Sat Mar  9 10:52:57 [initandlisten] connection accepted from 10.224.26.28:56502 #519123 (219 connections now open)
Sat Mar  9 10:52:57 [conn519120]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "21845db2defe0fd6", key: "474f4f2a40a36562cbfb2d01f5580ab7" }
Sat Mar  9 10:52:57 [conn519121]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "40188d57d5f4db20", key: "fa4ef022009abc566d2b9f14f03a97d8" }
Sat Mar  9 10:52:57 [conn519122]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "9ebb2aa804793ed1", key: "e0143972625fa1223a7bcf959680d00a" }
Sat Mar  9 10:52:57 [conn519119]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "42d28702a8c8344b", key: "aff41e7c016780ad8da32ca988ccc4cb" }
Sat Mar  9 10:52:57 [conn519123]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "6ae84543991a59ea", key: "7ece7c55d7c6238572f3669570dd80ba" }
Sat Mar  9 10:52:58 [conn519119] end connection 10.224.26.28:56498 (218 connections now open)
Sat Mar  9 10:52:58 [conn519122] end connection 10.224.26.28:56501 (217 connections now open)
Sat Mar  9 10:52:58 [conn519121] end connection 10.224.26.28:56500 (216 connections now open)
Sat Mar  9 10:52:58 [conn519120] end connection 10.224.26.28:56499 (215 connections now open)
Sat Mar  9 10:52:58 [conn519123] end connection 10.224.26.28:56502 (214 connections now open)
Sat Mar  9 10:52:58 [conn519111] end connection 10.76.197.40:40233 (213 connections now open)
Sat Mar  9 10:52:58 [conn519112] end connection 10.76.197.40:40234 (212 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58930 #519124 (213 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58932 #519125 (214 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58934 #519126 (215 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58936 #519127 (216 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58938 #519128 (217 connections now open)
Sat Mar  9 10:52:58 [conn519125] end connection 10.100.246.58:58932 (216 connections now open)
Sat Mar  9 10:52:58 [conn519126] end connection 10.100.246.58:58934 (215 connections now open)
Sat Mar  9 10:52:58 [conn519127] end connection 10.100.246.58:58936 (215 connections now open)
Sat Mar  9 10:52:58 [conn519124] end connection 10.100.246.58:58930 (216 connections now open)
Sat Mar  9 10:52:58 [conn519128] end connection 10.100.246.58:58938 (213 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58939 #519129 (213 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58940 #519130 (214 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58941 #519131 (215 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58942 #519132 (216 connections now open)
Sat Mar  9 10:52:58 [initandlisten] connection accepted from 10.100.246.58:58943 #519133 (217 connections now open)
Sat Mar  9 10:52:58 [conn519130]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "79337eefadbd95b3", key: "71f4d3544b41f89e800baeb70b1dd768" }
Sat Mar  9 10:52:58 [conn519129]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "9c0ae6ad209753ab", key: "b9d5579752c7b6878ecbf43e6f76c020" }
Sat Mar  9 10:52:58 [conn519132]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "1796e2e1fffd2d49", key: "3854672b5bd5b7241f6a56c427bf5e18" }
Sat Mar  9 10:52:58 [conn519131]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "9e1e61a9547e9403", key: "fd57686bd865b588b1d5db398cedfc6d" }
Sat Mar  9 10:52:58 [conn519133]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "7987820adda7b4c1", key: "b5b894f180b3d01ca5dc17ef799e25d3" }
Sat Mar  9 10:52:58 [conn519129] end connection 10.100.246.58:58939 (216 connections now open)
Sat Mar  9 10:52:58 [conn519132] end connection 10.100.246.58:58942 (215 connections now open)
Sat Mar  9 10:52:58 [conn519131] end connection 10.100.246.58:58941 (214 connections now open)
Sat Mar  9 10:52:58 [conn519130] end connection 10.100.246.58:58940 (213 connections now open)
Sat Mar  9 10:52:58 [conn519133] end connection 10.100.246.58:58943 (212 connections now open)
Sat Mar  9 10:52:59 [conn518945] end connection 10.6.146.171:54833 (211 connections now open)
Sat Mar  9 10:52:59 [initandlisten] connection accepted from 10.6.146.171:54837 #519134 (212 connections now open)
Sat Mar  9 10:52:59 [conn519134]  authenticate db: local { authenticate: 1, nonce: "1d99968e7248e99f", user: "__system", key: "3ac0578446781932fcc5f6c6b967f4c1" }
Sat Mar  9 10:52:59 [initandlisten] connection accepted from 10.100.246.58:58949 #519135 (213 connections now open)
Sat Mar  9 10:52:59 [conn519135] end connection 10.100.246.58:58949 (212 connections now open)
Sat Mar  9 10:53:00 [initandlisten] connection accepted from 10.80.226.11:59158 #519136 (213 connections now open)
Sat Mar  9 10:53:00 [initandlisten] connection accepted from 10.76.197.40:40238 #519137 (214 connections now open)
Sat Mar  9 10:53:00 [initandlisten] connection accepted from 10.76.197.40:40239 #519138 (215 connections now open)
Sat Mar  9 10:53:00 [conn519137] end connection 10.76.197.40:40238 (214 connections now open)
Sat Mar  9 10:53:00 [conn519138] end connection 10.76.197.40:40239 (213 connections now open)
Sat Mar  9 10:53:01 [conn519136] end connection 10.80.226.11:59158 (212 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59160 #519139 (213 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59162 #519140 (214 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59164 #519141 (215 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59165 #519142 (216 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59167 #519143 (217 connections now open)
Sat Mar  9 10:53:01 [conn519139] end connection 10.80.226.11:59160 (216 connections now open)
Sat Mar  9 10:53:01 [conn519140] end connection 10.80.226.11:59162 (216 connections now open)
Sat Mar  9 10:53:01 [conn519141] end connection 10.80.226.11:59164 (214 connections now open)
Sat Mar  9 10:53:01 [conn519142] end connection 10.80.226.11:59165 (214 connections now open)
Sat Mar  9 10:53:01 [conn519143] end connection 10.80.226.11:59167 (212 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59169 #519144 (213 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59170 #519145 (214 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59171 #519146 (215 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59172 #519147 (216 connections now open)
Sat Mar  9 10:53:01 [initandlisten] connection accepted from 10.80.226.11:59173 #519148 (217 connections now open)
Sat Mar  9 10:53:01 [conn519146]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "f890d54f5338ce00", key: "16ad7b186a85831576ad7055217dfbd8" }
Sat Mar  9 10:53:01 [conn519144]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "c1096ce2ff381a48", key: "c947970db22acc5a8b3b35dce4828643" }
Sat Mar  9 10:53:01 [conn519147]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "33fddc6b4ac6ccdf", key: "317265ffb342436ac1aed1caa8e2271b" }
Sat Mar  9 10:53:01 [conn519145]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "d8e2412607d0d632", key: "67f75a6a58752f761e6e9d1c54039f1e" }
Sat Mar  9 10:53:01 [conn519148]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "a9602a26438f1637", key: "871f77227ecf04e7ceef7464e84f7db0" }
Sat Mar  9 10:53:01 [conn519144] end connection 10.80.226.11:59169 (216 connections now open)
Sat Mar  9 10:53:01 [conn519147] end connection 10.80.226.11:59172 (216 connections now open)
Sat Mar  9 10:53:01 [conn519148] end connection 10.80.226.11:59173 (214 connections now open)
Sat Mar  9 10:53:01 [conn519146] end connection 10.80.226.11:59171 (214 connections now open)
Sat Mar  9 10:53:01 [conn519145] end connection 10.80.226.11:59170 (214 connections now open)
Sat Mar  9 10:53:01 Invalid access at address: 0x7ed31475bdeb from thread: repl writer worker 9
 
Sat Mar  9 10:53:01 Got signal: 11 (Segmentation fault).
 
Sat Mar  9 10:53:01 Backtrace:
0xaffd31 0x558bb9 0x559142 0x7fd4b1428500 0x83d714 0x8420a0 0x844937 0x823598 0x9a34c2 0x9a2d4f 0xad35cd 0xb45ba9 0x7fd4b1420851 0x7fd4b0ce211d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x558bb9]
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559142]
 /lib64/libpthread.so.0(+0xf500) [0x7fd4b1428500]
 /usr/bin/mongod(_ZN5mongo11checkNoModsENS_7BSONObjE+0x34) [0x83d714]
 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x3160) [0x8420a0]
 /usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0x844937]
 /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x1228) [0x823598]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x292) [0x9a34c2]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x4f) [0x9a2d4f]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xad35cd]
 /usr/bin/mongod() [0xb45ba9]
 /lib64/libpthread.so.0(+0x7851) [0x7fd4b1420851]
 /lib64/libc.so.6(clone+0x6d) [0x7fd4b0ce211d]

Comment by Rui Huang [ 22/Mar/13 ]

the mongo db crashed again, here is the most recent log.

Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.68.94.142:40905 #13611381 (216 connections now open)
Fri Mar 22 04:35:30 [conn13611377] end connection 10.68.94.142:40896 (215 connections now open)
Fri Mar 22 04:35:30 [conn13611378] end connection 10.68.94.142:40899 (215 connections now open)
Fri Mar 22 04:35:30 [conn13611380] end connection 10.68.94.142:40903 (213 connections now open)
Fri Mar 22 04:35:30 [conn13611381] end connection 10.68.94.142:40905 (213 connections now open)
Fri Mar 22 04:35:30 [conn13611379] end connection 10.68.94.142:40901 (213 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.68.94.142:40911 #13611382 (212 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.68.94.142:40912 #13611383 (213 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.68.94.142:40913 #13611384 (214 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.68.94.142:40914 #13611385 (215 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.68.94.142:40915 #13611386 (216 connections now open)
Fri Mar 22 04:35:30 [conn13611382]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "1d00928a307b1b50", key: "a1ded48898e6d7b3b2d43340ff5693c7" }
Fri Mar 22 04:35:30 [conn13611383]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "1b01b49c261172e8", key: "fbfbc0528d902e984ea24a8f3605e43e" }
Fri Mar 22 04:35:30 [conn13611384]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "91d1f044de0e77d3", key: "97c3bf7c1ed2a1d2c043cb270ff4a81d" }
Fri Mar 22 04:35:30 [conn13611385]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "7816aeb2addccebd", key: "cdeb6e53215933196f293e6e7de6b833" }
Fri Mar 22 04:35:30 [conn13611386]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "37a7d4084a0f325c", key: "e0405623b973fbf983cecc29b88877f3" }
Fri Mar 22 04:35:30 [conn13611382] end connection 10.68.94.142:40911 (215 connections now open)
Fri Mar 22 04:35:30 [conn13611383] end connection 10.68.94.142:40912 (214 connections now open)
Fri Mar 22 04:35:30 [conn13611384] end connection 10.68.94.142:40913 (213 connections now open)
Fri Mar 22 04:35:30 [conn13611386] end connection 10.68.94.142:40915 (212 connections now open)
Fri Mar 22 04:35:30 [conn13611385] end connection 10.68.94.142:40914 (211 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.100.246.58:60392 #13611387 (212 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.100.246.58:60393 #13611388 (213 connections now open)
Fri Mar 22 04:35:30 [initandlisten] connection accepted from 10.100.246.58:60394 #13611389 (214 connections now open)
Fri Mar 22 04:35:30 [conn13611387] end connection 10.100.246.58:60392 (213 connections now open)
Fri Mar 22 04:35:30 [conn13611388] end connection 10.100.246.58:60393 (212 connections now open)
Fri Mar 22 04:35:30 [conn13611389] end connection 10.100.246.58:60394 (211 connections now open)
Fri Mar 22 04:35:30 [conn13611374] end connection 10.80.226.11:33514 (210 connections now open)
Fri Mar 22 04:35:30 [conn13611375] end connection 10.80.226.11:33516 (209 connections now open)
Fri Mar 22 04:35:30 [conn13611376] end connection 10.80.226.11:33515 (208 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60395 #13611390 (209 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60397 #13611391 (210 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60399 #13611392 (211 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60402 #13611393 (212 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60404 #13611394 (213 connections now open)
Fri Mar 22 04:35:31 [conn13611390] end connection 10.100.246.58:60395 (212 connections now open)
Fri Mar 22 04:35:31 [conn13611391] end connection 10.100.246.58:60397 (212 connections now open)
Fri Mar 22 04:35:31 [conn13611393] end connection 10.100.246.58:60402 (210 connections now open)
Fri Mar 22 04:35:31 [conn13611394] end connection 10.100.246.58:60404 (210 connections now open)
Fri Mar 22 04:35:31 [conn13611392] end connection 10.100.246.58:60399 (208 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60410 #13611395 (209 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60411 #13611396 (210 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60412 #13611397 (211 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60413 #13611398 (212 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.100.246.58:60414 #13611399 (213 connections now open)
Fri Mar 22 04:35:31 [conn13611397]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "6e143ef3ae18e776", key: "6e14091d4fab136ac1a19436a3eeea39" }
Fri Mar 22 04:35:31 [conn13611398]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "6ab94a3330feeb01", key: "88754f8cdc66cee8d31c8a61aea17d19" }
Fri Mar 22 04:35:31 [conn13611395]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "f04d55276d3be6d2", key: "c43919926021a3a34f7dfb1be431b384" }
Fri Mar 22 04:35:31 [conn13611399]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "fc3074c1c0e54f79", key: "8e97c3298fd7520c8c91249abe918ead" }
Fri Mar 22 04:35:31 [conn13611396]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "25217de620afeb97", key: "d1ab096899ba7894e854cc22da9c6d94" }
Fri Mar 22 04:35:31 [conn13611396] end connection 10.100.246.58:60411 (212 connections now open)
Fri Mar 22 04:35:31 [conn13611399] end connection 10.100.246.58:60414 (211 connections now open)
Fri Mar 22 04:35:31 [conn13611398] end connection 10.100.246.58:60413 (210 connections now open)
Fri Mar 22 04:35:31 [conn13611395] end connection 10.100.246.58:60410 (209 connections now open)
Fri Mar 22 04:35:31 [conn13611397] end connection 10.100.246.58:60412 (208 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.224.26.28:33870 #13611400 (209 connections now open)
Fri Mar 22 04:35:31 [initandlisten] connection accepted from 10.224.26.28:33872 #13611401 (210 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33521 #13611402 (211 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33523 #13611403 (212 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33525 #13611404 (213 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33527 #13611405 (214 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33529 #13611406 (215 connections now open)
Fri Mar 22 04:35:32 [conn13611402] end connection 10.80.226.11:33521 (214 connections now open)
Fri Mar 22 04:35:32 [conn13611403] end connection 10.80.226.11:33523 (214 connections now open)
Fri Mar 22 04:35:32 [conn13611405] end connection 10.80.226.11:33527 (212 connections now open)
Fri Mar 22 04:35:32 [conn13611404] end connection 10.80.226.11:33525 (212 connections now open)
Fri Mar 22 04:35:32 [conn13611406] end connection 10.80.226.11:33529 (210 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 127.0.0.1:52776 #13611407 (211 connections now open)
Fri Mar 22 04:35:32 [conn13611407]  authenticate db: admin { authenticate: 1, nonce: "e72203b388644f9a", user: "stats", key: "34a61966b6a1586c6608e2a9876b5fed" }
Fri Mar 22 04:35:32 [conn13611407] end connection 127.0.0.1:52776 (210 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33536 #13611408 (211 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33537 #13611409 (212 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33538 #13611410 (213 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33539 #13611411 (214 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.80.226.11:33540 #13611412 (215 connections now open)
Fri Mar 22 04:35:32 [conn13611408]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "7ccbc95ebd910e89", key: "63ab5bccdf8a440321dedef569e1f5e8" }
Fri Mar 22 04:35:32 [conn13611411]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "bc00f8917d96027b", key: "3ea2a446b41629d357b30067c32d0af5" }
Fri Mar 22 04:35:32 [conn13611409]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "8ff18b86919aed63", key: "14d58f70ff8bbc3788e3fd440409c5b0" }
Fri Mar 22 04:35:32 [conn13611412]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "4cd42b605da69b89", key: "dfa50acb626ea4a048a21ec5820d27d7" }
Fri Mar 22 04:35:32 [conn13611410]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "11af7410a03808aa", key: "0c865794956b4af1cd1743f4d2fea4f5" }
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.100.246.58:60420 #13611413 (216 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.100.246.58:60419 #13611414 (217 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.100.246.58:60418 #13611415 (218 connections now open)
Fri Mar 22 04:35:32 [conn13611413] end connection 10.100.246.58:60420 (217 connections now open)
Fri Mar 22 04:35:32 [conn13611415] end connection 10.100.246.58:60418 (216 connections now open)
Fri Mar 22 04:35:32 [conn13611414] end connection 10.100.246.58:60419 (216 connections now open)
Fri Mar 22 04:35:32 [conn13611409] end connection 10.80.226.11:33537 (214 connections now open)
Fri Mar 22 04:35:32 [conn13611412] end connection 10.80.226.11:33540 (213 connections now open)
Fri Mar 22 04:35:32 [conn13611410] end connection 10.80.226.11:33538 (212 connections now open)
Fri Mar 22 04:35:32 [conn13611411] end connection 10.80.226.11:33539 (211 connections now open)
Fri Mar 22 04:35:32 [conn13611408] end connection 10.80.226.11:33536 (210 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33874 #13611416 (211 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33876 #13611417 (212 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33879 #13611418 (213 connections now open)
Fri Mar 22 04:35:32 [conn13611416] end connection 10.224.26.28:33874 (212 connections now open)
Fri Mar 22 04:35:32 [conn13611400] end connection 10.224.26.28:33870 (212 connections now open)
Fri Mar 22 04:35:32 [conn13611401] end connection 10.224.26.28:33872 (212 connections now open)
Fri Mar 22 04:35:32 [conn13611417] end connection 10.224.26.28:33876 (210 connections now open)
Fri Mar 22 04:35:32 [conn13611418] end connection 10.224.26.28:33879 (208 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33885 #13611419 (209 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33886 #13611420 (210 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33887 #13611421 (211 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33888 #13611422 (212 connections now open)
Fri Mar 22 04:35:32 [initandlisten] connection accepted from 10.224.26.28:33889 #13611423 (213 connections now open)
Fri Mar 22 04:35:32 [conn13611419]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "763a737d4dbf11aa", key: "9c5970aac1ebc780221d3d777be43011" }
Fri Mar 22 04:35:32 [conn13611420]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "334eb093002ee2e", key: "ed39ddb73bee633fb9fb22cc2abe7062" }
Fri Mar 22 04:35:32 [conn13611421]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "37a7228a8ffefdcf", key: "6484e6c6c419241ec9678b9e174e018d" }
Fri Mar 22 04:35:32 [conn13611422]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "b50d98e5bb19218d", key: "4f0ae380c0d607f09039df5d19cb4896" }
Fri Mar 22 04:35:32 [conn13611423]  authenticate db: admin { authenticate: 1, user: "admin", nonce: "af7bda01e828dd42", key: "e5b435976c6f837ad59da2032bd38760" }
Fri Mar 22 04:35:32 Invalid access at address: 0x7e4abf94a765 from thread: repl writer worker 3
 
Fri Mar 22 04:35:32 Got signal: 11 (Segmentation fault).
 
Fri Mar 22 04:35:32 Backtrace:
0xb07561 0x5598c9 0x559e52 0x348340f500 0x841f14 0x8468a0 0x849137 0x827eb8 0x9ac162 0x9ab9ef 0xadab5d 0xb4d3d9 0x3483407851 0x3482ce811d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x559e52]
 /lib64/libpthread.so.0() [0x348340f500]
 /usr/bin/mongod(_ZN5mongo11checkNoModsENS_7BSONObjE+0x34) [0x841f14]
 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x3160) [0x8468a0]
 /usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0x849137]
 /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x1228) [0x827eb8]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x292) [0x9ac162]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x4f) [0x9ab9ef]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
 /usr/bin/mongod() [0xb4d3d9]
 /lib64/libpthread.so.0() [0x3483407851]
 /lib64/libc.so.6(clone+0x6d) [0x3482ce811d]

Comment by Michael Grundy [ 21/Mar/13 ]

Can we get the full log please? The attached backtrace is helpful, but frequently the most important information is leading up to the crash, which is not present.

Thanks!
Mike

Generated at Thu Feb 08 03:19:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.