[SERVER-8147] Replication breaks with errorMessage (c_prod Assertion failure a.ext()->xprev.isNull()) Created: 11/Jan/13  Updated: 10/Dec/14  Resolved: 04/Apr/14

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Thomas Zimmer Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux


Issue Links:
Duplicate
duplicates SERVER-12956 Stopping a secondary in phase 2/3 of ... Closed
Operating System: ALL
Participants:

 Description   

We are using a replication set for a mongodb-installation. One day we realized that the replication has broken one database completely and we were unable to get the replication back to a synchronous state. Here is the stacktrace from the log-file:

c_prod Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x865d50 0x6ed1c6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7f177dd7c8ba 0x7f177d13002d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
/usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
/usr/bin/mongod(ZN5mongo11freeExtentsENS_7DiskLocES0+0x679) [0x863eb9]
/usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
/usr/bin/mongod(_ZN5mongo14dropCollectionERKSsRSsRNS_14BSONObjBuilderE+0x1a0) [0x865d50]
/usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x6ed1c6]
/usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
/usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
/usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
/usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xc0b) [0x83376b]
/usr/bin/mongod() [0x7b0b0d]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
/lib/libpthread.so.0(+0x68ba) [0x7f177dd7c8ba]
/lib/libc.so.6(clone+0x6d) [0x7f177d13002d]
Wed Dec 12 16:35:08 got signal 15 (Terminated), will terminate after current cmd ends
Wed Dec 12 16:35:08 [interruptThread] now exiting

It seems like that there is one file / index that is causing that error and will force the synchronization to break.



 Comments   
Comment by Thomas Rueckstiess [ 04/Apr/14 ]

This issue looks like it's a duplicate of SERVER-12956, which describes an index corruption if the index build is interrupted abnormally. This can corrupt the index catalog and will also require secondaries to resync from a good source. It's not clear why the index build here took so long in the first place, but we can see that the index was interrupted with a SIGTERM (-15).

SERVER-12956 has been fixed and is part of production release 2.4.10 and 2.6. As such, I'll close this as a duplicate of SERVER-12956.

Comment by Thomas Pöhler [ 23/Jan/13 ]

Hey Stephen,

thank you for investigating.

  • Can you confirm the specific version of MongoDB server (i.e. 2.2.2) so we can correctly decode the stacktrace?
    -> db version v2.2.1, pdfile version 4.5
    -> git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
    -> build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
  • What MongoDB driver & version are you using?
    -> PHP: MongoDB Support => enabled
    -> Version => 1.2.12
  • How many nodes do you have in this replica set ?
    -> 3: Master, Slave and Arbiter
  • Did all the secondaries encounter the same error ?
    -> Yes (its only one)
  • What node(s) and database(s) did you run the repairDatabase() on?
    -> Only on the secondary, we removed it from the active replica set and then tried to repair the collection, since this is a live production system.

Perhaps its a matter of size. We use GridFS for storing about 5TB of data in it.

some more logging:

Wed Dec 12 12:51:01 [conn58] serverStatus was very slow: { after basic: 0, middle of mem: 770, after mem: 770, after connections: 770, after extra info: 1070, after counters: 1070, after repl: 1070, after as
serts: 1070, after dur: 972045, at end: 972045 }
Wed Dec 12 12:51:01 [conn58] command admin.$cmd command: { serverStatus: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:37 reslen:2814 1008378ms
Wed Dec 12 12:51:01 [conn58] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.1.0.83:36677]
Wed Dec 12 12:51:01 [conn73] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.1.0.83:36745]
Wed Dec 12 12:51:01 [conn222] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.1.0.83:38185]
Wed Dec 12 12:51:01 [conn158] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.1.0.83:37629]
Wed Dec 12 12:51:01 [repl writer worker 1] build index somedatabase_2012-12-05.files { sessionId: 1, filename: 1 } background
Wed Dec 12 13:51:42 got signal 15 (Terminated), will terminate after current cmd ends
 
 
***** SERVER RESTARTED *****
 
 
Wed Dec 12 13:59:36 [initandlisten] MongoDB starting : pid=3203 port=58888 dbpath=/space1/mongodb 64-bit host=mongo1
Wed Dec 12 13:59:36 [initandlisten] db version v2.2.1, pdfile version 4.5
Wed Dec 12 13:59:36 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
Wed Dec 12 13:59:36 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Dec 12 13:59:36 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/etc/mongodb.conf", dbpath: "/space1/mongodb", directoryperdb: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log"
, oplogSize: 512000, port: 58888, quiet: "true", rest: "true" }
Wed Dec 12 13:59:36 [initandlisten] journal dir=/space1/mongodb/journal
Wed Dec 12 13:59:36 [initandlisten] recover begin
Wed Dec 12 13:59:36 [initandlisten] recover lsn: 51157
Wed Dec 12 13:59:36 [initandlisten] recover /space1/mongodb/journal/j._0
Wed Dec 12 13:59:36 [initandlisten] recover cleaning up
Wed Dec 12 13:59:36 [initandlisten] removeJournalFiles
Wed Dec 12 13:59:36 [initandlisten] recover done
Wed Dec 12 13:59:37 [initandlisten]
Wed Dec 12 13:59:37 [initandlisten] ** warning: mongod started without --replSet yet 1 documents are present in local.system.replset
Wed Dec 12 13:59:37 [initandlisten] **          restart with --replSet unless you are doing maintenance and no other clients are connected
Wed Dec 12 13:59:37 [initandlisten]
Wed Dec 12 13:59:37 [initandlisten] ** warning: not starting TTL monitor
Wed Dec 12 13:59:37 [initandlisten] **          if this member is not part of a replica set and you want to use
Wed Dec 12 13:59:37 [initandlisten] **          TTL collections, remove local.system.replset and restart
Wed Dec 12 13:59:37 [websvr] admin web console waiting for connections on port 59888
Wed Dec 12 13:59:37 [initandlisten] waiting for connections on port 58888
Wed Dec 12 14:25:28 [conn4] CMD: reIndex somedatabase_2012-12-05.files
Wed Dec 12 14:25:28 [conn4]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x78e360 0x794ea8 0x6f8ac6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7fd600d278ba 0x7fd6000db02d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo12IndexDetails8kill_idxEv+0x6c0) [0x78e360]
 /usr/bin/mongod(_ZN5mongo11dropIndexesEPNS_16NamespaceDetailsEPKcS3_RSsRNS_14BSONObjBuilderEb+0x768) [0x794ea8]
 /usr/bin/mongod(_ZN5mongo10CmdReIndex3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xd66) [0x6f8ac6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7fd600d278ba]
 /lib/libc.so.6(clone+0x6d) [0x7fd6000db02d]
Wed Dec 12 14:25:28 [conn4]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x78e360 0x794ea8 0x6f8ac6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7fd600d278ba 0x7fd6000db02d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo12IndexDetails8kill_idxEv+0x6c0) [0x78e360]
 /usr/bin/mongod(_ZN5mongo11dropIndexesEPNS_16NamespaceDetailsEPKcS3_RSsRNS_14BSONObjBuilderEb+0x768) [0x794ea8]
 /usr/bin/mongod(_ZN5mongo10CmdReIndex3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xd66) [0x6f8ac6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7fd600d278ba]
 /lib/libc.so.6(clone+0x6d) [0x7fd6000db02d]
Wed Dec 12 14:25:28 [conn4] build index somedatabase_2012-12-05.files { _id: 1 }
Wed Dec 12 14:25:28 [conn4] build index done.  scanned 0 total records. 0.001 secs
Wed Dec 12 14:25:28 [conn4] build index somedatabase_2012-12-05.files { uploadDate: 1 } background
Wed Dec 12 14:25:28 [conn4] build index done.  scanned 0 total records. 0 secs
Wed Dec 12 14:33:46 [conn4] build index somedatabase_2012-12-05.files { sessionId: 1.0, filename: 1.0 }
Wed Dec 12 14:42:19 [conn7] going to kill op: op: 174.0
Wed Dec 12 14:42:29 [conn7] going to kill op: op: 174.0
Wed Dec 12 14:47:10 got signal 15 (Terminated), will terminate after current cmd ends
 
 
***** SERVER RESTARTED *****
 
 
Wed Dec 12 14:53:26 [initandlisten] MongoDB starting : pid=3425 port=58888 dbpath=/space1/mongodb 64-bit host=mongo1
Wed Dec 12 14:53:26 [initandlisten] db version v2.2.1, pdfile version 4.5
Wed Dec 12 14:53:26 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
Wed Dec 12 14:53:26 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Dec 12 14:53:26 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/etc/mongodb.conf", dbpath: "/space1/mongodb", directoryperdb: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", oplogSize: 512000, port: 58888, quiet: "true", rest: "true" }
Wed Dec 12 14:53:26 [initandlisten] journal dir=/space1/mongodb/journal
Wed Dec 12 14:53:26 [initandlisten] recover begin
Wed Dec 12 14:53:26 [initandlisten] recover lsn: 1489477
Wed Dec 12 14:53:26 [initandlisten] recover /space1/mongodb/journal/j._0
Wed Dec 12 14:53:26 [initandlisten] recover cleaning up
Wed Dec 12 14:53:26 [initandlisten] removeJournalFiles
Wed Dec 12 14:53:26 [initandlisten] recover done
Wed Dec 12 14:53:27 [initandlisten]
Wed Dec 12 14:53:27 [initandlisten] ** warning: mongod started without --replSet yet 1 documents are present in local.system.replset
Wed Dec 12 14:53:27 [initandlisten] **          restart with --replSet unless you are doing maintenance and no other clients are connected
Wed Dec 12 14:53:27 [initandlisten]
Wed Dec 12 14:53:27 [initandlisten] ** warning: not starting TTL monitor
Wed Dec 12 14:53:27 [initandlisten] **          if this member is not part of a replica set and you want to use
Wed Dec 12 14:53:27 [initandlisten] **          TTL collections, remove local.system.replset and restart
Wed Dec 12 14:53:27 [initandlisten] waiting for connections on port 58888
Wed Dec 12 14:53:27 [websvr] admin web console waiting for connections on port 59888
Wed Dec 12 14:55:59 [conn1] build index somedatabase_2012-12-05.files { sessionId: 1.0, filename: 1.0 }
Wed Dec 12 15:10:26 [conn3] going to kill op: op: 13.0
Wed Dec 12 15:10:55 got signal 15 (Terminated), will terminate after current cmd ends
 
 
***** SERVER RESTARTED *****
 
 
Wed Dec 12 15:11:46 [initandlisten] MongoDB starting : pid=3534 port=58888 dbpath=/space1/mongodb 64-bit host=mongo1
Wed Dec 12 15:11:46 [initandlisten] db version v2.2.1, pdfile version 4.5
Wed Dec 12 15:11:46 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
Wed Dec 12 15:11:46 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Dec 12 15:11:46 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/etc/mongodb.conf", dbpath: "/space1/mongodb", directoryperdb: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", oplogSize: 512000, port: 58888, quiet: "true", rest: "true" }
Wed Dec 12 15:11:46 [initandlisten] journal dir=/space1/mongodb/journal
Wed Dec 12 15:11:46 [initandlisten] recover begin
Wed Dec 12 15:11:46 [initandlisten] info no lsn file in journal/ directory
Wed Dec 12 15:11:46 [initandlisten] recover lsn: 0
Wed Dec 12 15:11:46 [initandlisten] recover /space1/mongodb/journal/j._0
Wed Dec 12 15:11:46 [initandlisten] recover cleaning up
Wed Dec 12 15:11:46 [initandlisten] removeJournalFiles
Wed Dec 12 15:11:46 [initandlisten] recover done
Wed Dec 12 15:11:47 [initandlisten]
Wed Dec 12 15:11:47 [initandlisten] ** warning: mongod started without --replSet yet 1 documents are present in local.system.replset
Wed Dec 12 15:11:47 [initandlisten] **          restart with --replSet unless you are doing maintenance and no other clients are connected
Wed Dec 12 15:11:47 [initandlisten]
Wed Dec 12 15:11:47 [initandlisten] ** warning: not starting TTL monitor
Wed Dec 12 15:11:47 [initandlisten] **          if this member is not part of a replica set and you want to use
Wed Dec 12 15:11:47 [initandlisten] **          TTL collections, remove local.system.replset and restart
Wed Dec 12 15:11:47 [initandlisten] waiting for connections on port 58888
Wed Dec 12 15:11:47 [websvr] admin web console waiting for connections on port 59888
 
 
***** SERVER RESTARTED *****
 
 
Wed Dec 12 15:33:51 [initandlisten] MongoDB starting : pid=3636 port=58888 dbpath=/space1/mongodb 64-bit host=mongo1
Wed Dec 12 15:33:51 [initandlisten] db version v2.2.1, pdfile version 4.5
Wed Dec 12 15:33:51 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
Wed Dec 12 15:33:51 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Dec 12 15:33:51 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/etc/mongodb.conf", dbpath: "/space1/mongodb", directoryperdb: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", oplogSize: 512000, port: 58888, quiet: "true", rest: "true" }
Wed Dec 12 15:33:51 [initandlisten] journal dir=/space1/mongodb/journal
Wed Dec 12 15:33:51 [initandlisten] recover begin
Wed Dec 12 15:33:51 [initandlisten] info no lsn file in journal/ directory
Wed Dec 12 15:33:51 [initandlisten] recover lsn: 0
Wed Dec 12 15:33:51 [initandlisten] recover /space1/mongodb/journal/j._0
Wed Dec 12 15:33:51 [initandlisten] recover cleaning up
Wed Dec 12 15:33:51 [initandlisten] removeJournalFiles
Wed Dec 12 15:33:51 [initandlisten] recover done
Wed Dec 12 15:33:51 [initandlisten]
Wed Dec 12 15:33:51 [initandlisten] ** warning: mongod started without --replSet yet 1 documents are present in local.system.replset
Wed Dec 12 15:33:51 [initandlisten] **          restart with --replSet unless you are doing maintenance and no other clients are connected
Wed Dec 12 15:33:51 [initandlisten]
Wed Dec 12 15:33:51 [initandlisten] ** warning: not starting TTL monitor
Wed Dec 12 15:33:51 [initandlisten] **          if this member is not part of a replica set and you want to use
Wed Dec 12 15:33:51 [initandlisten] **          TTL collections, remove local.system.replset and restart
Wed Dec 12 15:33:51 [websvr] admin web console waiting for connections on port 59888
Wed Dec 12 15:33:51 [initandlisten] waiting for connections on port 58888
Wed Dec 12 15:37:24 [conn1]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
Wed Dec 12 15:37:24 [conn1]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x78e360 0x794ea8 0x865dbf 0x6ed1c6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7f00edda48ba 0x7f00ed15802d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo12IndexDetails8kill_idxEv+0x6c0) [0x78e360]
 /usr/bin/mongod(_ZN5mongo11dropIndexesEPNS_16NamespaceDetailsEPKcS3_RSsRNS_14BSONObjBuilderEb+0x768) [0x794ea8]
 /usr/bin/mongod(_ZN5mongo14dropCollectionERKSsRSsRNS_14BSONObjBuilderE+0x20f) [0x865dbf]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x6ed1c6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7f00edda48ba]
 /lib/libc.so.6(clone+0x6d) [0x7f00ed15802d]
Wed Dec 12 15:37:24 [conn1]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x78e360 0x794ea8 0x865dbf 0x6ed1c6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7f00edda48ba 0x7f00ed15802d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo12IndexDetails8kill_idxEv+0x6c0) [0x78e360]
 /usr/bin/mongod(_ZN5mongo11dropIndexesEPNS_16NamespaceDetailsEPKcS3_RSsRNS_14BSONObjBuilderEb+0x768) [0x794ea8]
 /usr/bin/mongod(_ZN5mongo14dropCollectionERKSsRSsRNS_14BSONObjBuilderE+0x20f) [0x865dbf]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x6ed1c6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7f00edda48ba]
 /lib/libc.so.6(clone+0x6d) [0x7f00ed15802d]
Wed Dec 12 15:37:24 [conn1]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x865d50 0x6ed1c6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7f00edda48ba 0x7f00ed15802d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo14dropCollectionERKSsRSsRNS_14BSONObjBuilderE+0x1a0) [0x865d50]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x6ed1c6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7f00edda48ba]
 /lib/libc.so.6(clone+0x6d) [0x7f00ed15802d]
Wed Dec 12 15:38:22 [conn1]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x865d50 0x6ed1c6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7f00edda48ba 0x7f00ed15802d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo14dropCollectionERKSsRSsRNS_14BSONObjBuilderE+0x1a0) [0x865d50]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x6ed1c6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7f00edda48ba]
 /lib/libc.so.6(clone+0x6d) [0x7f00ed15802d]
Wed Dec 12 15:38:35 [conn1]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x865d50 0x6ed1c6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7f00edda48ba 0x7f00ed15802d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo14dropCollectionERKSsRSsRNS_14BSONObjBuilderE+0x1a0) [0x865d50]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x6ed1c6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7f00edda48ba]
 /lib/libc.so.6(clone+0x6d) [0x7f00ed15802d]
Wed Dec 12 15:54:09 [conn1] compact somedatabase_2012-12-05.files begin
Wed Dec 12 15:54:09 [conn1] paddingFactor:1 paddingBytes:0
Wed Dec 12 15:54:09 [conn1] compact 1 extents
Wed Dec 12 15:54:09 [conn1] compact orphan deleted lists
Wed Dec 12 16:18:29 [conn2] going to kill op: op: 67
 
 
***** SERVER RESTARTED *****
 
 
Wed Dec 12 16:21:05 [initandlisten] MongoDB starting : pid=3790 port=58888 dbpath=/space1/mongodb 64-bit host=mongo1
Wed Dec 12 16:21:05 [initandlisten] db version v2.2.1, pdfile version 4.5
Wed Dec 12 16:21:05 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
Wed Dec 12 16:21:05 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Dec 12 16:21:05 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/etc/mongodb.conf", dbpath: "/space1/mongodb", directoryperdb: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", oplogSize: 512000, port: 58888, quiet: "true", rest: "true" }
Wed Dec 12 16:21:05 [initandlisten] journal dir=/space1/mongodb/journal
Wed Dec 12 16:21:05 [initandlisten] recover begin
Wed Dec 12 16:21:05 [initandlisten] recover lsn: 178572
Wed Dec 12 16:21:05 [initandlisten] recover /space1/mongodb/journal/j._0
Wed Dec 12 16:21:05 [initandlisten] recover cleaning up
Wed Dec 12 16:21:05 [initandlisten] removeJournalFiles
Wed Dec 12 16:21:05 [initandlisten] recover done
Wed Dec 12 16:21:05 [initandlisten]
Wed Dec 12 16:21:05 [initandlisten] ** warning: mongod started without --replSet yet 1 documents are present in local.system.replset
Wed Dec 12 16:21:05 [initandlisten] **          restart with --replSet unless you are doing maintenance and no other clients are connected
Wed Dec 12 16:21:05 [initandlisten]
Wed Dec 12 16:21:05 [initandlisten] ** warning: not starting TTL monitor
Wed Dec 12 16:21:05 [initandlisten] **          if this member is not part of a replica set and you want to use
Wed Dec 12 16:21:05 [initandlisten] **          TTL collections, remove local.system.replset and restart
Wed Dec 12 16:21:05 [websvr] admin web console waiting for connections on port 59888
Wed Dec 12 16:21:05 [initandlisten] waiting for connections on port 58888
Wed Dec 12 16:25:11 [conn1]  somedatabase Assertion failure a.ext()->xprev.isNull() src/mongo/db/pdfile.cpp 880
0xaf8c41 0xabf4cd 0x863eb9 0x86547b 0x865d50 0x6ed1c6 0x6e8321 0x6ea104 0x6eab4c 0x830028 0x83376b 0x7b0b0d 0x7b20e2 0x56fe42 0xae6ed1 0x7f177dd7c8ba 0x7f177d13002d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaf8c41]
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0xabf4cd]
 /usr/bin/mongod(_ZN5mongo11freeExtentsENS_7DiskLocES0_+0x679) [0x863eb9]
 /usr/bin/mongod(_ZN5mongo6dropNSERKSs+0x6ab) [0x86547b]
 /usr/bin/mongod(_ZN5mongo14dropCollectionERKSsRSsRNS_14BSONObjBuilderE+0x1a0) [0x865d50]
 /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xc6) [0x6ed1c6]
 /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6e8321]
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x12c4) [0x6ea104]
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ac) [0x6eab4c]
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x830028]
 /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xc0b) [0x83376b]
 /usr/bin/mongod() [0x7b0b0d]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0x7b20e2]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fe42]
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0xae6ed1]
 /lib/libpthread.so.0(+0x68ba) [0x7f177dd7c8ba]
 /lib/libc.so.6(clone+0x6d) [0x7f177d13002d]
Wed Dec 12 16:35:08 got signal 15 (Terminated), will terminate after current cmd ends
Wed Dec 12 16:35:08 [interruptThread] now exiting
Wed Dec 12 16:35:08 dbexit:
Wed Dec 12 16:35:08 [interruptThread] shutdown: going to close listening sockets...
Wed Dec 12 16:35:08 [interruptThread] closing listening socket: 6
Wed Dec 12 16:35:08 [interruptThread] closing listening socket: 7
Wed Dec 12 16:35:08 [interruptThread] closing listening socket: 8
Wed Dec 12 16:35:08 [interruptThread] removing socket file: /tmp/mongodb-58888.sock
Wed Dec 12 16:35:08 [interruptThread] shutdown: going to flush diaglog...
Wed Dec 12 16:35:08 [interruptThread] shutdown: going to close sockets...
Wed Dec 12 16:35:08 [interruptThread] shutdown: waiting for fs preallocator...
Wed Dec 12 16:35:08 [interruptThread] shutdown: lock for final commit...
Wed Dec 12 16:35:08 [interruptThread] shutdown: final commit...
Wed Dec 12 16:35:08 [interruptThread] shutdown: closing all files...
Wed Dec 12 16:35:09 [interruptThread] closeAllFiles() finished
Wed Dec 12 16:35:09 [interruptThread] journalCleanup...
Wed Dec 12 16:35:09 [interruptThread] removeJournalFiles
Wed Dec 12 16:35:09 [interruptThread] shutdown: removing fs lock...
Wed Dec 12 16:35:09 dbexit: really exiting now
 
 
***** SERVER RESTARTED *****
 
 
Wed Dec 12 16:35:14 [initandlisten] MongoDB starting : pid=3873 port=58888 dbpath=/space1/mongodb 64-bit host=mongo1
Wed Dec 12 16:35:14 [initandlisten] db version v2.2.1, pdfile version 4.5
Wed Dec 12 16:35:14 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
Wed Dec 12 16:35:14 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Dec 12 16:35:14 [initandlisten] options: { bind_ip: "0.0.0.0", config: "/etc/mongodb.conf", dbpath: "/space1/mongodb", directoryperdb: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", oplogSize: 512000, port: 58888, quiet: "true", rest: "true" }
Wed Dec 12 16:35:14 [initandlisten] journal dir=/space1/mongodb/journal
Wed Dec 12 16:35:14 [initandlisten] recover : no journal files present, no recovery needed
Wed Dec 12 16:35:14 [initandlisten]
Wed Dec 12 16:35:14 [initandlisten] ** warning: mongod started without --replSet yet 1 documents are present in local.system.replset
Wed Dec 12 16:35:14 [initandlisten] **          restart with --replSet unless you are doing maintenance and no other clients are connected
Wed Dec 12 16:35:14 [initandlisten]
Wed Dec 12 16:35:14 [initandlisten] ** warning: not starting TTL monitor
Wed Dec 12 16:35:14 [initandlisten] **          if this member is not part of a replica set and you want to use
Wed Dec 12 16:35:14 [initandlisten] **          TTL collections, remove local.system.replset and restart
Wed Dec 12 16:35:14 [initandlisten] waiting for connections on port 58888
Wed Dec 12 16:35:14 [websvr] admin web console waiting for connections on port 59888
Wed Dec 12 16:36:58 [conn1] command admin.$cmd command: { listDatabases: 1 } ntoreturn:1 keyUpdates:0 locks(micros) R:7 W:4531322 r:68 reslen:468 4554ms
Wed Dec 12 16:37:06 [conn1] command ac_linesman_prod.$cmd command: { dbstats: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:7124565 reslen:227 7124ms
Wed Dec 12 16:37:07 [conn1] command somedatabase_archive.$cmd command: { dbstats: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:1594838 reslen:226 1594ms
Wed Dec 12 16:37:14 [conn1] command somedatabase_error.$cmd command: { dbstats: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:6488819 reslen:224 6488ms
Wed Dec 12 16:37:16 [conn1] command ac_test.$cmd command: { dbstats: 1 } ntoreturn:1 keyUpdates:0 locks(micros) r:1999970 reslen:206 1999ms
 
***** SERVER RESTARTED *****

Comment by Stennie Steneker (Inactive) [ 18/Jan/13 ]

Hi Thomas,

We'll need some more information to investigate:

  • Can you confirm the specific version of MongoDB server (i.e. 2.2.2) so we can correctly decode the stacktrace?
  • What MongoDB driver & version are you using?
  • How many nodes do you have in this replica set ?
  • Did all the secondaries encounter the same error ?
  • What node(s) and database(s) did you run the repairDatabase() on?

It would be helpful if you are able to upload a more complete section of the log, as there may be relevant events happening before the stacktrace.

If you have suspected corruption, one diagnostic tool you can use is a validate(true) command, which needs to be run for each collection. The repairDatabase command will rebuild the database and indexes, so if you still have issues the problem may have been in the primary rather than the secondary.

If you have a node with suspected corruption, best practices for replica sets would be to do a file copy from a known "good" node or backup, or re-sync. Repairing a secondary database could result in inconsistencies versus the primary, and is not recommended.

Cheers,
Stephen

Comment by Thomas Pöhler [ 17/Jan/13 ]

some additions from operations side: We are using Debian Squeeze 64bit on a virtualized XEN 4.0.1 env. Master and Slave both are virtualized with 6 Cores and 16GB Ram. We use GridFS only on these nodes to store files in it. We are using the 10gen debian packages provided on your website.

Thanks for your help
Thomas

Comment by Thomas Zimmer [ 11/Jan/13 ]

I forget to mention some background - facts:

All the steps described bellow were done using one of the latest MongoDB version (2.2).

This what we did:
We detected this error that the replication does not seem to work. We found this error message in the log-file:

"Wed Dec 12 12:51:01 [repl writer worker 1] build index ac_prod.XXX.files

{ id: 1, name: 1 }

" where XXX is a collection name and with the stack trace (see above) attached to it.

Now we tried to drop the index and try to add it manually which also does not seem to work (100% CPU Load for an empty collection [We waited for 10 minutes until we reboot the mongodb-server]). Afterwards we used the repair command, suggested in some forum threads, which did not fix our problems. After that we tried to remove the whole collection and re-enabled the mongodb replication which also generates the reported error message.

Conclusion: It seems like that the mongodb - replication has generated a corrupt collection and you were not able to repair it. So we needed a complete rsync of all data-files which cost us much effort.

Generated at Thu Feb 08 03:16:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.