We killed mongodb and the error stopped. So far it looks like no data corruption. And it looks like the master-slave replication stayed sane. I've marked this as priority:minor because I don't expect to run this script again. But as a mongod server failure, you might set a higher priority.

I ran a cleanup script to fix bad data caused by previous bugs in our software. The script queried from a large collection (url) on our slave, then started updating records on our master. We we're updating using a key. The script got killed and was restarted. As far as I can tell, the original script was dead, but based on the symptoms here, I'll guess that we had two versions of this script running, and it was the "killed" script that was causing these problems. (That's the only explanation I have for these updates showing up in our log after the script was apparently done).

The script was basically running this update (I tried to translate from php to json, so there may be syntax errors below, hopefully you get the idea). We ran this on about 10,000 scan_id's, each matching many records:.

mongo> db.url.update(

{scan_id' : 12345}

, {'$set' : {'scan':

{client_id:9876, active:0, status:0}

}},

{multiple:true}

)

The collection is big:

mongo> db.url.stats()
{
"ns" : "scan_stats.url",
"count" : 45004170,
"size" : 9725083024,
"storageSize" : 10536955808,
"nindexes" : 7,
"ok" : 1
}

And this is what we saw in our logs:

Fri Dec 18 18:41:23 connection accepted from 192.168.1.200:39880 #2348670
ASSERT failure BtreeBucket::advance, caller: BtreeCursor::advance
thisLoc: 8:78557190
keyOfs: 79 n:79 direction: 1
Bucket info:
n: 79
parent: c:36cd4de0
nextChild: c:36cd4de0
Size: 8192 flags:0
emptySize: 5027 topSize: 1703

Fri Dec 18 18:41:23 scan_stats.url Assertion failure false db/btree.cpp 713
0x463de6 0x46bc81 0x4bf54e 0x4e7ee1 0x4b6d87 0x53afea 0x53e401 0x59b5c5 0x5a7dc0 0x30aba064a7 0x30ab2d3c2d
/usr/local/mongodb/bin/mongod [0x463de6]
/usr/local/mongodb/bin/mongod [0x46bc81]
/usr/local/mongodb/bin/mongod [0x4bf54e]
/usr/local/mongodb/bin/mongod [0x4e7ee1]
/usr/local/mongodb/bin/mongod [0x4b6d87]
/usr/local/mongodb/bin/mongod [0x53afea]
/usr/local/mongodb/bin/mongod [0x53e401]
/usr/local/mongodb/bin/mongod(_ZN5mongo10connThreadEv+0x225) [0x59b5c5]
/usr/local/mongodb/bin/mongod [0x5a7dc0]
/lib64/libpthread.so.0 [0x30aba064a7]
/lib64/libc.so.6(clone+0x6d) [0x30ab2d3c2d]
Fri Dec 18 18:41:16 update scan_stats.url query:

{ scan_id: 3667 }

exception assertion db/btree.cpp:713 6453ms