[SERVER-8241] assertion failures (saveerrmsg) during repair Created: 18/Jan/13  Updated: 10/Dec/14  Resolved: 03/Jun/13

Status: Closed
Project: Core Server
Component/s: Admin, Logging
Affects Version/s: 2.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zac Witte Assignee: Aaron Staple
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ubuntu on EC2


Issue Links:
Duplicate
duplicates SERVER-8432 Mongod can fail to output error messa... Closed
is duplicated by SERVER-10219 Assertion failure when creating text ... Closed
Related
is related to SERVER-8242 assertion failures (btree) during repair Closed
Operating System: ALL
Participants:

 Description   

Was running a repair with the following command and got this assertion error in the log of 2 out of 3 shards. The third shard had a different assertion error, which I'm creating a different bug for:

ubuntu@mongo1:~$ mongod --version
db version v2.2.0, pdfile version 4.5
Fri Jan 18 20:56:42 git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207

sudo -u mongodb mongod --dbpath=/db/mongodb --repair --repairpath=/export/mongodb --nojournal

Fri Jan 18 18:52:26 [initandlisten] 1926460400/3967177714 48%
Fri Jan 18 18:52:36 [initandlisten] 1927349300/3967177714 48%
Fri Jan 18 18:52:46 [initandlisten] 1928551200/3967177714 48%
Fri Jan 18 18:52:56 [initandlisten] 1929329200/3967177714 48%
Fri Jan 18 18:53:06 [initandlisten] 1930147300/3967177714 48%
Fri Jan 18 18:53:16 [initandlisten] 1931131500/3967177714 48%
Fri Jan 18 18:53:27 [initandlisten] 1931958000/3967177714 48%
Fri Jan 18 18:53:37 [initandlisten] 1933142600/3967177714 48%
Fri Jan 18 18:53:47 [initandlisten] 1934033200/3967177714 48%
Fri Jan 18 18:53:57 [initandlisten] 1935221200/3967177714 48%
Fri Jan 18 18:53:59 [initandlisten] pb3 Assertion failure le && !saveerrmsg.empty() src/mongo/db/pdfile.cpp 1381
0xade6e1 0x803dfd 0x74849e 0x7490bd 0xae2c0d 0x737f9c 0x8535bc 0x8560c9 0x856269 0x74037a 0x5596d8 0x55b78d 0x55c6f0 0x55d2dd 0x563dbe 0x565399 0x7f0873f5830d 0x557869
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0x803dfd]
mongod(_ZN5mongo16insert_makeIndexEPNS_16NamespaceDetailsERKSsRKNS_7DiskLocE+0x7be) [0x74849e]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbPb+0x93d) [0x7490bd]
mongod(_ZN5mongo17ensureHaveIdIndexEPKc+0x7ad) [0xae2c0d]
mongod(_ZN5mongo21ensureIdIndexForNewNsEPKc+0x9c) [0x737f9c]
mongod(_ZN5mongo6Cloner2goEPKcRKNS_12CloneOptionsERSt3setISsSt4lessISsESaISsEERSsPi+0xd1c) [0x8535bc]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbbbbPi+0x139) [0x8560c9]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbbbbPi+0x59) [0x856269]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x52a) [0x74037a]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x68) [0x5596d8]
mongod() [0x55b78d]
mongod(_ZN5mongo14_initAndListenEi+0x480) [0x55c6f0]
mongod(_ZN5mongo13initAndListenEi+0x1d) [0x55d2dd]
mongod() [0x563dbe]
mongod(main+0x9) [0x565399]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f0873f5830d]
mongod(__gxx_personality_v0+0x469) [0x557869]
Fri Jan 18 18:54:00 [initandlisten] exception in initAndListen: 0 assertion src/mongo/db/pdfile.cpp:1381, terminating
Fri Jan 18 18:54:00 dbexit:
Fri Jan 18 18:54:00 [initandlisten] shutdown: going to close listening sockets...
Fri Jan 18 18:54:00 [initandlisten] shutdown: going to flush diaglog...
Fri Jan 18 18:54:00 [initandlisten] shutdown: going to close sockets...
Fri Jan 18 18:54:00 [initandlisten] shutdown: waiting for fs preallocator...
Fri Jan 18 18:54:00 [initandlisten] shutdown: closing all files...
Fri Jan 18 18:54:02 [initandlisten] 279/2819 9%
Fri Jan 18 18:54:04 [initandlisten] 578/2819 20%
Fri Jan 18 18:54:06 [initandlisten] 876/2819 31%
Fri Jan 18 18:54:08 [initandlisten] 1173/2819 41%
Fri Jan 18 18:54:10 [initandlisten] 1420/2819 50%
Fri Jan 18 18:54:12 [initandlisten] 1426/2819 50%
Fri Jan 18 18:54:14 [initandlisten] 1434/2819 50%
Fri Jan 18 18:54:16 [initandlisten] 1440/2819 51%
Fri Jan 18 18:54:18 [initandlisten] 1455/2819 51%
Fri Jan 18 18:54:20 [initandlisten] 1479/2819 52%
Fri Jan 18 18:54:22 [initandlisten] 1485/2819 52%
Fri Jan 18 18:54:24 [initandlisten] 1492/2819 52%
Fri Jan 18 18:54:26 [initandlisten] 1765/2819 62%
Fri Jan 18 18:54:28 [initandlisten] 2060/2819 73%
Fri Jan 18 18:54:30 [initandlisten] 2356/2819 83%
Fri Jan 18 18:54:32 [initandlisten] 2615/2819 92%
Fri Jan 18 18:54:34 [initandlisten] 2753/2819 97%
Fri Jan 18 18:54:34 [initandlisten] closeAllFiles() finished
Fri Jan 18 18:54:34 [initandlisten] shutdown: removing fs lock...
Fri Jan 18 18:54:34 dbexit: really exiting now



 Comments   
Comment by Daniel Pasette (Inactive) [ 03/Jun/13 ]

This is likely due to running out of disk space in the dbpath while building the index, but cannot say for sure without size statistics and record of available disk space. SERVER-8432 will fix the error messaging issue.

Comment by Zac Witte [ 24/Jan/13 ]

Unfortunately I don't have the original DB anymore. I had to delete the files to complete the repair. I ended up deleting a large collection, doing the repair again, and regenerating the collection from my source files.

Comment by Aaron Staple [ 23/Jan/13 ]

Hi Zac,

Do you still have the original data files available? Is it possible send us a coll.stats() for the collection on which this error occurred?

Generated at Thu Feb 08 03:16:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.