[SERVER-4198] Assertion failure isOk() db/pdfile.h 300 Created: 03/Nov/11  Updated: 15/Aug/12  Resolved: 14/May/12

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kyle Liu Assignee: Scott Hernandez (Inactive)
Resolution: Incomplete Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Mac OSX 10.7.2, Darwin Kernel Version 9.6.0


Attachments: Zip Archive db-admin.zip    
Operating System: OS X
Participants:

 Description   

I've used mongodb a few days, it has been working fine until today.

I got a error when querying a database.

Wed Nov 2 17:56:44 [initandlisten] connection accepted from 127.0.0.1:49234 #1
Wed Nov 2 17:56:45 [conn1] command admin.$cmd command:

{ listDatabases: 1 }

ntoreturn:1 reslen:338 813ms
Wed Nov 2 17:56:46 [conn1] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x100603394 0x100148d8c 0x10060a554
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1 + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
18 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
19 mongod 0x000000010060a554 thread_proxy + 132
Wed Nov 2 17:56:46 [conn1] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 17:56:51 [conn1] end connection 127.0.0.1:49234

I restart the mongod process but no help.

I tried to repair the db but also got the error.

bin: $ ./mongod --repair --dbpath ../data/db
Wed Nov 2 18:10:51 [initandlisten] MongoDB starting : pid=678 port=27017 dbpath=../data/db 64-bit host=KTs-MacBook-Pro.local
Wed Nov 2 18:10:51 [initandlisten] db version v2.0.0, pdfile version 4.5
Wed Nov 2 18:10:51 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
Wed Nov 2 18:10:51 [initandlisten] build info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
Wed Nov 2 18:10:51 [initandlisten] options:

{ dbpath: "../data/db", repair: true }

Wed Nov 2 18:10:51 [initandlisten] journal dir=../data/db/journal
Wed Nov 2 18:10:51 [initandlisten] recover : no journal files present, no recovery needed
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] need to upgrade database admin with pdfile version 4.5, new version: 4.5
Wed Nov 2 18:10:51 [initandlisten] starting upgrade
Wed Nov 2 18:10:51 [initandlisten] admin repairDatabase admin
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x1003e480f 0x1000ed8ec 0x1000c8b73
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1 + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x00000001003e480f _ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs + 111
18 mongod 0x00000001000ed8ec _ZN5mongo14DBClientCursor4initEv + 76
19 mongod 0x00000001000c8b73 _ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii + 739
Wed Nov 2 18:10:51 [initandlisten] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 18:10:51 [initandlisten] admin clone failed for admin with error: query failed admin.system.namespaces
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin Assertion failure doDBUpgrade( dbName , errmsg , h ) db/db.cpp 298
0x10007d703 0x100087eec 0x1005f4f2f 0x1005f5c3c 0x1005f6274 0x1005f7d6f 0x1000013c4
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001005f4f2f _ZN5mongo30repairDatabasesAndCheckVersionEv + 2063
3 mongod 0x00000001005f5c3c _ZN5mongo14_initAndListenEi + 1036
4 mongod 0x00000001005f6274 _ZN5mongo13initAndListenEi + 36
5 mongod 0x00000001005f7d6f main + 3647
6 mongod 0x00000001000013c4 start + 52
Wed Nov 2 18:10:51 [initandlisten] exception in initAndListen: 0 assertion db/db.cpp:298, terminating
Wed Nov 2 18:10:51 dbexit:
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close listening sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to flush diaglog...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: waiting for fs preallocator...
Wed Nov 2 18:10:51 [initandlisten] shutdown: lock for final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: closing all files...
Wed Nov 2 18:10:51 [initandlisten] closeAllFiles() finished
Wed Nov 2 18:10:51 [initandlisten] journalCleanup...
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] shutdown: removing fs lock...
Wed Nov 2 18:10:51 dbexit: really exiting now

The mongodb version is v2.0.0, pdfile version is 4.5, git version is 695c67dff0ffc361b8568a13366f027caa406222.

There are 5 databases in the mongodb, but only one database to be wrong.

I also tried to dump data by mongodump, but got empty bson files.



 Comments   
Comment by Michael Mikhjian [ 15/Aug/12 ]

I had this same issue recently (https://groups.google.com/forum/?fromgroups#!topic/mongodb-user/QCbFaIXcLTM). I was able to fix the issue by removing the master=true setting then running a repair.

Comment by Scott Hernandez (Inactive) [ 02/Jul/12 ]

It would be those logs, but please increase to level 5.

Comment by Tarang [ 02/Jul/12 ]

Hi Scott, i'll create a new issue but where can I find these logs? I looked at it in one degree of verbose and got the following when the error occurs:

Assertion failure isOk() tools/../db/pdfile.h 300
0x10007628f 0x100081cc2 0x1005a806b 0x1005a8a01 0x1005aa802 0x1005ab4b4 0x1005aba1c 0x10059e297 0x1005a5a2a 0x100001694
0 mongodump 0x000000010007628f _ZN5mongo12sayDbContextEPKc + 191
1 mongodump 0x0000000100081cc2 _ZN5mongo8assertedEPKcS1_j + 322
2 mongodump 0x00000001005a806b _ZN4Dump13_repairExtentEPN5mongo8DatabaseESsbNS0_7DiskLocERNS_6WriterE + 2587
3 mongodump 0x00000001005a8a01 _ZN4Dump7_repairEPN5mongo8DatabaseESsN5boost11filesystem210basic_pathISsNS4_11path_traitsEEE + 929
4 mongodump 0x00000001005aa802 _ZN4Dump7_repairESs + 1794
5 mongodump 0x00000001005ab4b4 _ZN4Dump6repairEv + 1156
6 mongodump 0x00000001005aba1c _ZN4Dump3runEv + 492
7 mongodump 0x000000010059e297 _ZN5mongo4Tool4mainEiPPc + 5447
8 mongodump 0x00000001005a5a2a main + 58
9 mongodump 0x0000000100001694 start + 52

Comment by Scott Hernandez (Inactive) [ 02/Jul/12 ]

Tarang, can you please create a new issue and attach your logs?

Comment by Tarang [ 01/Jul/12 ]

I also have this same problem see:

https://groups.google.com/forum/?fromgroups#!topic/mongodb-user/1Sw0Nf5e3aA

It results in being unable to repair, unable to export, unable to index the database

Comment by Ian Whalen (Inactive) [ 09/Apr/12 ]

Hi Christian, could you please attach the full logs when you get a chance?

Comment by Eliot Horowitz (Inactive) [ 23/Feb/12 ]

Can you attach the full logs?

Comment by Christian Klukas [ 22/Feb/12 ]

I am now deleting some files from the problematic file collection, during this process I get every once in a while the error "Invalid BSONObj size: -7127274116" (or similar) and also even more often "Asseration failure isOk() db/pdfile.h 300". So it seems that over time invalid data structures have been created or some other problem becomes apparant.

Comment by Christian Klukas [ 22/Feb/12 ]

I forgot to mention that the server was running fine with journalling enabled until the error appeared.

Comment by Christian Klukas [ 22/Feb/12 ]

I have the same problem with MongoDB 2.0.2 and 2.0.3-rc1 (see http://groups.google.com/group/mongodb-user/browse_thread/thread/98873692e898ea6a)

Comment by Eliot Horowitz (Inactive) [ 01/Feb/12 ]

Was this server ever shut down unsafely or crashed without journalling?

Generated at Thu Feb 08 03:05:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.