[SERVER-22212] Deleted record list corrupted in bucket 9, link number 0, invalid link is 792301568:79beb000, throwing Fatal Assertion Created: 14/Jan/16  Updated: 10/Feb/16  Resolved: 10/Feb/16

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: IronMan Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows server 2008 r2 Standard sp1 mongodb 2.6.5


Operating System: ALL
Participants:

 Description   

2016-01-14T09:59:23.707+0800 [conn16] DV.Snr_PortTraffic_Real Deleted record list corrupted in bucket 9, link number 0, invalid link is 792301568:79beb000, throwing Fatal Assertion
2016-01-14T09:59:23.707+0800 [conn16] DV.Snr_PortTraffic_Real Fatal Assertion 16469
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\util\log.cpp(127)                                mongo::logContext+0x9c
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\util\assert_util.cpp(139)                        mongo::fassertFailed+0x84
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(268)  mongo::NamespaceDetails::__stdAlloc+0x60e
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(189)  mongo::NamespaceDetails::alloc+0x78
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(72)                mongo::RecordStoreV1Base::insertRecord+0x6c
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1517)               mongo::BtreeBucket<mongo::BtreeData_V1>::addBucket+0x56
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1433)               mongo::BtreeBucket<mongo::BtreeData_V1>::split+0x5d
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1377)               mongo::BtreeBucket<mongo::BtreeData_V1>::insertHere+0x16a
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1866)               mongo::BtreeBucket<mongo::BtreeData_V1>::_insert+0x597
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1869)               mongo::BtreeBucket<mongo::BtreeData_V1>::_insert+0x519
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1869)               mongo::BtreeBucket<mongo::BtreeData_V1>::_insert+0x519
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1915)               mongo::BtreeBucket<mongo::BtreeData_V1>::bt_insert+0x458
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\index\btree_interface.cpp(67)                 mongo::BtreeInterfaceImpl<mongo::BtreeData_V1>::bt_insert+0x70
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\index\btree_based_access_method.cpp(87)       mongo::BtreeBasedAccessMethod::insert+0x250
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(1166)               mongo::IndexCatalog::_indexRecord+0xa0
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(1223)               mongo::IndexCatalog::indexRecord+0xa8
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(239)                   mongo::Collection::_insertDocument+0x144
2016-01-14T09:59:24.144+0800 [conn16] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(189)                   mongo::Collection::insertDocument+0x21d
2016-01-14T09:59:24.144+0800 [conn16] DV.Snr_PortTraffic_Real 
2016-01-14T09:59:24.144+0800 [conn16] 
 
***aborting after fassert() failure
 
 
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\util\signal_handlers.cpp(107)                    mongo::`anonymous namespace'::abruptQuit+0xf2
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\winsig.c(593)      raise+0x1ed
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\abort.c(81)        abort+0x18
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(268)  mongo::NamespaceDetails::__stdAlloc+0x60e
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(189)  mongo::NamespaceDetails::alloc+0x78
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(72)                mongo::RecordStoreV1Base::insertRecord+0x6c
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1517)               mongo::BtreeBucket<mongo::BtreeData_V1>::addBucket+0x56
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1433)               mongo::BtreeBucket<mongo::BtreeData_V1>::split+0x5d
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1377)               mongo::BtreeBucket<mongo::BtreeData_V1>::insertHere+0x16a
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1866)               mongo::BtreeBucket<mongo::BtreeData_V1>::_insert+0x597
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1869)               mongo::BtreeBucket<mongo::BtreeData_V1>::_insert+0x519
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1869)               mongo::BtreeBucket<mongo::BtreeData_V1>::_insert+0x519
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\structure\btree\btree.cpp(1915)               mongo::BtreeBucket<mongo::BtreeData_V1>::bt_insert+0x458
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\index\btree_interface.cpp(67)                 mongo::BtreeInterfaceImpl<mongo::BtreeData_V1>::bt_insert+0x70
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\index\btree_based_access_method.cpp(87)       mongo::BtreeBasedAccessMethod::insert+0x250
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(1166)               mongo::IndexCatalog::_indexRecord+0xa0
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(1223)               mongo::IndexCatalog::indexRecord+0xa8
2016-01-14T09:59:24.565+0800 [conn16] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(239)                   mongo::Collection::_insertDocument+0x144
2016-01-14T09:59:24.565+0800 [conn16] SEVERE: Got signal: 22 (SIGABRT).



 Comments   
Comment by Ramon Fernandez Marina [ 10/Feb/16 ]

ironman, we haven't heard back from you for some time so I'm going to close this ticket. If this happens again please post the logs for the affected node and the result of running validate() on the affected collection.

Thanks,
Ramón.

Comment by Ramon Fernandez Marina [ 19/Jan/16 ]

ironman, the message indicates that corrupted data was found and mongod shut down to avoid any further damage. If this is a bug in mongod it may be very hard to track down, as there's no way to tell when the data was corrupted. However this type of errors is often caused by a flaky storage layer, so I'd recommend you check the state of your disks/storage.

Have you tried running validate() on the DV.Snr_PortTraffic_Real collection? Looks like corruption was found on an index; if that's the case you may want to rebuild the indexes on this collection.

Note also that this part of the system was rewritten for 3.0, so if the issue appears again and you find no issues with your storage layer you may want to consider upgrading to a newer version of MongoDB (3.2.1 is the latest stable version at the time of this writing).

Thanks,
Ramón.

Comment by Craig Wilson [ 17/Jan/16 ]

Hi IronMan,

I've moved this ticket from the CSHARP project to the SERVER project. If that was incorrect, please let us know.

Craig

Generated at Thu Feb 08 03:59:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.