[SERVER-16171] mongodump --repair not working under WiredTiger Created: 15/Nov/14  Updated: 11/Jul/16  Resolved: 17/Nov/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.8.0-rc0
Fix Version/s: 2.8.0-rc1

Type: Bug Priority: Critical - P2
Reporter: Bruce Lucas (Inactive) Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Even with an undamaged db it errors out with

2014-11-15T08:56:29.882-0500	writing repair of test.c to dump/test/c.bson
2014-11-15T08:56:29.891-0500	repair error: error reading collection: EOF

and mongod aborts with an invariant failure

2014-11-15T09:00:21.473-0500 I -        [conn1] Invariant failure session->cursorsOut() == 0 src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 172



 Comments   
Comment by Githook User [ 18/Nov/14 ]

Author:

{u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}

Message: SERVER-16171 skip repair_cursor1.js in sharding passthrough
Branch: master
https://github.com/mongodb/mongo/commit/aa435e0f1303cd618b1df78ab086d5e06a52b1d3

Comment by Githook User [ 17/Nov/14 ]

Author:

{u'username': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-16171: fix repairIterator releasing under WT
Branch: master
https://github.com/mongodb/mongo/commit/d6ac44b4ac2f20c9a3145f575e72ebe2528db3c6

Comment by Michael O'Brien [ 17/Nov/14 ]

matt.kangas@10gen.com --repair is supported in the tools, even without --dbpath, because of the addition of the new repairCursor command (see https://jira.mongodb.org/browse/SERVER-15544) in 2.8. So the 2.8 tools can be used with --repair on databases running 2.8 or newer.

The actual problem here is that repairCursor is crashing the server with wiredtiger.
To reproduce, in the shell:

> db.test.find()
> db.test.insert({x:1})
WriteResult({ "nInserted" : 1 })
> db.test.find()
{ "_id" : ObjectId("54696ea77a79df8707789e90"), "x" : 1 }
> db.runCommand({repairCursor:"test"})
{
	"cursor" : {
		"id" : NumberLong("10656699964"),
		"ns" : "test.test",
		"firstBatch" : [ ]
	},
	"ok" : 1
}
2014-11-16T22:42:49.722-0500 I NETWORK  DBClientCursor::init call() failed

In the server logs:

2014-11-16T22:47:28.855-0500 I -        [conn1] Invariant failure session->cursorsOut() == 0 src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 172
2014-11-16T22:47:28.858-0500 I CONTROL  [conn1]
 0x10812c529 0x1080de9f0 0x1080cd0e3 0x107f5a6cc 0x107f57641 0x107f575de 0x107ca0e16 0x107999d92 0x1080ee726 0x108160c31 0x7fff8c9fd2fc 0x7fff8c9fd279 0x7fff8c9fb4b1
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"107990000","o":"79C529"},{"b":"107990000","o":"74E9F0"},{"b":"107990000","o":"73D0E3"},{"b":"107990000","o":"5CA6CC"},{"b":"107990000","o":"5C7641"},{"b":"107990000","o":"5C75DE"},{"b":"107990000","o":"310E16"},{"b":"107990000","o":"9D92"},{"b":"107990000","o":"75E726"},{"b":"107990000","o":"7D0C31"},{"b":"7FFF8C9FA000","o":"32FC"},{"b":"7FFF8C9FA000","o":"3279"},{"b":"7FFF8C9FA000","o":"14B1"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc0", "gitVersion" : "b6c4e2491c1442b05a160acda0d78001f56a2ade", "uname" : { "sysname" : "Darwin", "release" : "14.0.0", "version" : "Darwin Kernel Version 14.0.0: Fri Sep 19 00:26:44 PDT 2014; root:xnu-2782.1.97~2/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/private/tmp/mongodb-osx-x86_64-2.8.0-rc0/./bin/mongod", "machType" : 2, "b" : "107990000", "buildId" : "D47BCDC8F92C37308186943350D47BE2" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF93EFF000", "buildId" : "DA954461EC6A3DF085516FC810627627" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF8F66C000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF9445A000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF89D3C000", "buildId" : "D381EBC669D831D380845A80A32CB748" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8B191000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF94A4F000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF87B2A000", "buildId" : "E178980139853949B7366B3378873301" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF863E8000", "buildId" : "502CF32B669B3709886208188225E4F0" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF8D9AD000", "buildId" : "19FAF435C16533749DEFD7BBA7D61DB6" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF93850000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF8F1BD000", "buildId" : "8A9889248BE735FEBF7D322E90EFE49E" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF87BA7000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF8E12E000", "buildId" : "DC0416272D92361CBABFA869A5C72293" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF8BC80000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8FB90000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF8811D000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF87F2F000", "buildId" : "C185E86274243210B5286B822577A4B8" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF8F1DA000", "buildId" : "9FBA1CE497D0347EA44393ED94512E92" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF8F315000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF87EB4000", "buildId" : "EBBF7EF680D83F8F825CB412BD6D22C0" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF8D9A0000", "buildId" : "62B70ECAE40D3C63896E7F00EC386DDB" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF89C71000", "buildId" : "B85A85D585303A93B0C34DEC41F79478" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF89B55000", "buildId" : "93E0E0A975B63904BB4E4BC7C05F4B6B" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8E327000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF89D8B000", "buildId" : "19BCC25757173502A71F95D65AFA861B" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF8C1B4000", "buildId" : "C0B2313D47BE38A9BEE62634A4F5E14B" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF8F782000", "buildId" : "29AB225BD7FB30ED960065D44B9A9442" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF88F44000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF853AE000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF8C9FA000", "buildId" : "26B1897F0CD330F3B55A37CB45062D73" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF894DE000", "buildId" : "DB9962EF889831CC9B87E01F8CE74C9D" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF91AA0000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF8F979000", "buildId" : "1DB0443659743F1686CC5FF5F390339C" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF8F020000", "buildId" : "A9E6B7D8C3273742AC5486C94218B1DF" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF8C593000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF94BB2000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF8C4D2000", "buildId" : "9437C02EA07B38C891CB299FAA63083D" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF935C3000", "buildId" : "3B60CD9074A23A5D9686B0772159792A" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF8AC53000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF87E85000", "buildId" : "88A22A0F87C63002BFBAAC0F2808B8B9" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF8E358000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }}
 mongod(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x10812c529]
 mongod(_ZN5mongo10logContextEPKc+0x100) [0x1080de9f0]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0x183) [0x1080cd0e3]
 mongod(_ZN5mongo22WiredTigerSessionCache14releaseSessionEPNS_17WiredTigerSessionE+0x18C) [0x107f5a6cc]
 mongod(_ZN5mongo22WiredTigerRecoveryUnitD2Ev+0x41) [0x107f57641]
 mongod(_ZN5mongo22WiredTigerRecoveryUnitD0Ev+0xE) [0x107f575de]
 mongod(_ZN5mongo20OperationContextImplD0Ev+0x46) [0x107ca0e16]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x632) [0x107999d92]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x526) [0x1080ee726]
 mongod(_ZN5boost12_GLOBAL__N_112thread_proxyEPv+0xB1) [0x108160c31]
 libsystem_pthread.dylib(_pthread_body+0x83) [0x7fff8c9fd2fc]
 libsystem_pthread.dylib(_pthread_body+0x0) [0x7fff8c9fd279]
 libsystem_pthread.dylib(thread_start+0xD) [0x7fff8c9fb4b1]
-----  END BACKTRACE  -----
2014-11-16T22:47:28.859-0500 I -        [conn1]
 
***aborting after invariant() failure

Generated at Thu Feb 08 03:40:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.