[SERVER-14056] Moving large collection across databases with renameCollection triggers fatal assertion Created: 27/May/14  Updated: 11/Jul/16  Resolved: 28/May/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.6.1, 2.7.1
Fix Version/s: 2.6.2, 2.7.2

Type: Bug Priority: Major - P3
Reporter: Simon Cater [X] Assignee: J Rassi
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-14068 Consolidate "copy collection" logic Closed
Operating System: ALL
Backport Completed:
Participants:

 Description   
Issue Status as of Jun 6, 2014

ISSUE SUMMARY
When moving a collection across databases, the renameCollection command does not issue a journal commit until after the data copy has finished. As a result, moving a collection of size ~0.5GB or larger across databases will trigger a fatal assertion when the journal commit buffer exceeds its maximum size.

USER IMPACT
Users running with journaling enabled can't move large collections across databases.

WORKAROUNDS
If the destination database is empty, a possible workaround is a follows:

  • use the copydb command to copy all collections to the target database
  • use the drop command to remove unneeded collections
  • use the renameCollection command to give the collection the desired name in the target database.

If the destination database is not empty there are no workarounds.

AFFECTED VERSIONS
MongoDB 2.6.0 and 2.6.1 are affected by this issue.

FIX VERSION
The fix is included in the 2.6.2 production release.

RESOLUTION DETAILS
The renameCollection command issues a journal commit after each data insertion in the target collection.

Original description

When moving a collection across databases, the renameCollection command does not issue a journal commit until after the data copy has finished. As a result, moving a collection of size ~0.5GB or larger across databases will trigger a fatal assertion when the journal commit buffer exceeds its maximum size.

Regression since 2.4 (introduced in 0abf27ae); confirmed against 2.6.1 and 2.7.1.

Reproduce with the following:

var s = '';
for (var i=0; i<1024; i++) {
  s += 'x';
}
for (var i=0; i<1000*1000; i++) {
  db.foo.insert({a: s});
}
db.adminCommand({renameCollection: "test.foo", to: "test2.foo"})

The above generates the following output in the mongod log:

2014-05-27T12:13:25.641-0400 [conn2] test2.foo Assertion failure a <= 512*1024*1024 src/mongo/util/alignedbuilder.cpp 104
2014-05-27T12:13:25.656-0400 [conn2] test2.foo 0x1006ac7ab 0x1006642c2 0x100654d52 0x1006533f9 0x1001e1dd0 0x1001e2081 0x1001d84f0 0x1001d65a1 0x1001d8159 0x1001d6628 0x1000fb54b 0x1000fc26d 0x1000f6bd5 0x10016eea1 0x1001b7605 0x1001b8375 0x1001b907c 0x1003cfc7f 0x1002a1050 0x100006854
 0   mongod261                           0x00000001006ac7ab _ZN5mongo15printStackTraceERSo + 43
 1   mongod261                           0x00000001006642c2 _ZN5mongo10logContextEPKc + 114
 2   mongod261                           0x0000000100654d52 _ZN5mongo12verifyFailedEPKcS1_j + 274
 3   mongod261                           0x00000001006533f9 _ZN5mongo14AlignedBuilder14growReallocateEj + 191
 4   mongod261                           0x00000001001e1dd0 _ZN5mongo3durL21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE + 880
 5   mongod261                           0x00000001001e2081 _ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE + 401
 6   mongod261                           0x00000001001d84f0 _ZN5mongo3durL11groupCommitEPNS_4Lock11GlobalWriteE + 272
 7   mongod261                           0x00000001001d65a1 _ZN5mongo3dur11DurableImpl9commitNowEv + 21
 8   mongod261                           0x00000001001d8159 _ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv + 113
 9   mongod261                           0x00000001001d6628 _ZN5mongo3dur11DurableImpl14commitIfNeededEb + 76
 10  mongod261                           0x00000001000fb54b _ZN5mongo18addExistingToIndexEPNS_10CollectionEPKNS_15IndexDescriptorEPNS_17IndexAccessMethodEb + 1275
 11  mongod261                           0x00000001000fc26d _ZN5mongo12buildAnIndexEPNS_10CollectionEPNS_17IndexCatalogEntryEb + 2253
 12  mongod261                           0x00000001000f6bd5 _ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE + 1231
 13  mongod261                           0x000000010016eea1 _ZN5mongo19CmdRenameCollection3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 7681
 14  mongod261                           0x00000001001b7605 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 37
 15  mongod261                           0x00000001001b8375 _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2873
 16  mongod261                           0x00000001001b907c _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1388
 17  mongod261                           0x00000001003cfc7f _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 1615
 18  mongod261                           0x00000001002a1050 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1968
 19  mongod261                           0x0000000100006854 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 308
2014-05-27T12:13:25.694-0400 [conn2] dbexception in groupCommit causing immediate shutdown: 0 assertion src/mongo/util/alignedbuilder.cpp:104
2014-05-27T12:13:25.746-0400 [conn2] SEVERE: gc1
2014-05-27T12:13:25.756-0400 [conn2] SEVERE: Got signal: 6 (Abort trap: 6).
Backtrace:0x1006ac7ab 0x1006ac50f 0x101cfb92a 0x1d12866 0x101d52dfa 0x100298b98 0x1001d873b 0x1001d65a1 0x1001d8159 0x1001d6628 0x1000fb54b 0x1000fc26d 0x1000f6bd5 0x10016eea1 0x1001b7605 0x1001b8375 0x1001b907c 0x1003cfc7f 0x1002a1050 0x100006854
 0   mongod261                           0x00000001006ac7ab _ZN5mongo15printStackTraceERSo + 43
 1   mongod261                           0x00000001006ac50f _ZN5mongo12_GLOBAL__N_110abruptQuitEi + 191
 2   libsystem_c.dylib                   0x0000000101cfb92a _sigtramp + 26
 3   ???                                 0x0000000001d12866 0x0 + 30484582
 4   libsystem_c.dylib                   0x0000000101d52dfa abort + 143
 5   mongod261                           0x0000000100298b98 _ZN5mongo10mongoAbortEPKc + 104
 6   mongod261                           0x00000001001d873b _ZN5mongo3durL11groupCommitEPNS_4Lock11GlobalWriteE + 859
 7   mongod261                           0x00000001001d65a1 _ZN5mongo3dur11DurableImpl9commitNowEv + 21
 8   mongod261                           0x00000001001d8159 _ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv + 113
 9   mongod261                           0x00000001001d6628 _ZN5mongo3dur11DurableImpl14commitIfNeededEb + 76
 10  mongod261                           0x00000001000fb54b _ZN5mongo18addExistingToIndexEPNS_10CollectionEPKNS_15IndexDescriptorEPNS_17IndexAccessMethodEb + 1275
 11  mongod261                           0x00000001000fc26d _ZN5mongo12buildAnIndexEPNS_10CollectionEPNS_17IndexCatalogEntryEb + 2253
 12  mongod261                           0x00000001000f6bd5 _ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE + 1231
 13  mongod261                           0x000000010016eea1 _ZN5mongo19CmdRenameCollection3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 7681
 14  mongod261                           0x00000001001b7605 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 37
 15  mongod261                           0x00000001001b8375 _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2873
 16  mongod261                           0x00000001001b907c _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1388
 17  mongod261                           0x00000001003cfc7f _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 1615
 18  mongod261                           0x00000001002a1050 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1968
 19  mongod261                           0x0000000100006854 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 308

Original description:

Hi,

We have a large data load that loads data from other systems in to MongoDB. At the end of the load it switched the data live by renaming the collection.

db.runCommand({renameCollection:"contacts_load.contacts_new", to:"contacts.contacts",dropTarget:true})

In version 2.4.5 this has been working without a problem.

However, in version 2.6.1 this crashes the MongoD.

Please help.

Cheers

Simon



 Comments   
Comment by Githook User [ 28/May/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14056 Cross-DB renameCollection needs to call commitIfNeeded

Fixes regression introduced in 0abf27ae.

Before 0abf27a, renameCollection used a DBDirectClient to read objects
from the source namespace. The call to DBClientCursor::next() would
issue a getmore after finishing each batch, which would reach a commit
point (in DBDirectClient::call()). The port to the new collection
interface removed the use of DBDirectClient, which removed
renameCollection's commit point.

(cherry picked from commit b27a764852873f5ecdc2dde6fffc317383c032c2)

Conflicts:
src/mongo/db/commands/rename_collection.cpp
Branch: v2.6
https://github.com/mongodb/mongo/commit/6a8781ee93fca9c8371275545d8190b1c291d5cb

Comment by Githook User [ 28/May/14 ]

Author:

{u'username': u'jrassi', u'name': u'Jason Rassi', u'email': u'rassi@10gen.com'}

Message: SERVER-14056 Cross-DB renameCollection needs to call commitIfNeeded

Fixes regression introduced in 0abf27ae.

Before 0abf27a, renameCollection used a DBDirectClient to read objects
from the source namespace. The call to DBClientCursor::next() would
issue a getmore after finishing each batch, which would reach a commit
point (in DBDirectClient::call()). The port to the new collection
interface removed the use of DBDirectClient, which removed
renameCollection's commit point.
Branch: master
https://github.com/mongodb/mongo/commit/b27a764852873f5ecdc2dde6fffc317383c032c2

Comment by Simon Cater [X] [ 27/May/14 ]

Hi Jason,

Thank you, Appreciate the help.

Cheers

Simon

Comment by J Rassi [ 27/May/14 ]

I am able to reproduce this. Thanks for the report. Please continue to watch this ticket for updates on workaround options and fix schedule.

Comment by Simon Cater [X] [ 27/May/14 ]

Hi Jason,

Thank you for another quick reply.

Item 1:

> use admin
switched to db admin
> db.getSiblingDB("contacts_load").contacts_new.stats()
{
        "ns" : "contacts_load.contacts_new",
        "count" : 46026,
        "size" : 1533330272,
        "avgObjSize" : 33314,
        "storageSize" : 1579986944,
        "numExtents" : 17,
        "nindexes" : 10,
        "lastExtentSize" : 415174656,
        "paddingFactor" : 1,
        "systemFlags" : 1,
        "userFlags" : 1,
        "totalIndexSize" : 26539296,
        "indexSizes" : {
                "_id_" : 3777312,
                "accounts._id_1" : 2788016,
                "accounts.kyc._id_1" : 2730784,
                "code_1" : 1226400,
                "user.broker_1" : 1242752,
                "accounts.products.financial.holdings.sedol_1" : 7464688,
                "accounts.products.code_1" : 2967888,
                "accounts.products.service.type_1" : 1553440,
                "accounts.code_1" : 1373568,
                "accounts.personCodes_1" : 1414448
        },
        "ok" : 1
}
>

Item 2:

> db.getSiblingDB("contacts_load").contacts_new.getIndexes()
[
        {
                "v" : 1,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "accounts._id" : 1
                },
                "name" : "accounts._id_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "accounts.kyc._id" : 1
                },
                "name" : "accounts.kyc._id_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "code" : 1
                },
                "name" : "code_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "user.broker" : 1
                },
                "name" : "user.broker_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "accounts.products.financial.holdings.sedol" : 1
                },
                "name" : "accounts.products.financial.holdings.sedol_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "accounts.products.code" : 1
                },
                "name" : "accounts.products.code_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "accounts.products.service.type" : 1
                },
                "name" : "accounts.products.service.type_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "accounts.code" : 1
                },
                "name" : "accounts.code_1",
                "ns" : "contacts_load.contacts_new"
        },
        {
                "v" : 1,
                "key" : {
                        "accounts.personCodes" : 1
                },
                "name" : "accounts.personCodes_1",
                "ns" : "contacts_load.contacts_new"
        }
]
>

Item 3:

There are a few arrays, one of which has a couple of documents with a large number of array elements. Probably around the 450 mark. But most less than 20.

In fact, i run into a bug in 2.6 when updating arrays > 128 elements. This was fixed in 2.6.1

Hope this helps

Cheers

Simon

Comment by J Rassi [ 27/May/14 ]

A couple of follow-up questions:

  • Could you post the output of running db.getSiblingDB("contacts_load").contacts_new.stats() at the shell (right before the renameCollection is attempted)?
  • Could you post the output of running db.getSiblingDB("contacts_load").contacts_new.getIndexes()?
  • Are you storing any arrays with large numbers of elements in this collection?
Comment by Simon Cater [X] [ 27/May/14 ]

Hi Jason,

Thank you for the quick reply.

I have added a comment with details and log info.

Cheers

Simon

Comment by Simon Cater [X] [ 27/May/14 ]

Hi Jason,

The target collection does exist.
It is reproducible, it happens every time, via our node script and via the mongo shell.

the log is below, after this the terminal that was running mongod is back at the command prompt.

Hope this helps.

Cheers

Simon

2014-05-27T10:17:16.521+0100 [conn2] warning: DR102 too much data written uncommitted 335.049MB
2014-05-27T10:17:17.050+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2014-05-27T10:17:17.050+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                        mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                         mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(143)                        mongo::dur::DurableImpl::declareWriteIntent+0x162
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(250)                                  mongo::dur::DurableImpl::writingPtr+0xf
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(112)  mongo::NamespaceDetails::addDeletedRec+0x3e
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(239)  mongo::NamespaceDetails::alloc+0x220
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(94)                mongo::RecordStoreV1Base::insertRecord+0x6c
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(232)                   mongo::Collection::_insertDocument+0xb5
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(189)                   mongo::Collection::insertDocument+0x21d
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(416)                mongo::IndexCatalog::IndexBuildBlock::init+0x24e
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(316)                mongo::IndexCatalog::createIndex+0x4af
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)           mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                          mongo::_execCommand+0x5e
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                          mongo::Command::execCommand+0xf41
2014-05-27T10:17:17.051+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                          mongo::_runCommands+0x4a7
2014-05-27T10:17:17.052+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                       mongo::runCommands+0x41
2014-05-27T10:17:17.052+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                       mongo::newRunQuery+0x49d
2014-05-27T10:17:17.052+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                             mongo::receivedQuery+0x44f
2014-05-27T10:17:17.052+0100 [conn2]
2014-05-27T10:17:17.052+0100 [conn2] warning: DR102 too much data written uncommitted 463.614MB
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                        mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                         mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(143)                        mongo::dur::DurableImpl::declareWriteIntent+0x162
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(250)                                  mongo::dur::DurableImpl::writingPtr+0xf
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(112)  mongo::NamespaceDetails::addDeletedRec+0x3e
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(239)  mongo::NamespaceDetails::alloc+0x220
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(94)                mongo::RecordStoreV1Base::insertRecord+0x6c
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(232)                   mongo::Collection::_insertDocument+0xb5
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(189)                   mongo::Collection::insertDocument+0x21d
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(416)                mongo::IndexCatalog::IndexBuildBlock::init+0x24e
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(316)                mongo::IndexCatalog::createIndex+0x4af
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)           mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:17.509+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                          mongo::_execCommand+0x5e
2014-05-27T10:17:17.510+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                          mongo::Command::execCommand+0xf41
2014-05-27T10:17:17.510+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                          mongo::_runCommands+0x4a7
2014-05-27T10:17:17.510+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                       mongo::runCommands+0x41
2014-05-27T10:17:17.510+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                       mongo::newRunQuery+0x49d
2014-05-27T10:17:17.510+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                             mongo::receivedQuery+0x44f
2014-05-27T10:17:17.510+0100 [conn2]
2014-05-27T10:17:17.510+0100 [conn2] warning: DR102 too much data written uncommitted 575.275MB
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                        mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                         mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(143)                        mongo::dur::DurableImpl::declareWriteIntent+0x162
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(250)                                  mongo::dur::DurableImpl::writingPtr+0xf
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(112)  mongo::NamespaceDetails::addDeletedRec+0x3e
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(239)  mongo::NamespaceDetails::alloc+0x220
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(94)                mongo::RecordStoreV1Base::insertRecord+0x6c
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(232)                   mongo::Collection::_insertDocument+0xb5
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(189)                   mongo::Collection::insertDocument+0x21d
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(416)                mongo::IndexCatalog::IndexBuildBlock::init+0x24e
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(316)                mongo::IndexCatalog::createIndex+0x4af
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)           mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                          mongo::_execCommand+0x5e
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                          mongo::Command::execCommand+0xf41
2014-05-27T10:17:17.935+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                          mongo::_runCommands+0x4a7
2014-05-27T10:17:17.936+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                       mongo::runCommands+0x41
2014-05-27T10:17:17.936+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                       mongo::newRunQuery+0x49d
2014-05-27T10:17:17.936+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                             mongo::receivedQuery+0x44f
2014-05-27T10:17:17.936+0100 [conn2]
2014-05-27T10:17:17.936+0100 [conn2] warning: DR102 too much data written uncommitted 575.279MB
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                        mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                         mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(143)                        mongo::dur::DurableImpl::declareWriteIntent+0x162
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(250)                                  mongo::dur::DurableImpl::writingPtr+0xf
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(112)  mongo::NamespaceDetails::addDeletedRec+0x3e
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(239)  mongo::NamespaceDetails::alloc+0x220
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(94)                mongo::RecordStoreV1Base::insertRecord+0x6c
2014-05-27T10:17:18.361+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(232)                   mongo::Collection::_insertDocument+0xb5
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(189)                   mongo::Collection::insertDocument+0x21d
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(416)                mongo::IndexCatalog::IndexBuildBlock::init+0x24e
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(316)                mongo::IndexCatalog::createIndex+0x4af
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)           mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                          mongo::_execCommand+0x5e
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                          mongo::Command::execCommand+0xf41
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                          mongo::_runCommands+0x4a7
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                       mongo::runCommands+0x41
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                       mongo::newRunQuery+0x49d
2014-05-27T10:17:18.362+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                             mongo::receivedQuery+0x44f
2014-05-27T10:17:18.362+0100 [conn2]
2014-05-27T10:17:18.362+0100 [conn2] warning: DR102 too much data written uncommitted 575.283MB
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                        mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                         mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(143)                        mongo::dur::DurableImpl::declareWriteIntent+0x162
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(250)                                  mongo::dur::DurableImpl::writingPtr+0xf
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(112)  mongo::NamespaceDetails::addDeletedRec+0x3e
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(239)  mongo::NamespaceDetails::alloc+0x220
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(94)                mongo::RecordStoreV1Base::insertRecord+0x6c
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(232)                   mongo::Collection::_insertDocument+0xb5
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(189)                   mongo::Collection::insertDocument+0x21d
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(416)                mongo::IndexCatalog::IndexBuildBlock::init+0x24e
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(316)                mongo::IndexCatalog::createIndex+0x4af
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)           mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                          mongo::_execCommand+0x5e
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                          mongo::Command::execCommand+0xf41
2014-05-27T10:17:18.786+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                          mongo::_runCommands+0x4a7
2014-05-27T10:17:18.787+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                       mongo::runCommands+0x41
2014-05-27T10:17:18.787+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                       mongo::newRunQuery+0x49d
2014-05-27T10:17:18.787+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                             mongo::receivedQuery+0x44f
2014-05-27T10:17:18.787+0100 [conn2]
2014-05-27T10:17:18.787+0100 [conn2] warning: DR102 too much data written uncommitted 575.287MB
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                         mongo::printStackTrace+0x43
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                        mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                         mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(143)                        mongo::dur::DurableImpl::declareWriteIntent+0x162
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(250)                                  mongo::dur::DurableImpl::writingPtr+0xf
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(112)  mongo::NamespaceDetails::addDeletedRec+0x3e
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\catalog\namespace_details.cpp(239)  mongo::NamespaceDetails::alloc+0x220
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(203)               mongo::SimpleRecordStoreV1::allocRecord+0x87
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\structure\record_store.cpp(94)                mongo::RecordStoreV1Base::insertRecord+0x6c
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(232)                   mongo::Collection::_insertDocument+0xb5
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\collection.cpp(189)                   mongo::Collection::insertDocument+0x21d
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(416)                mongo::IndexCatalog::IndexBuildBlock::init+0x24e
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(316)                mongo::IndexCatalog::createIndex+0x4af
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)           mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                          mongo::_execCommand+0x5e
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                          mongo::Command::execCommand+0xf41
2014-05-27T10:17:19.212+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                          mongo::_runCommands+0x4a7
2014-05-27T10:17:19.213+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                       mongo::runCommands+0x41
2014-05-27T10:17:19.213+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                       mongo::newRunQuery+0x49d
2014-05-27T10:17:19.213+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                             mongo::receivedQuery+0x44f
2014-05-27T10:17:19.213+0100 [conn2]
2014-05-27T10:17:19.213+0100 [conn2] build index on: contacts.contacts properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "contacts.contacts" }
2014-05-27T10:17:19.213+0100 [conn2]     building index using bulk method
2014-05-27T10:17:19.213+0100 [conn2] warning: DR102 too much data written uncommitted 575.291MB
2014-05-27T10:17:19.636+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                           mongo::printStackTrace+0x43
2014-05-27T10:17:19.636+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                          mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:19.636+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                           mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:19.636+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(133)                          mongo::dur::unspoolWriteIntents+0x62
2014-05-27T10:17:19.636+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(355)                                    mongo::dur::DurableImpl::commitIfNeeded+0x47
2014-05-27T10:17:19.636+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(170)                   mongo::addExistingToIndex+0x60f
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)                   mongo::buildAnIndex+0x621
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(341)                  mongo::IndexCatalog::createIndex+0x776
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)             mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                            mongo::_execCommand+0x5e
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                            mongo::Command::execCommand+0xf41
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                            mongo::_runCommands+0x4a7
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                         mongo::runCommands+0x41
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                         mongo::newRunQuery+0x49d
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                               mongo::receivedQuery+0x44f
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(434)                               mongo::assembleResponse+0x30b
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\db\db.cpp(202)                                     mongo::MyMessageHandler::process+0x111
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\mongo\util\net\message_server_port.cpp(210)              mongo::PortMessageServer::handleIncomingMsg+0x677
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)  boost::`anonymous namespace'::thread_start_function+0x21
2014-05-27T10:17:19.637+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
2014-05-27T10:17:19.637+0100 [conn2]
2014-05-27T10:17:19.637+0100 [conn2] warning: DR102 too much data written uncommitted 575.295MB
2014-05-27T10:17:20.059+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                           mongo::printStackTrace+0x43
2014-05-27T10:17:20.059+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                          mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:20.059+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                           mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:20.059+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(133)                          mongo::dur::unspoolWriteIntents+0x62
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(355)                                    mongo::dur::DurableImpl::commitIfNeeded+0x47
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(170)                   mongo::addExistingToIndex+0x60f
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)                   mongo::buildAnIndex+0x621
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(341)                  mongo::IndexCatalog::createIndex+0x776
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)             mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                            mongo::_execCommand+0x5e
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                            mongo::Command::execCommand+0xf41
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                            mongo::_runCommands+0x4a7
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                         mongo::runCommands+0x41
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                         mongo::newRunQuery+0x49d
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                               mongo::receivedQuery+0x44f
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(434)                               mongo::assembleResponse+0x30b
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\db\db.cpp(202)                                     mongo::MyMessageHandler::process+0x111
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\mongo\util\net\message_server_port.cpp(210)              mongo::PortMessageServer::handleIncomingMsg+0x677
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)  boost::`anonymous namespace'::thread_start_function+0x21
2014-05-27T10:17:20.060+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
2014-05-27T10:17:20.060+0100 [conn2]
2014-05-27T10:17:20.060+0100 [conn2] warning: DR102 too much data written uncommitted 575.3MB
2014-05-27T10:17:20.484+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                           mongo::printStackTrace+0x43
2014-05-27T10:17:20.484+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                          mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:20.484+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                           mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(133)                          mongo::dur::unspoolWriteIntents+0x62
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(355)                                    mongo::dur::DurableImpl::commitIfNeeded+0x47
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(170)                   mongo::addExistingToIndex+0x60f
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)                   mongo::buildAnIndex+0x621
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(341)                  mongo::IndexCatalog::createIndex+0x776
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)             mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                            mongo::_execCommand+0x5e
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                            mongo::Command::execCommand+0xf41
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                            mongo::_runCommands+0x4a7
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                         mongo::runCommands+0x41
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                         mongo::newRunQuery+0x49d
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                               mongo::receivedQuery+0x44f
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(434)                               mongo::assembleResponse+0x30b
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\db\db.cpp(202)                                     mongo::MyMessageHandler::process+0x111
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\mongo\util\net\message_server_port.cpp(210)              mongo::PortMessageServer::handleIncomingMsg+0x677
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)  boost::`anonymous namespace'::thread_start_function+0x21
2014-05-27T10:17:20.485+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
2014-05-27T10:17:20.485+0100 [conn2]
2014-05-27T10:17:20.485+0100 [conn2] warning: DR102 too much data written uncommitted 575.304MB
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                           mongo::printStackTrace+0x43
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(286)                          mongo::dur::CommitJob::note+0x1ae
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(73)                           mongo::dur::ThreadLocalIntents::_unspool+0x4f
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_commitjob.cpp(133)                          mongo::dur::unspoolWriteIntents+0x62
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(355)                                    mongo::dur::DurableImpl::commitIfNeeded+0x47
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(170)                   mongo::addExistingToIndex+0x60f
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)                   mongo::buildAnIndex+0x621
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(341)                  mongo::IndexCatalog::createIndex+0x776
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)             mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                            mongo::_execCommand+0x5e
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                            mongo::Command::execCommand+0xf41
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                            mongo::_runCommands+0x4a7
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                         mongo::runCommands+0x41
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)                         mongo::newRunQuery+0x49d
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(269)                               mongo::receivedQuery+0x44f
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(434)                               mongo::assembleResponse+0x30b
2014-05-27T10:17:20.910+0100 [conn2] mongod.exe  ...\src\mongo\db\db.cpp(202)                                     mongo::MyMessageHandler::process+0x111
2014-05-27T10:17:20.911+0100 [conn2] mongod.exe  ...\src\mongo\util\net\message_server_port.cpp(210)              mongo::PortMessageServer::handleIncomingMsg+0x677
2014-05-27T10:17:20.911+0100 [conn2] mongod.exe  ...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)  boost::`anonymous namespace'::thread_start_function+0x21
2014-05-27T10:17:20.911+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c(314)      _callthreadstartex+0x17
2014-05-27T10:17:20.911+0100 [conn2]
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.308MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.312MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.316MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.32MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.324MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.328MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.341MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.345MB
2014-05-27T10:17:20.911+0100 [conn2] warning: DR102 too much data written uncommitted 575.353MB
2014-05-27T10:17:21.108+0100 [conn2] contacts.contacts warning assertion failure a <= 256*1024*1024 src\mongo\util\alignedbuilder.cpp 103
2014-05-27T10:17:21.531+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                mongo::printStackTrace+0x43
2014-05-27T10:17:21.531+0100 [conn2] mongod.exe  ...\src\mongo\util\log.cpp(127)                       mongo::logContext+0x9c
2014-05-27T10:17:21.531+0100 [conn2] mongod.exe  ...\src\mongo\util\assert_util.cpp(102)               mongo::wasserted+0x1b4
2014-05-27T10:17:21.531+0100 [conn2] mongod.exe  ...\src\mongo\util\alignedbuilder.cpp(104)            mongo::AlignedBuilder::growReallocate+0x75
2014-05-27T10:17:21.531+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_preplogbuffer.cpp(114)           mongo::dur::prepBasicWrite_inlock+0x2ba
2014-05-27T10:17:21.531+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_preplogbuffer.cpp(161)           mongo::dur::prepBasicWrites+0xea
2014-05-27T10:17:21.531+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_preplogbuffer.cpp(202)           mongo::dur::PREPLOGBUFFER+0x84
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(693)                         mongo::dur::_groupCommit+0x111
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(751)                         mongo::dur::groupCommit+0x29
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(228)                         mongo::dur::DurableImpl::commitNow+0x15
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(327)                         mongo::dur::DurableImpl::_aCommitIsNeeded+0x21b
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(170)        mongo::addExistingToIndex+0x60f
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)        mongo::buildAnIndex+0x621
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(341)       mongo::IndexCatalog::createIndex+0x776
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)  mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                 mongo::_execCommand+0x5e
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                 mongo::Command::execCommand+0xf41
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                 mongo::_runCommands+0x4a7
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)              mongo::runCommands+0x41
2014-05-27T10:17:21.532+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)              mongo::newRunQuery+0x49d
2014-05-27T10:17:21.532+0100 [conn2] contacts.contacts
2014-05-27T10:17:21.645+0100 [conn2] rate limiting wassert
2014-05-27T10:17:21.800+0100 [conn2] contacts.contacts Assertion failure a <= 512*1024*1024 src\mongo\util\alignedbuilder.cpp 104
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                mongo::printStackTrace+0x43
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\util\log.cpp(127)                       mongo::logContext+0x9c
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\util\assert_util.cpp(115)               mongo::verifyFailed+0x14a
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\util\alignedbuilder.cpp(104)            mongo::AlignedBuilder::growReallocate+0xc5
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_preplogbuffer.cpp(114)           mongo::dur::prepBasicWrite_inlock+0x2ba
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_preplogbuffer.cpp(161)           mongo::dur::prepBasicWrites+0xea
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dur_preplogbuffer.cpp(202)           mongo::dur::PREPLOGBUFFER+0x84
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(693)                         mongo::dur::_groupCommit+0x111
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(751)                         mongo::dur::groupCommit+0x29
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(228)                         mongo::dur::DurableImpl::commitNow+0x15
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(327)                         mongo::dur::DurableImpl::_aCommitIsNeeded+0x21b
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(170)        mongo::addExistingToIndex+0x60f
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)        mongo::buildAnIndex+0x621
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(341)       mongo::IndexCatalog::createIndex+0x776
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)  mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                 mongo::_execCommand+0x5e
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                 mongo::Command::execCommand+0xf41
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                 mongo::_runCommands+0x4a7
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)              mongo::runCommands+0x41
2014-05-27T10:17:22.219+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(423)              mongo::newRunQuery+0x49d
2014-05-27T10:17:22.219+0100 [conn2] contacts.contacts
2014-05-27T10:17:22.219+0100 [conn2] dbexception in groupCommit causing immediate shutdown: 0 assertion src\mongo\util\alignedbuilder.cpp:104
2014-05-27T10:17:22.220+0100 [conn2] SEVERE: gc1
2014-05-27T10:17:22.643+0100 [conn2] mongod.exe  ...\src\mongo\util\stacktrace.cpp(169)                                      mongo::printStackTrace+0x43
2014-05-27T10:17:22.643+0100 [conn2] mongod.exe  ...\src\mongo\util\signal_handlers.cpp(107)                                 mongo::`anonymous namespace'::abruptQuit+0xf2
2014-05-27T10:17:22.643+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\winsig.c(593)                   raise+0x1ed
2014-05-27T10:17:22.643+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\src\abort.c(81)                     abort+0x18
2014-05-27T10:17:22.643+0100 [conn2] mongod.exe  ...\src\mongo\db\instance.cpp(342)                                          mongo::mongoAbort+0x6e
2014-05-27T10:17:22.643+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(755)                                               `mongo::dur::groupCommit'::`1'::catch$0+0xa0
2014-05-27T10:17:22.643+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\eh\amd64\handlers.asm(44)  _CallSettingFrame+0x20
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  f:\dd\vctools\crt_bld\self_64_amd64\crt\prebuild\eh\frame.cpp(1337)         __CxxCallCatchBlock+0xeb
2014-05-27T10:17:22.644+0100 [conn2] ntdll.dll                                                                               RtlRestoreContext+0x2e2
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(751)                                               mongo::dur::groupCommit+0x29
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(228)                                               mongo::dur::DurableImpl::commitNow+0x15
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\dur.cpp(327)                                               mongo::dur::DurableImpl::_aCommitIsNeeded+0x21b
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(170)                              mongo::addExistingToIndex+0x60f
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_create.cpp(258)                              mongo::buildAnIndex+0x621
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\catalog\index_catalog.cpp(341)                             mongo::IndexCatalog::createIndex+0x776
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\commands\rename_collection.cpp(311)                        mongo::CmdRenameCollection::run+0x1f6f
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1357)                                       mongo::_execCommand+0x5e
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1575)                                       mongo::Command::execCommand+0xf41
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\dbcommands.cpp(1650)                                       mongo::_runCommands+0x4a7
2014-05-27T10:17:22.644+0100 [conn2] mongod.exe  ...\src\mongo\db\query\new_find.cpp(120)                                    mongo::runCommands+0x41
2014-05-27T10:17:22.644+0100 [conn2] SEVERE: Got signal: 22 (SIGABRT).
Backtrace:

Comment by J Rassi [ 27/May/14 ]

Hi,

I'd like to gather some additional information to further diagnose this issue:

  • Could you post the excerpt from mongod log generated when you observe this crash?
  • Does the target collection exist when you are performing the rename?
  • Is this crash reproducible?

Thanks.
~ Jason Rassi

Generated at Thu Feb 08 03:33:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.