[SERVER-20531] Mongodb server crash: Invariant failure res.existing Created: 21/Sep/15  Updated: 13/Oct/15  Resolved: 25/Sep/15

Status: Closed
Project: Core Server
Component/s: Querying, Write Ops
Affects Version/s: 3.0.6
Fix Version/s: 3.0.7

Type: Bug Priority: Major - P3
Reporter: Peak Ji Assignee: David Storch
Resolution: Done Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.5 x64


Issue Links:
Depends
depends on DOCS-6274 Add dochub link pointing to http://do... Closed
Backwards Compatibility: Fully Compatible
Operating System: Linux
Sprint: QuInt A (10/12/15)
Participants:

 Description   

We are running a test server using v3.0.6 with WiredTiger in standalone mode, the docs are like:

{"_id":"Article Title", "status":0}

There are about 5 million docs in the collection, and we call findAndModify to get the doc and set the "status" field to 1 every second, from two other test servers.

After running for about 2 hours, the mongod crashed and we got these errors:

2015-09-21T22:51:25.539+0800 I -        [conn76] Invariant failure res.existing src/mongo/db/commands/find_and_modify.cpp 492
2015-09-21T22:51:25.545+0800 I CONTROL  [conn76] 
 0xf75549 0xf12271 0xef6022 0x956049 0x956718 0x9cf044 0x9cffcd 0x9d0cdb 0xba223a 0xab3ba0 0x7fb86d 0xf268ab 0x3fe0a07a51 0x3fe06e89ad
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B75549"},{"b":"400000","o":"B12271"},{"b":"400000","o":"AF6022"},{"b":"400000","o":"556049"},{"b":"400000","o":"556718"},{"b":"400000","o":"5CF044"},{"b":"400000","o":"5CFFCD"},{"b":"400000","o":"5D0CDB"},{"b":"400000","o":"7A223A"},{"b":"400000","o":"6B3BA0"},{"b":"400000","o":"3FB86D"},{"b":"400000","o":"B268AB"},{"b":"3FE0A00000","o":"7A51"},{"b":"3FE0600000","o":"E89AD"}],"processInfo":{ "mongodbVersion" : "3.0.6", "gitVersion" : "1ef45a23a4c5e3480ac919b28afcba3c615488f2", "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.3.1.el6.x86_64", "version" : "#1 SMP Thu Aug 13 22:55:16 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7F7AA372EDE22BA34234ADA10A8AF2E665681140" }, { "b" : "7FFCD0CC2000", "elfType" : 3, "buildId" : "77385010026236DADA9BFEADC4717B7518E8BB29" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D467973C46E563CDCF64B5F12B2D6A50C7A25BA1" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AC11C021C127E84A2724F3A750461A2183B1069" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "78A423B7DD3C00F7A77027D8D764BB432B57F1E7" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "58C5A5FF5C82D7BE3113BE36DD87C7004E3C4DB1" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "B5AE05CEDC0CE917F50A3A468CFA2ACD8592E8F6" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "C03877A9EE01DDC572E2B0F55F64C757773CF8D6" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9350579A4970FA47F3144AD8F40B183B0954497D" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "285709B5B3FC083FE60B19F5DC2E5282B1708AB1" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "04202A4A8BE624D2193E812A25589E2DD02D5B5C" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F704FA7D21D05EF31E90FB4890FCA7F3D91DA138" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf75549]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf12271]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xef6022]
 mongod(_ZN5mongo16CmdFindAndModify7runImplEPNS_16OperationContextERKSsS4_RKNS_7BSONObjES7_S7_S7_bbbRNS_14BSONObjBuilderERSs+0x1BA9) [0x956049]
 mongod(_ZN5mongo16CmdFindAndModify3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x358) [0x956718]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9cf044]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC1D) [0x9cffcd]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9d0cdb]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_+0x77A) [0xba223a]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xB10) [0xab3ba0]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xDD) [0x7fb86d]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x34B) [0xf268ab]
 libpthread.so.0(+0x7A51) [0x3fe0a07a51]
 libc.so.6(clone+0x6D) [0x3fe06e89ad]
-----  END BACKTRACE  -----
2015-09-21T22:51:25.545+0800 I -        [conn76] 
 
***aborting after invariant() failure



 Comments   
Comment by Githook User [ 25/Sep/15 ]

Author:

{u'username': u'dstorch', u'name': u'David Storch', u'email': u'david.storch@10gen.com'}

Message: SERVER-20531 uassert rather than invariant if findAndModify fails due to missing entries in the _id index
Branch: v3.0
https://github.com/mongodb/mongo/commit/ea2cc1388cf707512a04f4437def3aedd78c7211

Comment by Peak Ji [ 22/Sep/15 ]

Great! I'll check out the dev brach. We manually filtered all fields larger than 1024 bytes and everything is working well so far.
Thanks for your attention, and looking forward to the new version

Peak

Comment by J Rassi [ 22/Sep/15 ]

Thanks so much for the reproducible case. I can confirm that I am able to reproduce this issue by repeating the commands in your shell session against a 3.0.6 mongod instance started with the "--storageEngine=wiredTiger --setParameter failIndexKeyTooLong=false" options.

I am also unable to reproduce this issue on the development branch, and suspect that it was fixed in 3.1.2 by SERVER-16063. As a result, I'm tentatively assigning this issue a fixVersion of 3.0.7.

Please continue to watch this ticket for updates, and thanks again.

~ Jason Rassi

Comment by Peak Ji [ 22/Sep/15 ]

Seems like the findAndModify command has the limit of 1024 bytes on indexed fields even we are not using that index in the query?

Comment by Peak Ji [ 22/Sep/15 ]

I think I found whats going wrong exactly, it's caused by _ids larger than 1024 bytes.

Only two steps needed to reproduce the error using the mongo shell, tested on both Mac OS X and Linux, with Mongodb 3.0.5/3.0.6 WiredTiger:

> use testdb
switched to db testdb
> db.test.save({ "_id" : "http://game.feng.com/gamenew/default/statisticsDownload-id-1707569-url-aHR0cCUzQSUyRiUyRnd3dy50dXR1YXBwLmNvbSUyRiUzRnIlM0RhcGklMkZzaGFyZUlwYSUyNnR1dHVpZCUzREhSTkhUS01TUkolMjZ0dXR1YXBwdXJsJTNEaHR0cCUyNTNBJTI1MkYlMjUyRnd3dy53ZWFwcC5jb20lMjUyRmluZGV4LnBocCUyNTNGciUyNTNEYXBpJTI1MkZkb3dubG9hZCUyNTI2ZW50aXR5aWQlMjUzRDE3MDc1NjklMjZ0dXR1YXBwaWNvbiUzRGh0dHAlM0ElMkYlMkZ3d3cud2VhcHAuY29tJTJGJTJGcGljdHVyZSUyRmFwcF9pb3MlMkZjbiUyRjAwMSUyRjcwJTJGNzUlMkY2OSUyRm16bC5jb3Zlci4xNzV4MTc1LTc1LmpwZyUyNnR1dHVhcHB2ZXIlM0QlMjZ0dXR1YXBwbmFtZSUzRCUyNUU3JTI1OEMlMjVBQiUyNUU1JTI1OTIlMjU4QyUyNUU4JTI1ODAlMjU4MSUyNUU5JTI1QkMlMjVBMCUyNUU1JTI1QUUlMjU5OCUyNUU2JTI1OTYlMjVCOSUyNUU2JTI1ODklMjU4QiUyNUU2JTI1QjglMjVCOC0lMjVFNiUyNTk2JTI1QjAlMjVFNSUyNTlDJTI1QkElMjVFNiUyNTk5JTI1QUYlMjVFRiUyNUJDJTI1OEMlMjVFNiUyNTk2JTI1QjAlMjVFOCUyNUE3JTI1OTIlMjVFOCUyNTg5JTI1QjIlMjVFRiUyNUJDJTI1OEMlMjVFNiUyNTk2JTI1QjAlMjVFNyUyNThFJTI1QTklMjVFNiUyNUIzJTI1OTUlMjVFRiUyNUJDJTI1OEMlMjVFNyUyNTlCJTI1OUIlMjVFNSUyNUE0JTI1OEYlMjVFNiUyNTlEJTI1QTUlMjVFOCUyNUEyJTI1QUQlMjZ0dXR1YXBwaW5mbyUzRCUyNnR1dHVmbGFnJTNEMQ%3D%3D.shtml", "f" : 0 })
 
 
WriteResult({
	"nMatched" : 0,
	"nUpserted" : 1,
	"nModified" : 0,
	"_id" : "http://game.feng.com/gamenew/default/statisticsDownload-id-1707569-url-aHR0cCUzQSUyRiUyRnd3dy50dXR1YXBwLmNvbSUyRiUzRnIlM0RhcGklMkZzaGFyZUlwYSUyNnR1dHVpZCUzREhSTkhUS01TUkolMjZ0dXR1YXBwdXJsJTNEaHR0cCUyNTNBJTI1MkYlMjUyRnd3dy53ZWFwcC5jb20lMjUyRmluZGV4LnBocCUyNTNGciUyNTNEYXBpJTI1MkZkb3dubG9hZCUyNTI2ZW50aXR5aWQlMjUzRDE3MDc1NjklMjZ0dXR1YXBwaWNvbiUzRGh0dHAlM0ElMkYlMkZ3d3cud2VhcHAuY29tJTJGJTJGcGljdHVyZSUyRmFwcF9pb3MlMkZjbiUyRjAwMSUyRjcwJTJGNzUlMkY2OSUyRm16bC5jb3Zlci4xNzV4MTc1LTc1LmpwZyUyNnR1dHVhcHB2ZXIlM0QlMjZ0dXR1YXBwbmFtZSUzRCUyNUU3JTI1OEMlMjVBQiUyNUU1JTI1OTIlMjU4QyUyNUU4JTI1ODAlMjU4MSUyNUU5JTI1QkMlMjVBMCUyNUU1JTI1QUUlMjU5OCUyNUU2JTI1OTYlMjVCOSUyNUU2JTI1ODklMjU4QiUyNUU2JTI1QjglMjVCOC0lMjVFNiUyNTk2JTI1QjAlMjVFNSUyNTlDJTI1QkElMjVFNiUyNTk5JTI1QUYlMjVFRiUyNUJDJTI1OEMlMjVFNiUyNTk2JTI1QjAlMjVFOCUyNUE3JTI1OTIlMjVFOCUyNTg5JTI1QjIlMjVFRiUyNUJDJTI1OEMlMjVFNiUyNTk2JTI1QjAlMjVFNyUyNThFJTI1QTklMjVFNiUyNUIzJTI1OTUlMjVFRiUyNUJDJTI1OEMlMjVFNyUyNTlCJTI1OUIlMjVFNSUyNUE0JTI1OEYlMjVFNiUyNTlEJTI1QTUlMjVFOCUyNUEyJTI1QUQlMjZ0dXR1YXBwaW5mbyUzRCUyNnR1dHVmbGFnJTNEMQ%3D%3D.shtml"
})
> 
> 
> db.test.findAndModify({query:{f:0}, update:{$set:{f:1}}})
 
2015-09-22T10:21:36.883+0800 I NETWORK  DBClientCursor::init call() failed
2015-09-22T10:21:36.885+0800 E QUERY    Error: error doing query: failed
    at DBQuery._exec (src/mongo/shell/query.js:83:36)
    at DBQuery.hasNext (src/mongo/shell/query.js:240:10)
    at DBCollection.findOne (src/mongo/shell/collection.js:187:19)
    at DB.runCommand (src/mongo/shell/db.js:58:41)
    at DBCollection.findAndModify (src/mongo/shell/collection.js:631:24)
    at (shell):1:9 at src/mongo/shell/query.js:83
2015-09-22T10:21:36.887+0800 I NETWORK  trying reconnect to 127.0.0.1:29999 (127.0.0.1) failed
2015-09-22T10:21:36.888+0800 W NETWORK  Failed to connect to 127.0.0.1:29999, reason: errno:61 Connection refused
2015-09-22T10:21:36.888+0800 I NETWORK  reconnect 127.0.0.1:29999 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:29999 (127.0.0.1), connection attempt failed

Comment by Peak Ji [ 22/Sep/15 ]

Thanks Jason!

We've simplified the data model and process now:

Database: crawldb
Collection: high
Doc:

{"_id":"article title", "f": 0}

Index:

{"f": 1}

The "f" field is a flag indicating the article status: 0 = unread, 1 = read

We use findAndModify to get a "unread" doc (f=0), and mark it as "read" (f=1):

db.collection.findAndModify({query:

{"f": 0 }

, update: {"$set":

{"f": 1 }

} })

Tested again and got the same errors:

2015-09-22T05:16:58.194+0800 I COMMAND  [conn111] command crawldb.$cmd command: findAndModify { findAndModify: "high", query: { f: 0 }, update: { $set: { f: 1 } } } update: { $set: { f: 1 } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:1 keyUpdates:1 writeConflicts:1 numYields:0 reslen:164 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } 121ms
2015-09-22T05:16:58.195+0800 I COMMAND  [conn116] command crawldb.$cmd command: update { update: "low", updates: 43, ordered: true, writeConcern: { w: 1 } } keyUpdates:0 writeConflicts:0 numYields:0 reslen:322 locks:{ Global: { acquireCount: { r: 44, w: 44 } }, Database: { acquireCount: { w: 44 } }, Collection: { acquireCount: { w: 44 } } } 160ms
2015-09-22T05:16:58.195+0800 I COMMAND  [conn83] command crawldb.$cmd command: findAndModify { findAndModify: "high", query: { f: 0 }, update: { $set: { f: 1 } } } update: { $set: { f: 1 } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:1 keyUpdates:1 writeConflicts:1 numYields:0 reslen:170 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } 104ms
2015-09-22T05:16:58.198+0800 I COMMAND  [conn57] command crawldb.$cmd command: update { update: "high", updates: 107, ordered: true, writeConcern: { w: 1 } } keyUpdates:0 writeConflicts:0 numYields:0 reslen:2428 locks:{ Global: { acquireCount: { r: 108, w: 108 } }, Database: { acquireCount: { w: 108 } }, Collection: { acquireCount: { w: 108 } } } 149ms
2015-09-22T06:13:51.385+0800 I -        [conn57] Invariant failure res.existing src/mongo/db/commands/find_and_modify.cpp 492
2015-09-22T06:13:51.395+0800 I CONTROL  [conn57] 
 0xf75549 0xf12271 0xef6022 0x956049 0x956718 0x9cf044 0x9cffcd 0x9d0cdb 0xba223a 0xab3ba0 0x7fb86d 0xf268ab 0x3ac9607a51 0x3ac92e89ad
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B75549"},{"b":"400000","o":"B12271"},{"b":"400000","o":"AF6022"},{"b":"400000","o":"556049"},{"b":"400000","o":"556718"},{"b":"400000","o":"5CF044"},{"b":"400000","o":"5CFFCD"},{"b":"400000","o":"5D0CDB"},{"b":"400000","o":"7A223A"},{"b":"400000","o":"6B3BA0"},{"b":"400000","o":"3FB86D"},{"b":"400000","o":"B268AB"},{"b":"3AC9600000","o":"7A51"},{"b":"3AC9200000","o":"E89AD"}],"processInfo":{ "mongodbVersion" : "3.0.6", "gitVersion" : "1ef45a23a4c5e3480ac919b28afcba3c615488f2", "uname" : { "sysname" : "Linux", "release" : "2.6.32-573.3.1.el6.x86_64", "version" : "#1 SMP Thu Aug 13 22:55:16 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7F7AA372EDE22BA34234ADA10A8AF2E665681140" }, { "b" : "7FFEF4A84000", "elfType" : 3, "buildId" : "77385010026236DADA9BFEADC4717B7518E8BB29" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D467973C46E563CDCF64B5F12B2D6A50C7A25BA1" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AC11C021C127E84A2724F3A750461A2183B1069" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "78A423B7DD3C00F7A77027D8D764BB432B57F1E7" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "58C5A5FF5C82D7BE3113BE36DD87C7004E3C4DB1" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "B5AE05CEDC0CE917F50A3A468CFA2ACD8592E8F6" }, { "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "C03877A9EE01DDC572E2B0F55F64C757773CF8D6" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9350579A4970FA47F3144AD8F40B183B0954497D" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "285709B5B3FC083FE60B19F5DC2E5282B1708AB1" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "04202A4A8BE624D2193E812A25589E2DD02D5B5C" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "441FA45097A11508E50D55A3D1FF169BF2BE7C62" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "F62622218875795666E08B92D176A50791183EEC" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B8DEDADC140347276164C729418C7A37B7224135" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BDFC7A19C1F328EB4FCFBCE7A1E27606928610D" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F704FA7D21D05EF31E90FB4890FCA7F3D91DA138" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "E6798A06BEE17CF102BBA44FD512FF8B805CEAF1" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf75549]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf12271]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xef6022]
 mongod(_ZN5mongo16CmdFindAndModify7runImplEPNS_16OperationContextERKSsS4_RKNS_7BSONObjES7_S7_S7_bbbRNS_14BSONObjBuilderERSs+0x1BA9) [0x956049]
 mongod(_ZN5mongo16CmdFindAndModify3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x358) [0x956718]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9cf044]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC1D) [0x9cffcd]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x28B) [0x9d0cdb]
 mongod(_ZN5mongo8runQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERKNS_15NamespaceStringERNS_5CurOpES3_+0x77A) [0xba223a]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xB10) [0xab3ba0]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xDD) [0x7fb86d]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x34B) [0xf268ab]
 libpthread.so.0(+0x7A51) [0x3ac9607a51]
 libc.so.6(clone+0x6D) [0x3ac92e89ad]
-----  END BACKTRACE  -----
2015-09-22T06:13:51.395+0800 I -        [conn57] 
 
***aborting after invariant() failure

Comment by J Rassi [ 21/Sep/15 ]

Briefly discussed with david.storch. Our working theory is that findAndModify's call to update() is generating a WriteConflictException, and UpdateStage::work() is handling the WCE by dropping the current snapshot and returning an empty result set. Our recommended next action item on this ticket is to manually inspect the code in UpdateStage::work() in order to help build a reproducible case.

Note also that the relevant assertion does not exist in master, so it's possible that this issue affects the v3.0 branch only. Setting fixVersion to "3.1 Required" anyway; we can verify on the master branch once we can reproduce. I'll try to find an assignee at our planning meeting this afternoon.

Comment by J Rassi [ 21/Sep/15 ]

Hi,

Sorry to hear that you're encountering this issue.

I'd like to ask for additional information to help further diagnose the problem:

  • Could you copy/paste an excerpt of the findAndModify call from your application? I'm particularly interested in the query predicate that you are using to select the document.
  • Has this issue recurred since you first noticed it, or are you otherwise able to reliably reproduce this issue?

Thanks,
~ Jason Rassi

Generated at Thu Feb 08 03:54:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.