[SERVER-3447] journal exception during large operations within db.eval (multi update) Created: 19/Jul/11  Updated: 12/Jul/16  Resolved: 01/Aug/11

Status: Closed
Project: Core Server
Component/s: Stability, Storage
Affects Version/s: 1.8.2
Fix Version/s: 1.9.2

Type: Bug Priority: Major - P3
Reporter: chrisferry@gmail.com Assignee: Mathias Stearn
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 5.6


Attachments: Text File mongodb.log    
Issue Links:
Depends
Duplicate
is duplicated by SERVER-3439 mongod complains about too much data ... Closed
is duplicated by SERVER-3278 Master stopped allowing connections, ... Closed
Operating System: ALL
Participants:

 Description   

Query is:
db.ActivityEvent.update(

{type:"NEWS"}

,
{$set: {className:"com.activitystream.common.dto.News"}}, false, true);

Collection has 700k entries and the raw bson is about 685MB

If we turn journaling off the query completes and mongo doesn't die.

Attached mongodb.log. Pertinent lines:
Mon Jul 18 21:37:43 [conn88] notifications_uat Assertion failure a <= 256*1024*1024 util/alignedbuilder.cpp 51

Mon Jul 18 21:37:43 [conn88] dbexception in groupCommit causing immediate shutdown: 0 assertion util/alignedbuilder.cpp:52
Mon Jul 18 21:37:43 Got signal: 6 (Aborted).



 Comments   
Comment by auto [ 04/Aug/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: Revert "SERVER-3447 we were overcounting bytes since last commit in some cases"

This reverts commit 3b4b0026cd2e018346bf068109ba0a9368a07eeb.
Branch: master
https://github.com/mongodb/mongo/commit/7dd8b4c1dc20f1a5bd84efae63fdca1d8cfd1bc6

Comment by auto [ 04/Aug/11 ]

Author:

{u'login': u'erh', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: backport of SERVER-3447
Branch: v1.8
https://github.com/mongodb/mongo/commit/fba78d46345423318b9eb08bcc51c733e6dc0c5e

Comment by auto [ 03/Aug/11 ]

Author:

{u'login': u'dwight', u'name': u'dwight', u'email': u'dwight@10gen.com'}

Message: SERVER-3447 we were overcounting bytes since last commit in some cases
Branch: master
https://github.com/mongodb/mongo/commit/3b4b0026cd2e018346bf068109ba0a9368a07eeb

Comment by auto [ 02/Aug/11 ]

Author:

{u'login': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: Add test for journal exception during large operations within db.eval SERVER-3447
Branch: master
https://github.com/mongodb/mongo/commit/681a4c7225eed0cf813f7cb5d7c1079dde9175b5

Comment by auto [ 01/Aug/11 ]

Author:

{u'login': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: always commitIfNeeded in multi-update. Fixes db.eval() SERVER-3447
Branch: master
https://github.com/mongodb/mongo/commit/ad9b50e4e45ec704305173f7081100a0090de672

Comment by Mathias Stearn [ 01/Aug/11 ]

I think if you add $atomic:1 (http://www.mongodb.org/display/DOCS/Atomic+Operations#AtomicOperations-ApplyingtoMultipleObjectsAtOnce) to the query it will work with journaling. The issue is that we assume journal commits will happen automatically if you are not using $atomic, but you are using db.eval() to make it atomic (really isolated in ACID-speak) without using $atomic. I can fix the code to remove this assumption, but in the meantime adding $atomic should work around this while allowing you to use journaling.

Implementation note: https://github.com/mongodb/mongo/blob/master/db/ops/update.cpp#L1295-1296

Comment by Dwight Merriman [ 31/Jul/11 ]

we will fix bug it will be a while. in the meantime i would recommend running without journaling, or making java code that does the first case as a regular update from that code, and does what you have already above for the rest. perhaps it is only the first multi-update that is problematic (maybe 2nd one is too though not sure would have to try).

Comment by OLEKSII IEPISHKIN [ 31/Jul/11 ]

I can not. The idea of that tool - developers add mongo queries into xml file like this

<mongoChangeLog>
<changeSet changeId="SMS-1221" author="oleksii">
<script>
db.ActivityEvent.update(

{type:"NEWS"}

, {$set: {className:"com.activitystream.common.dto.News"}}, false, true);
</script>
</changeSet>

<changeSet changeId="SMS-1221 rename newsGuid" author="oleksii">
<script>
var events = db.ActivityEvent.find({type:"NEWS_VOTED_UP", "properties.newsGuid": {$exists: true}});
events.forEach(function(item) {
db.ActivityEvent.update({_id: item._id},
{$set:

{"properties.guid": item.properties.newsGuid}

, $unset: {"properties.newsGuid": 1}},
false, false);
});
</script>
</changeSet>

<changeSet changeId="SMS-1258" author="oleksii">
<script>
db.ActivityEvent.ensureIndex(

{"type":1, "properties.guid":1, "active": 1}

);
db.ActivityEvent.ensureIndex(

{"type":1, "active": 1, "actor": 1, "timestamp": -1, "rating.upVoteCount": -1}

);
</script>
</changeSet>
</mongoChangeLog>

and every time application starts

1. every query is verified if it has been executed or not
2. if it's a new query it's executed using

import com.mongodb.DB;
public class MongeezDao {
private DB db;
public void runScript(String code) {
db.eval(code);
}
}

I have an experimental impl of that dao that uses mongo shell instead of mongo java driver. DB doesn't crash in that mode but that impl has some problems so it's not ready to be used in the production code.

I'm going to test with mongodb-osx-x86_64-1.9.1.tgz and will report the result.

Comment by Dwight Merriman [ 31/Jul/11 ]

that helps i think i know what is happening then.
can you use update from your tool instead of eval()?

Comment by OLEKSII IEPISHKIN [ 31/Jul/11 ]

Dwight,

More details that could be helpful to fix the issue.
We run that update query by executing db.eval( ) in our db change management tool http://mongeez.org

db.eval(function() {
db.ActivityEvent.update(

{type:"NEWS"}

,
{$set: {className:"com.activitystream.common.dto.News"}}, false, true);
});

When I run that update query using mongo shell DB doesn't crash.

Comment by Dwight Merriman [ 31/Jul/11 ]

there is a change that might fix this in 1.9

more testing needed to be sure.

Generated at Thu Feb 08 03:03:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.