[SERVER-1989] Splitting big chunk caused assertion failed Created: 22/Oct/10  Updated: 12/Jul/16  Resolved: 23/Oct/10

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 1.6.3
Fix Version/s: 1.7.2

Type: Bug Priority: Major - P3
Reporter: Erez Zarum Assignee: Alberto Lerner
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

debian lenny with bpo kernel (2.6.32), ext4 filesystem, 32GB RAM


Attachments: File chunklist     File mongos.log    
Operating System: Linux
Participants:

 Description   

while trying to split a chunk manually, the server responded with an assertion error and data retrieval was not possible.

> db.runCommand({split:"storage.object_interaction", middle:{_id:877070867}})

{
"assertion" : "saving chunks failed. cmd: { applyOps: [ { op:
\"u\", b: true, ns: \"config.chunks\
", o: { _id: \"storage.object_interaction-_id_832352546\", lastmod:
Timestamp 179000|4, ns: \"storage.objec
t_interaction\", min:

{ _id: 832352546 }

, max:

{ _id: 832352546.0 }

,
shard: \"shard0001\" }, o2:

{ _id: \"s torage.object_interaction-_id_832352546\" }

}, { op: \"u\", b: true,
ns: \"config.chunks\", o: { _id: \"sto
rage.object_interaction-_id_832352546.0\", lastmod: Timestamp 179000|
5, ns: \"storage.object_interaction\",
min:

{ _id: 832352546.0 }

, max:

{ _id: 832352547.0 }

, shard:
\"shard0001\" }, o2:

{ _id: \"storage.object_ interaction-_id_832352546.0\" }

}, { op: \"u\", b: true, ns:
\"config.chunks\", o: { id: \"storage.object
interaction-_id_877070867\", lastmod: Timestamp 179000|6, ns:
\"storage.object_interaction\", min:

{ _id: 8 77070867 }

, max:

{ _id: 877070867.0 }

, shard: \"shard0001\" }, o2:

{ _id: \"storage.object_interaction-_id_ 877070867\" }

}, { op: \"u\", b: true, ns: \"config.chunks\", o:
{ _id: \"storage.object_interaction-_id_877070867.0\", lastmod:
Timestamp 179000|7, ns: \"storage.object_interaction\", min:

{ _id: 877070867.0 }

, max:

{ _id: 877217527 }

, shard: \"shard0001\" }, o2:

{ _id: \"storage.object_interaction-_id_877070867.0\" }

} ],
preCondition: [ { ns: \"config.chunks\", q: { query:

{ ns: \"storage.object_interaction\" }

, orderby:

{ lastmod: -1 }

}, res:

{ lastmod: Timestamp 179000|3 }

} ] } result: { got: { _id:
\"storage.object_interaction-_id_832352546\", lastmod: Timestamp
179000|4, ns: \"storage.object_interaction\", min:

{ _id: 832352546 }

,
max:

{ _id: 832352546.0 }

, shard: \"shard0001\" }, whatFailed: { ns:
\"config.chunks\", q: { query:

{ ns: \"storage.object_interaction\" }

,
orderby:

{ lastmod: -1 }

}, res:

{ lastmod: Timestamp 179000|3 }

},
errmsg: \"pre-condition failed\", ok: 0.0 }",
"assertionCode" : 13327,
"errmsg" : "db assertion failure",
"ok" : 0
}

> db.runCommand({split:"storage.object_interaction", find:{_id:854711706}})
{
"assertion" : "Couldn't load a valid config for
storage.object_interaction after 3 tries. Giving up",
"assertionCode" : 13282,
"errmsg" : "db assertion failure",
"ok" : 0
}

> db.object_interaction.findOne({_id:854711706})
Fri Oct 22 00:04:24 uncaught exception: error

{ "$err" : "assertion s/chunk.cpp:768", "code" : 0 }

I'm attaching the cunklist (db.printShardingStatus as well as a mongos log file that contain the events)

What can i do in order to split the chunk? right now the index is not even on 2 of the shard servers.



 Comments   
Comment by Eliot Horowitz (Inactive) [ 26/Oct/10 ]

Backporting this is hard - and only human error can cause - so i think its ok not to.

Comment by Alberto Lerner [ 23/Oct/10 ]

You'd need to adjust the chunk starting at 832352546 by hand.

To do that, log in a mongos and switch to the 'config' database. Change the chunks collection

db.chunks.update( { "ns": "storage.object_ineraction" , "min" :

{ "_id" : 832352546}

} , { $set : { "max" :

{ "_id" : 877070867 }

} } )

This patch will be in 1.6.4. In the meantime, as Eliot suggested, please use the find option.

Comment by Alberto Lerner [ 23/Oct/10 ]

Author:

{'login': 'alerner', 'name': 'Alberto Lerner', 'email': 'alerner@10gen.com'}

Message: fix error when key provided is not a valid split keyhttp://github.com/mongodb/mongo/commit/18938d62ec3e04470de9d555056534e92ef4069d

Comment by Eliot Horowitz (Inactive) [ 23/Oct/10 ]

can you upload th config server log.

I thikn the problem was using middle.

just use split with find, its much safer.

Generated at Thu Feb 08 02:58:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.