[SERVER-40881] Jumbo chunk not marked as jumbo Created: 29/Apr/19  Updated: 22/Aug/19  Resolved: 11/Jul/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.2
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Björn Kautler Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod-configServer.log     Text File mongod-shard01.log     Text File mongod-shard02.log     Text File mongod-shard03.log     Text File mongos.log    
Issue Links:
Related
is related to DOCS-12978 Improve nuance around Jumbo chunks Closed
Participants:

 Description   

I'm new to MongoDB and took some courses at the MongoDB University.
When it came to Jumbo chunks I played around with it, but it behaves strange to me.
I intentionally created a jumbo chunk and it is treated as it should be treated
(not split even if it could by inserting another shard key value, not splittable manually except after adding another shard key value).
But it is not marked as jumbo, neither in the respecitve table, nor in the sh.status() output.
 
At https://discourse.university.mongodb.com/t/divisible-jumbo-chunks I already wrote detailed information and log file excerpts.
Can you tell me why this chunk is not marked as jumbo?
Is this a bug?
 
PS: version is 3.4.2 because that is the version in the vagrant box of the university course



 Comments   
Comment by Eric Sedor [ 11/Jul/19 ]

Hi,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Thank you!

Comment by Eric Sedor [ 24/Jun/19 ]

Hi,

I wanted to see if you got my last message and had a chance to check the chunks collection in the config database for the chunks the logs reported as becoming jumbo.

Thanks,
Eric

Comment by Eric Sedor [ 30/May/19 ]

Hi vampire; one way of looking at what kevin.pulo is saying is that it is the Balancer which decides when a chunk is jumbo. And it looks like from the logs that it does so.

shard01 reports:

2019-04-26T21:59:03.167+0000 W SHARDING [conn97] Chunk move failed :: caused by :: ChunkTooBig: Cannot move chunk: the maximum number of documents for a chunk is 3, the maximum chunk size is 1048576, average document size is 290965. Found 20 documents in chunk  ns: test.test { test: "foo" } -> { test: "fooo" }
2019-04-26T21:59:03.167+0000 I SHARDING [conn97] about to log metadata event into changelog: { _id: "m312-2019-04-26T21:59:03.167+0000-5cc37f27b15e94cf3537f3b2", server: "m312", clientAddr: "127.0.0.1:56463", time: new Date(1556315943167), what: "moveChunk.from", ns: "test.test", details: { min: { test: "foo" }, max: { test: "fooo" }, step 1 of 7: 0, step 2 of 7: 1, to: "shard03", from: "shard01", note: "aborted" } }
2019-04-26T21:59:03.183+0000 I SHARDING [conn97] request split points lookup for chunk test.test { : "foo" } -->> { : "fooo" }
2019-04-26T21:59:03.183+0000 W SHARDING [conn97] possible low cardinality key detected in test.test - key is { test: "foo" }

and the balancer reports:

2019-04-26T21:59:03.183+0000 I SHARDING [Balancer] Marking chunk ns: test.test, shard: shard01, lastmod: 9|1||5cc1f21c7d2ebf0a897998eb, min: { test: "foo" }, max: { test: "fooo" } as jumbo.

If this chunk is not seen as jumbo when you inspect the chunk document in the config server, then we would consider this problem as a duplicate of SERVER-26531. Can you confirm this?

Gratefully,
Eric

Comment by Björn Kautler [ 25/May/19 ]

Sorry for the late answer, I somehow didn't see there was an answer and missed checking manually.

eric.sedor I attached the requested logfile unmodified.
There were no replica sets anyway, so these are actually all log files.

kevin.pulo
Chunks are only marked as jumbo by the balancer in certain limited circumstances:

  1. The balancer attempts to migrate the chunk to a different shard.
  2. The migration fails due to the chunk having too many documents. This causes the balancer to attempt to split the chunk.
  3. The chunk split fails. (Prior to SERVER-21931, if it fails for any reason. After SERVER-21931, if it fails due to no split points being identified.)

This means that chunks will never be marked as jumbo during any of:

  • manual chunk migrations
  • manual chunk splits
  • chunk splits arising from inserts

I am aware that manual migrations or splits do not cause marking as jumbo.
But for your statements that "balancer tries to split if too many documents" and "chunk splits arising from inserts will not mark as jumbo",
if these statements are true, then all training and documentation material I've see so far is either wrong or misleading.

Here some examples:

https://docs.mongodb.com/manual/core/sharding-data-partitioning/#jumbo-chunks says "In some cases, chunks can grow beyond the specified chunk size but cannot undergo a split. The most common scenario is when a chunk represents a single shard key value. Since the chunk cannot split, it continues to grow beyond the chunk size, becoming a jumbo chunk."

The video of M312 MongoDB University Course in Chapter 4 labeled "Lecture: Sharding Issues" also suggests that the chunk gets marked jumbo if shard size gets exceeded.

https://docs.mongodb.com/manual/core/sharding-data-partitioning/index.html says "Automatic splitting only occurs during inserts or updates." and "Inserts and updates may trigger splits." and later on that the balancer just just moves the chunks around that were split during insert or update to regain an even distribution amongst the shards.
 

kevin.pulo
Have you tried arranged things such that the balancer will attempt to migrate the chunk in question? And have you ensured that prior to this, the chunk contains a large number (more than 250,000, or possibly even more than this) of small documents with identical shard key value?

No, not yet, as all material I've seen so far suggested to me that it should work like I tried it. :-/

Comment by Kevin Pulo [ 01/May/19 ]

Chunks are only marked as jumbo by the balancer in certain limited circumstances:

  1. The balancer attempts to migrate the chunk to a different shard.
  2. The migration fails due to the chunk having too many documents. This causes the balancer to attempt to split the chunk.
  3. The chunk split fails. (Prior to SERVER-21931, if it fails for any reason. After SERVER-21931, if it fails due to no split points being identified.)

This means that chunks will never be marked as jumbo during any of:

  • manual chunk migrations
  • manual chunk splits
  • chunk splits arising from inserts

Have you tried arranged things such that the balancer will attempt to migrate the chunk in question? And have you ensured that prior to this, the chunk contains a large number (more than 250,000, or possibly even more than this) of small documents with identical shard key value?

Comment by Eric Sedor [ 29/Apr/19 ]

One possibility is that you are encountering SERVER-26531. To help us confirm this, can you please attach the complete logs (without filtering via grep) of the primary member of the config server replica set, and the primary member of each shard, to this ticket?

Generated at Thu Feb 08 04:56:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.