[SERVER-20659] balancer ignores collection storage engine options on moveChunk Created: 28/Sep/15  Updated: 06/Dec/22  Resolved: 02/Dec/21

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.0.6
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Clement R Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Duplicate Votes: 0
Labels: pm-1051-legacy-tickets
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS : debian 7.8 64
MongoDB : mongodb-org v3.0.6 via "http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.0 main"


Issue Links:
Duplicate
duplicates SERVER-30344 prevent shards from implicitly creati... Closed
Related
is related to SERVER-30344 prevent shards from implicitly creati... Closed
Assigned Teams:
Sharding
Operating System: ALL
Steps To Reproduce:

Apply the following patch to the sharding_system_namespaces.js test:

diff --git a/jstests/sharding/sharding_system_namespaces.js b/jstests/sharding/sharding_system_namespaces.js
index b27c363..a0aac3a 100644
--- a/jstests/sharding/sharding_system_namespaces.js
+++ b/jstests/sharding/sharding_system_namespaces.js
@@ -46,6 +46,7 @@ if (Array.contains(storageEngines, "wiredTiger")) {
     assert.commandWorked(db.adminCommand({ enableSharding: 'test' }));
     st.ensurePrimaryShard('test', 'shard0001');
     assert.commandWorked(db.adminCommand({ shardCollection: coll + '', key: { x: 1 }}));
+    assert.commandWorked(coll.createIndex({x: 1}));
 
     coll.insert({x: 0});
     coll.insert({x: 10});

[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.373-0500 s20015| 2015-12-12T18:50:16.373-0500 D COMMAND  [conn1] RunOnAllShardsCommand db: test cmd:{ createIndexes: "sharding_system_namespaces", indexes: [ { ns: "test.sharding_system_namespaces", key: { x: 1.0 }, name: "x_1" } ] }
...
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 [ { "name" : "sharding_system_namespaces", "options" : { } } ]
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 2015-12-12T18:50:16.587-0500 E QUERY    [thread1] TypeError: info.options.storageEngine is undefined :
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 checkCollectionOptions@jstests/sharding/sharding_system_namespaces.js:34:7
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 @jstests/sharding/sharding_system_namespaces.js:68:5
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500 failed to load: jstests/sharding/sharding_system_namespaces.js

Sprint: QuInt A (10/12/15), QuInt B (11/02/15), QuInt C (11/23/15)
Participants:

 Description   

The mongos forwards the createIndexes command to all shards, including ones without any chunks for the sharded collection. This causes the .chunks collection to be implicitly created without any storage engine options on the other shard.


Original description

Hello,

In a sharded cluser with default "blockCompressor" set to "zlib", I created a collection with "block_compressor=snappy".

After sharding this "snappy" collection, the balancer creates the collection on other replica-sets with "zlib" (the config default), and not "snappy" (the explicit block_compressor for this particular collection).

=> It would be nice if the not default block_compressor is preserved across the shards of the same collection.

=> Can I safely manualy create the collection directy on the secondary shards (connecting directy to each replica-set primary) before sharding the collection ?

Best regards,
Clement

Original steps to reproduce

My use case is on a GridFS collection. I dont think it's relevant. But just in case, I let the ".chunks".

The issue probably also exists with default compression left to "snappy", and collection explicitely created with "zlib". But again, here is my use case :

  1. set the default block compressor to "zlib" :

    mongod config file

    ...
    storage:
      dbPath: /var/lib/mongodb
      directoryPerDB: true
      engine: wiredTiger
      wiredTiger:
        collectionConfig:
          blockCompressor: zlib
    ...
    

  2. (remove dbPath content and restart mongods)
  3. Manualy create a collection with "snappy" :

    mongos> use myDB
    mongos> db.createCollection( "myGridfsCollection.chunks", { storageEngine: { wiredTiger: { configString: 'block_compressor=snappy' }}})
    

  4. Check the block_compressor :

    mongos> use myDB
    mongos> db.myGridfsCollection.chunks.stats().wiredTiger.creationString
    

    => must be "block_compressor=snappy"

  5. Start inserting documents (in my case GridFS files) in this collection.
  6. Shard the collection :

    mongos> use admin
    mongos> db.runCommand( { shardCollection : "myDB.myGridfsCollection.chunks" , key : { files_id : 1 , n : 1 } } )
    

  7. Check the block_compressor again :

    mongos> use myDB
    mongos> db.myGridfsCollection.chunks.stats().wiredTiger.creationString
    

    => ok nothing - the collection is sharded.
    In my case, I have 2 replica-sets "rs0" and "rs1" :
    For rs0 :

    mongos> db.myGridfsCollection.chunks.stats().shards.rs0.wiredTiger.creationString
    

    => block_compressor=snappy
    => the primary shard of the collection, here "rs0", have the correct block compressor.
    For rs1 :
    If their is nothing yet. Continue data inserts to trigger the balancer.

    mongos> db.myGridfsCollection.chunks.stats().shards.rs1.wiredTiger.creationString
    

    => block_compressor=zlib
    => the other shard of the collection is not in snappy.



 Comments   
Comment by Jack Mulrow [ 02/Dec/21 ]

This issue was fixed in SERVER-30344, so closing this ticket as a duplicate of that one.

Comment by Ramon Fernandez Marina [ 01/Apr/16 ]

clement@compilatio, unfortunately this ticket is not part of the current development cycle at the moment, but will be considered in the next round of planning. A fix for 3.2/3.4 is not scheduled right now, but if that changes will update this ticket accordingly.

Regards,
Ramón.

Comment by Clement R [ 01/Apr/16 ]

Hello,

Any update on this issue ?
A fix, even in 3.2 only, would be appreciated.

Thanks,
Clement

Comment by Max Hirschhorn [ 13/Dec/15 ]

The reason that the storage engine options are not propagated during the moveChunk operation is because the .chunks collection does in fact already exist on the other shard. The GridFS specification requires that the driver create the index {files_id: 1, n: 1} on the .chunks collection before the first write operation if the index doesn't already exist. mongos forwards the createIndexes command to all shards in the cluster, even if they don't store any chunks of the sharded collection. The createIndexes command is what causes the .chunks collection to be implicitly created without any storage engine options on the other shard.

This issue isn't really GridFS-specific and can easily be observed by applying the following patch to the sharding_system_namespaces.js test:

diff --git a/jstests/sharding/sharding_system_namespaces.js b/jstests/sharding/sharding_system_namespaces.js
index b27c363..a0aac3a 100644
--- a/jstests/sharding/sharding_system_namespaces.js
+++ b/jstests/sharding/sharding_system_namespaces.js
@@ -46,6 +46,7 @@ if (Array.contains(storageEngines, "wiredTiger")) {
     assert.commandWorked(db.adminCommand({ enableSharding: 'test' }));
     st.ensurePrimaryShard('test', 'shard0001');
     assert.commandWorked(db.adminCommand({ shardCollection: coll + '', key: { x: 1 }}));
+    assert.commandWorked(coll.createIndex({x: 1}));
 
     coll.insert({x: 0});
     coll.insert({x: 10});

[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.373-0500 s20015| 2015-12-12T18:50:16.373-0500 D COMMAND  [conn1] RunOnAllShardsCommand db: test cmd:{ createIndexes: "sharding_system_namespaces", indexes: [ { ns: "test.sharding_system_namespaces", key: { x: 1.0 }, name: "x_1" } ] }
...
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 [ { "name" : "sharding_system_namespaces", "options" : { } } ]
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 2015-12-12T18:50:16.587-0500 E QUERY    [thread1] TypeError: info.options.storageEngine is undefined :
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 checkCollectionOptions@jstests/sharding/sharding_system_namespaces.js:34:7
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 @jstests/sharding/sharding_system_namespaces.js:68:5
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500
[js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500 failed to load: jstests/sharding/sharding_system_namespaces.js

Comment by Clement R [ 24/Nov/15 ]

No. I dont think so.
But just to but sure, I tested again with a new database "testcompressor2" and a new collection "myGridfsCollection2", created for the occasion and never used before.
And the bug is still there :

(set the default block compressor to "zlib")
--
# create collection snappy :
mongos> use testcompressor2
mongos> db.createCollection( "myGridfsCollection2.chunks", { storageEngine: { wiredTiger: { configString: 'block_compressor=snappy' }}})
mongos> db.myGridfsCollection2.chunks.stats().wiredTiger.creationString
allocation_size=4KB,app_metadata=(formatVersion=1),block_allocation=best,block_compressor=snappy
mongos> show collections
myGridfsCollection2.chunks
mongos>
--
# insert files
# insert files
# insert files
# insert files
--
# check
rs0:PRIMARY> use testcompressor2
switched to db testcompressor2
rs0:PRIMARY> show collections
rs0:PRIMARY>
-
rs1:PRIMARY> use testcompressor2
switched to db testcompressor2
rs1:PRIMARY> show collections
myGridfsCollection2.chunks
myGridfsCollection2.files
rs1:PRIMARY> db.myGridfsCollection2.chunks.stats().wiredTiger.creationString
allocation_size=4KB,app_metadata=(formatVersion=1),block_allocation=best,block_compressor=snappy
--
# sharding :
mongos> use admin
mongos> db.runCommand( { shardCollection : "testcompressor2.myGridfsCollection2.chunks" , key : { files_id : 1 , n : 1 } } )
	{ "ok" : 0, "errmsg" : "sharding not enabled for db" }
mongos> use testcompressor2
mongos> sh.enableSharding("testcompressor2")
mongos> use admin
mongos> db.runCommand( { shardCollection : "testcompressor2.myGridfsCollection2.chunks" , key : { files_id : 1 , n : 1 } } )
--
# check :
rs0:PRIMARY> show collections
rs0:PRIMARY>
-
rs1:PRIMARY> show collections
myGridfsCollection2.chunks
myGridfsCollection2.files
rs1:PRIMARY>
--
# insert files
# insert files
# insert files
# insert files
--
# check
rs0:PRIMARY> show collections
myGridfsCollection2.chunks
rs0:PRIMARY> db.myGridfsCollection2.chunks.stats().wiredTiger.creationString
allocation_size=4KB,app_metadata=(formatVersion=1),block_allocation=best,block_compressor=zlib     <==== /!\ **bug** /!\
-
rs1:PRIMARY> show collections
myGridfsCollection2.chunks
myGridfsCollection2.files
rs1:PRIMARY> db.myGridfsCollection2.chunks.stats().wiredTiger.creationString
allocation_size=4KB,app_metadata=(formatVersion=1),block_allocation=best,block_compressor=snappy
# check of a 'failed to create collection' in the logs (all mongos/configsrv/mongod)
-> nothing.

  • But with your comment in mind, I tried to reproduce the bug on a basic collection : "testcompressor.coll1" => NO bug O.o
  • Maybe with a dot in it ? "testcompressor.coll1.abcd" => NO bug
  • Maybe if named ".chunks" ? "testcompressor.coll1.chunks" => NO bug
  • Maybe if named ".chunks" with a unique index on { files_id: 1, n: 1 }

    => NO bug

  • Maybe the GridFS PHP pecl driver does something wrong ? -> test with the GridFS Python driver => still the BUG with the GridFS Python driver.

=> So this bug seems "GridFS" related...

Regards,
Clement

Comment by Max Hirschhorn [ 21/Nov/15 ]

Hi Clement,

Sorry about the delayed response. Dan answered your question regarding a possible workaround by creating an empty collection on each of the shards with the block_compressor option you wanted.

SERVER-16498, which was caught during the 2.8 RC period, sounds like the issue you are describing. A fix for it was committed in 2.8.0-rc4, along with the test sharding_system_namespaces.js. The test creates a collection with block_compressor=zlib on shards with system-wide defaults of block_compressor=snappy and verifies that the options are propagated to the other shard during a moveChunk operation (which is what the balancer would trigger). This sounds similar enough to your situation that I'm having trouble understanding why you found that the recipient shard ended up with the system-wide default block compressor, rather than the one you specified during the initial collection creation.

  1. Did the myDB.myGridfsCollection.chunks collection already exist on the shard that the balancer migrated a chunk to?
  2. Was there a warning message "failed to create collection ..." present in the logs of the recipient shard around the time the moveChunk was being performed?

Thanks,
Max

Comment by Daniel Pasette (Inactive) [ 28/Sep/15 ]

Thanks for the detailed report Clement. The intention is that the expressly defined storageEngine options for a collection will override the server defaults and that moveChunk will honor those options. As a workaround, it should be safe to create an empty collection with the blockCompressor options you want on the empty shard. We'll investigate what needs to happen in the server.

Generated at Thu Feb 08 03:54:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.