Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20659

balancer ignores collection storage engine options on moveChunk

    • Sharding
    • ALL
    • Hide

      Apply the following patch to the sharding_system_namespaces.js test:

      Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      diff --git a/jstests/sharding/sharding_system_namespaces.js b/jstests/sharding/sharding_system_namespaces.js
      index b27c363..a0aac3a 100644
      --- a/jstests/sharding/sharding_system_namespaces.js
      +++ b/jstests/sharding/sharding_system_namespaces.js
      @@ -46,6 +46,7 @@ if (Array.contains(storageEngines, "wiredTiger")) {
           assert.commandWorked(db.adminCommand({ enableSharding: 'test' }));
           st.ensurePrimaryShard('test', 'shard0001');
           assert.commandWorked(db.adminCommand({ shardCollection: coll + '', key: { x: 1 }}));
      +    assert.commandWorked(coll.createIndex({x: 1}));
       
           coll.insert({x: 0});
           coll.insert({x: 10});
      
      [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.373-0500 s20015| 2015-12-12T18:50:16.373-0500 D COMMAND  [conn1] RunOnAllShardsCommand db: test cmd:{ createIndexes: "sharding_system_namespaces", indexes: [ { ns: "test.sharding_system_namespaces", key: { x: 1.0 }, name: "x_1" } ] }
      ...
      [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 [ { "name" : "sharding_system_namespaces", "options" : { } } ]
      [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 2015-12-12T18:50:16.587-0500 E QUERY    [thread1] TypeError: info.options.storageEngine is undefined :
      [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 checkCollectionOptions@jstests/sharding/sharding_system_namespaces.js:34:7
      [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 @jstests/sharding/sharding_system_namespaces.js:68:5
      [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500
      [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500 failed to load: jstests/sharding/sharding_system_namespaces.js
      
      Show
      Apply the following patch to the sharding_system_namespaces.js test: Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml diff --git a/jstests/sharding/sharding_system_namespaces.js b/jstests/sharding/sharding_system_namespaces.js index b27c363..a0aac3a 100644 --- a/jstests/sharding/sharding_system_namespaces.js +++ b/jstests/sharding/sharding_system_namespaces.js @@ -46,6 +46,7 @@ if (Array.contains(storageEngines, "wiredTiger" )) { assert .commandWorked(db.adminCommand({ enableSharding: 'test' })); st.ensurePrimaryShard( 'test' , 'shard0001' ); assert .commandWorked(db.adminCommand({ shardCollection: coll + '', key: { x: 1 }})); + assert .commandWorked(coll.createIndex({x: 1})); coll.insert({x: 0}); coll.insert({x: 10}); [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.373-0500 s20015| 2015-12-12T18:50:16.373-0500 D COMMAND [conn1] RunOnAllShardsCommand db: test cmd:{ createIndexes: "sharding_system_namespaces", indexes: [ { ns: "test.sharding_system_namespaces", key: { x: 1.0 }, name: "x_1" } ] } ... [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 [ { "name" : "sharding_system_namespaces", "options" : { } } ] [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 2015-12-12T18:50:16.587-0500 E QUERY [thread1] TypeError: info.options.storageEngine is undefined : [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 checkCollectionOptions@jstests/sharding/sharding_system_namespaces.js:34:7 [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.590-0500 @jstests/sharding/sharding_system_namespaces.js:68:5 [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500 [js_test:sharding_system_namespaces] 2015-12-12T18:50:16.591-0500 failed to load: jstests/sharding/sharding_system_namespaces.js
    • QuInt A (10/12/15), QuInt B (11/02/15), QuInt C (11/23/15)

      The mongos forwards the createIndexes command to all shards, including ones without any chunks for the sharded collection. This causes the .chunks collection to be implicitly created without any storage engine options on the other shard.


      Original description

      Hello,

      In a sharded cluser with default "blockCompressor" set to "zlib", I created a collection with "block_compressor=snappy".

      After sharding this "snappy" collection, the balancer creates the collection on other replica-sets with "zlib" (the config default), and not "snappy" (the explicit block_compressor for this particular collection).

      => It would be nice if the not default block_compressor is preserved across the shards of the same collection.

      => Can I safely manualy create the collection directy on the secondary shards (connecting directy to each replica-set primary) before sharding the collection ?

      Best regards,
      Clement

      Original steps to reproduce

      My use case is on a GridFS collection. I dont think it's relevant. But just in case, I let the ".chunks".

      The issue probably also exists with default compression left to "snappy", and collection explicitely created with "zlib". But again, here is my use case :

      1. set the default block compressor to "zlib" :
        mongod config file
        ...
        storage:
          dbPath: /var/lib/mongodb
          directoryPerDB: true
          engine: wiredTiger
          wiredTiger:
            collectionConfig:
              blockCompressor: zlib
        ...
        
      2. (remove dbPath content and restart mongods)
      3. Manualy create a collection with "snappy" :
        mongos> use myDB
        mongos> db.createCollection( "myGridfsCollection.chunks", { storageEngine: { wiredTiger: { configString: 'block_compressor=snappy' }}})
        
      4. Check the block_compressor :
        mongos> use myDB
        mongos> db.myGridfsCollection.chunks.stats().wiredTiger.creationString
        

        => must be "block_compressor=snappy"

      5. Start inserting documents (in my case GridFS files) in this collection.
      6. Shard the collection :
        mongos> use admin
        mongos> db.runCommand( { shardCollection : "myDB.myGridfsCollection.chunks" , key : { files_id : 1 , n : 1 } } )
        
      7. Check the block_compressor again :
        mongos> use myDB
        mongos> db.myGridfsCollection.chunks.stats().wiredTiger.creationString
        

        => ok nothing - the collection is sharded.
        In my case, I have 2 replica-sets "rs0" and "rs1" :
        For rs0 :

        mongos> db.myGridfsCollection.chunks.stats().shards.rs0.wiredTiger.creationString
        

        => block_compressor=snappy
        => the primary shard of the collection, here "rs0", have the correct block compressor.
        For rs1 :
        If their is nothing yet. Continue data inserts to trigger the balancer.

        mongos> db.myGridfsCollection.chunks.stats().shards.rs1.wiredTiger.creationString
        

        => block_compressor=zlib
        => the other shard of the collection is not in snappy.

            Assignee:
            backlog-server-sharding [DO NOT USE] Backlog - Sharding Team
            Reporter:
            clement@compilatio Clement R
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: