-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Sharding
-
Catalog and Routing
-
ALL
-
2
-
🟥 DDL
-
None
-
None
-
None
-
None
-
None
-
None
Summary
On 7.0 and 8.0, a 2d index created with a fractional bits option on a populated collection keeps that value verbatim (as a double). When a chunk migration later makes another shard create the collection, the recipient stores bits rounded to an int. The index spec then differs between shards, and checkMetadataConsistency reports InconsistentIndex.
9.0 does not reproduce this: there, createIndex stores bits as an int from the start, so every shard agrees.
Versions
- Reproduces: 7.0.46, 8.0
- Does not reproduce: 9.0.3
Steps to reproduce
Run the script below against a mongos of a cluster with at least 2 shards:
mongosh --quiet mongodb://localhost:27017 repro_2d_bits_migration.js
It:
- creates a populated, unsharded collection on one shard;
- creates {{
{p: "2d"}
}} with bits: 10.01262704984186;
- shards the collection on {_id: "hashed"}, splits at 0, and moves the upper chunk to a shard that lacks the collection;
- prints each shard’s copy of the index (via $indexStats) and runs checkMetadataConsistency with checkIndexes: true.
Actual result (7.0.46)
== After createIndex, before sharding
sh01: {"v":{"$numberInt":"2"},"key":{"p":"2d"},"name":"p_2d","bits":{"$numberDouble":"10.01262704984186"}}
sh02: (no such index)
== After moveChunk
sh01: {"v":{"$numberInt":"2"},"key":{"p":"2d"},"name":"p_2d","bits":{"$numberDouble":"10.01262704984186"}}
sh02: {"v":{"$numberInt":"2"},"key":{"p":"2d"},"name":"p_2d","bits":{"$numberInt":"10"}}
INCONSISTENT: “bits” differs across shards: {"sh01":{"$numberDouble":"10.01262704984186"},"sh02":{"$numberInt":"10"}}
checkMetadataConsistency found 1 inconsistency(ies):
{
type: 'InconsistentIndex',
description: 'Found an index of a sharded collection that is inconsistent between different shards',
details: {
namespace: 'repro_2d_bits.coll',
info: {
missingFromShards: [],
inconsistentProperties: [ { k: 'bits', v: 10 }, { k: 'bits', v: 10.01262704984186 } ],
indexName: 'p_2d'
}
}
}
Expected result
Every shard has the same spec for the index. Either the recipient preserves the donor’s spec, or createIndex normalizes bits at creation, as 9.0 does.
Notes
- The recipient creates the index on an empty collection, while the donor created it on a populated one.
SERVER-92282describes the normalized spec being applied on the empty-collection path, so the two paths may persist different forms of the same spec. - We found this in MongoDB’s Cluster-to-Cluster Sync (mongosync) tests, which shard already-indexed collections and move chunks before a migration starts. mongosync then sees the 2d index as inconsistent on the source and declines to copy it. jstests/core/index/geo/geo_small_large.js creates such indexes.
Repro script
// Tests whether a chunk migration preserves a 2d index’s fractional `bits` option. // // It creates a populated, unsharded collection on one shard, adds a 2d index with a fractional // `bits`, shards the collection on a hashed `_id`, and moves half of its chunks to a shard that // lacks the collection. Then it compares each shard’s copy of the index and runs the server’s own // metadata consistency check. // // Run it against a mongos of a cluster with at least 2 shards: // // mongosh --quiet mongodb://localhost:27017 repro_2d_bits_migration.js "use strict"; const DB_NAME = "repro_2d_bits"; const COLL_NAME = "coll"; const NS = `${DB_NAME}.${COLL_NAME}`; const BITS = 10.01262704984186; // Canonical extended JSON shows each number’s BSON type, so an int is distinguishable from a // double. (mongosh lacks the legacy shell’s tojson.) const tojson = (value) => EJSON.stringify(value, { relaxed: false }); const admin = db.getSiblingDB("admin"); const shards = commandWorked(admin.runCommand({ listShards: 1 })).shards; if (shards.length < 2) throw new Error("need a cluster with at least 2 shards"); const [donor, recipient] = shards; print(`Server version: ${db.version()}`); print(`Donor: ${donor._id}; recipient: ${recipient._id}`); const testDB = db.getSiblingDB(DB_NAME); testDB.dropDatabase(); // Put the database, and thus the unsharded collection, on the donor. commandWorked(admin.runCommand({ enableSharding: DB_NAME, primaryShard: donor._id })); const coll = testDB.getCollection(COLL_NAME); const docs = []; for (let i = 0; i < 1000; i++) { docs.push({ _id: i, p: [(i % 360) - 180, (i % 180) - 90] }); } coll.insertMany(docs); coll.createIndex({ p: "2d" }, { name: "p_2d", bits: BITS }); coll.createIndex({ _id: "hashed" }); printSpecs("After createIndex, before sharding"); commandWorked(admin.runCommand({ shardCollection: NS, key: { _id: "hashed" } })); // A populated collection starts with one chunk on the donor. Split the hashed range in half and // move the upper half to the recipient, which makes the recipient create the collection and its // indexes from the donor’s. commandWorked(admin.runCommand({ split: NS, middle: { _id: NumberLong("0") } })); commandWorked( admin.runCommand({ moveChunk: NS, bounds: [{ _id: NumberLong("0") }, { _id: MaxKey }], to: recipient._id, _waitForDelete: true, }), ); const specs = printSpecs("After moveChunk"); const bitsByShard = Object.fromEntries( Object.entries(specs).map(([shard, spec]) => [shard, spec == null ? null : spec.bits]), ); const distinctBits = new Set(Object.values(bitsByShard).map((b) => tojson(b))); if (distinctBits.size > 1) { print(`\nINCONSISTENT: “bits” differs across shards: ${tojson(bitsByShard)}`); } else { print(`\nConsistent: every shard has bits ${[...distinctBits][0]}`); } // checkMetadataConsistency exists on 7.0+ servers. const res = testDB.runCommand({ checkMetadataConsistency: 1, checkIndexes: true }); if (res.ok) { const inconsistencies = res.cursor.firstBatch; print(`\ncheckMetadataConsistency found ${inconsistencies.length} inconsistency(ies):`); inconsistencies.forEach((i) => printjson(i)); } else { print(`\ncheckMetadataConsistency unavailable: ${tojson(res)}`); } // Prints and returns each shard’s spec for the 2d index. $indexStats reports every shard’s own // copy of each index, which spares us connecting to shards directly (9.0+ forbids that). function printSpecs(label) { print(`\n== ${label}`); const specsByShard = {}; for (const shard of shards) { specsByShard[shard._id] = null; } coll.aggregate([{ $indexStats: {} }, { $match: { name: "p_2d" } }]).forEach((stat) => { specsByShard[stat.shard] = stat.spec; }); for (const [shard, spec] of Object.entries(specsByShard)) { print(`${shard}: ${spec == null ? "(no such index)" : tojson(spec)}`); } return specsByShard; } // Returns a command’s response, or throws if the command failed. (mongosh lacks resmoke’s // assert.commandWorked.) function commandWorked(res) { if (!res.ok) { throw new Error(`command failed: ${tojson(res)}`); } return res; }