[SERVER-68130] AutoSplitVector could generate response bigger than BSONObjMaxUserSize Created: 19/Jul/22  Updated: 29/Oct/23  Resolved: 25/Jul/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.1, 4.4.16, 4.2.22, 5.0.11, 6.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Tommaso Tocci Assignee: Pierlauro Sciarelli
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.0, v5.0, v4.4, v4.2
Sprint: Sharding EMEA 2022-08-08
Participants:

 Description   

In HELP-35620 AutoSplitVector was called by the create collection coordinator and generated a response with too many split points. So when the response was serialized to a BSONObject the command failed with a BSONObjectTooLarge error.

The AutoSplitVector has a logic in place to prevent this. In fact, while building the split points array, it tries to predict the final BSON object size and stops in case the maximum size is reached.

As demonstrated in HELP-35620 this logic is somehow broken. From the customer's log we can see that the AutoSplitVector failed with BSONObjectTooLarge:

2022-07-06T11:40:55.206-05:00 E  COMMAND  4879802 [conn124862] "Failed to handle request","attr":{"error":"BSONObjectTooLarge: BSON size limit hit while building Message. Size: 17973680 (0x11241B0); maxSize: 16793600(16MB)"}

From the collection entry we can derive the ShardKey pattern for the collection:

{"_id":"<collection.name>","lastmodEpoch":{"$oid":"62c70e4d59dd43b776579a8c"},"lastmod":{"$date":{"$numberLong":"1657212494323"}},"timestamp":{"$timestamp":{"t":1657212495,"i":218}},"uuid":{"$binary":{"base64":"pMa3VhfSS2ChFp59zYnnTg==","subType":"04"}},"key":{"ne_unique_id":{"$numberInt":"1"},"area-2":{"$numberInt":"1"},"cnum":{"$numberInt":"1"},"timestamp":{"$numberInt":"1"}},"unique":true,"noBalance":false}

And from the chunks we can derive a possible split point structure:

{
  "_id": {
    "$oid": "62c740258c72e513f017a59c"
  },
  "uuid": {
    "$binary": {
      "base64": "pMa3VhfSS2ChFp59zYnnTg==",
      "subType": "04"
    }
  },
  "min": {
    "ne_unique_id": "AFK_247123",
    "area-2": {
      "$numberInt": "247"
    },
    "cnum": {
      "$numberInt": "45"
    },
    "timestamp": {
      "$date": {
        "$numberLong": "1627797600000"
      }
    }
  },
  "max": {
    "ne_unique_id": "AFK_247123",
    "area-2": {
      "$numberInt": "247"
    },
    "cnum": {
      "$numberInt": "55"
    },
    "timestamp": {
      "$date": {
        "$numberLong": "1636117200000"
      }
    }
  },
  "shard": "myShard_0",
  "lastmod": {
    "$timestamp": {
      "t": 1540,
      "i": 40
    }
  },
  "history": [
    {
      "validAfter": {
        "$timestamp": {
          "t": 1657212495,
          "i": 218
        }
      },
      "shard": "myShard_0"
    }
  ]
}



 Comments   
Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}

Message: SERVER-68130 adjust autoSplitVector's estimatedAdditionalBytesPerItemInBSONArray
Branch: v4.2
https://github.com/mongodb/mongo/commit/7438eb5acd7f99ad7e1a618eb13598f3e5e5cff4

Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}

Message: SERVER-68130 adjust autoSplitVector's estimatedAdditionalBytesPerItemInBSONArray
Branch: v4.4
https://github.com/mongodb/mongo/commit/daa784ee5f0deb97b15c78fd1adb6b9186fed414

Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}

Message: SERVER-68130 adjust autoSplitVector's estimatedAdditionalBytesPerItemInBSONArray
Branch: v5.0
https://github.com/mongodb/mongo/commit/c42b4e0287cee7894cc4fc6d295a28c03f49baaa

Comment by Githook User [ 26/Jul/22 ]

Author:

{'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}

Message: SERVER-68130 adjust autoSplitVector's estimatedAdditionalBytesPerItemInBSONArray
Branch: v6.0
https://github.com/mongodb/mongo/commit/5f5940439f371d712670567ef117c0dd5bfe7fba

Comment by Githook User [ 25/Jul/22 ]

Author:

{'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}

Message: SERVER-68130 adjust autoSplitVector's estimatedAdditionalBytesPerItemInBSONArray
Branch: master
https://github.com/mongodb/mongo/commit/c0e3f3796a417747ccd1a8a97cb4fc21d78f2fc5

Generated at Thu Feb 08 06:09:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.