[SERVER-65475] 4.0.21 -> 4.2 Upgrade issue, mongos adding limit geoNear queries, resulting in errors Created: 12/Apr/22  Updated: 17/May/22  Resolved: 17/May/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rinat N/A Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-22949 Remove default limit from $geoNear ag... Closed
Operating System: ALL
Steps To Reproduce:
  1. Install mongodb 4.0.21 with multiple shards
  2. update replicaset to 4.2 with mongos being left 4.0
  3. run query like this through mongos

db.bb_delivery_point.aggregate({$geoNear:{near: [ 11.766666, 22.62222 ],distanceField: "dis",maxDistance: 500000,spherical: true}}) 

  1. Mongos 4.0 adds limit of 100 to the query which results in mongod errors "$geoNear no longer supports the 'limit' parameter"
Participants:

 Description   

sHello, we have problems updating 11-sharded mongodb cluster from 4.0.21 (minor version update won't fix this, since there are no mentions of this bug in changelog) to 4.2. We want to update mongos without a downtime, which with 11 primaries and then mongos rolling update - will be significant.

After updating all confivsvrs and secondary replicas, when switching master, we get errors

 

$geoNear no longer supports the 'limit' parameter. Use a $limit stage instead. 

Thing is - we don't have any limits in queries.

It looks like mongos adding hardcoded 100 limit to queries, which results in errors above

https://github.com/mongodb/mongo/blob/r4.0.21/src/mongo/db/pipeline/document_source_geo_near.cpp#L282 
kDefaultLimit  = 100,

4.2 uses consctructor https://github.com/mongodb/mongo/blob/r4.2.17/src/mongo/db/pipeline/document_source_geo_near.cpp#L236 

notice: num: 100, while query doesn't have any limit:

jojo-db2-1 2022-02-25T10:53:34.165+0300 I COMMAND  [conn7462301] command jojo.bb_delivery_point appName: "MongoDB Shell" command: geoNear { geoNear: "bb_delivery_point", near: [ 57.766666, 37.62222 ], num: 100, maxDistance: 5000000.0, query: {}, collation: { locale: "simple" }, spherical: true, distanceMultiplier: 1.0, $db: "jojo" } planSummary: GEO_NEAR_2D { location: "2d" } keysExamined:489 docsExamined:462 numYields:4 reslen:126672 locks:{ Global: { acquireCount: { r: 5 } }, Database: { acquireCount: { r: 5 } }, Collection: { acquireCount: { r: 5 } } } storage:{} protocol:op_msg 322ms
jojo-db2-1 2022-02-25T10:53:34.167+0300 I COMMAND  [conn7462301] command jojo.bb_delivery_point appName: "MongoDB Shell" command: getMore { getMore: 5753461764780339751, collection: "bb_delivery_point", batchSize: 101, lsid: { id: UUID("b90747a7-6cf4-4b24-8161-032033d6baec"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) }, $clusterTime: { clusterTime: Timestamp(1645775613, 5526), signature: { hash: BinData(0, D94D58D7A3E272E81F946C403B32C5E6CBA96D62), keyId: 7033356981284897085 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.0.21" }, os: { type: "Linux", name: "CentOS Linux release 7.6.1810 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-1160.2.2.el7.x86_64" }, mongos: { host: "jojo-db0-2.p:27018", client: "127.0.0.1:49737", version: "4.0.21" } }, $configServerState: { opTime: { ts: Timestamp(1645775613, 4622), t: 34 } }, $db: "jojo" } originatingCommand: { aggregate: "bb_delivery_point", pipeline: [ { $geoNear: { near: [ 57.766666, 37.62222 ], distanceField: "dis", limit: 100, maxDistance: 5000000.0, query: {}, spherical: true, distanceMultiplier: 1.0 } } ], fromMongos: true, needsMerge: true, mergeByPBRT: false, collation: { locale: "simple" }, cursor: { batchSize: 0 }, allowImplicitCollectionCreation: true, shardVersion: [ Timestamp(7200, 0), ObjectId('58ca9326adcf092b7a927230') ], lsid: { id: UUID("b90747a7-6cf4-4b24-8161-032033d6baec"), uid: BinData(0, E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855) }, $clusterTime: { clusterTime: Timestamp(1645775613, 5518), signature: { hash: BinData(0, D94D58D7A3E272E81F946C403B32C5E6CBA96D62), keyId: 7033356981284897085 } }, $client: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.0.21" }, os: { type: "Linux", name: "CentOS Linux release 7.6.1810 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-1160.2.2.el7.x86_64" }, mongos: { host: "jojo-db0-2.p:27018", client: "127.0.0.1:49737", version: "4.0.21" } }, $configServerState: { opTime: { ts: Timestamp(1645775613, 4622), t: 34 } }, $db: "jojo" } cursorid:5753461764780339751 keysExamined:0 docsExamined:0 cursorExhausted:1 numYields:0 nreturned:100 reslen:128134 locks:{ Global: { acquireCount: { r: 9 } }, Database: { acquireCount: { r: 6 } }, Collection: { acquireCount: { r: 5 } } } storage:{} protocol:op_msg 324ms
 
 
[root@jojo-logs1 ~]# cat /logs/jojo_mongodb/mongos.log | grep '57.766666'
jojo-db0-2 2022-02-25T10:53:34.172+0300 I COMMAND  [conn27254798] command jojo.bb_delivery_point appName: "MongoDB Shell" command: aggregate { aggregate: "bb_delivery_point", pipeline: [ { $geoNear: { near: [ 57.766666, 37.62222 ], distanceField: "dis", maxDistance: 5000000.0, spherical: true } } ], cursor: {}, lsid: { id: UUID("b90747a7-6cf4-4b24-8161-032033d6baec") }, $clusterTime: { clusterTime: Timestamp(1645775611, 226), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $db: "jojo" } nShards:11 cursorExhausted:1 numYields:0 nreturned:100 reslen:132462 protocol:op_msg 331ms 

Thanks,

Rinat



 Comments   
Comment by Chris Kelly [ 17/May/22 ]

Hi Rinat,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Christopher

Comment by Chris Kelly [ 09/May/22 ]

Hi Rinat,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please help with the following:

  • Explain in greater detail the steps you took in your upgrade process?
  • Point out any issues in the steps I outlined above in reproducing your problem, if any?

Regards,
Christopher

 

Comment by Chris Kelly [ 21/Apr/22 ]

Hi Rinat,

 

I attempted to replicate your issue on a cluster with multiple shards, each with 3 nodes (PSS).

 

When using MongoDB 4.2.19 on config and all mongods (but not mongos), connecting to mongos (on 4.0.21) and running the query you mentioned doesn't produce the error on my end, which is odd. I also attempted this while only a subset of the mongod nodes were upgraded as well.

-> upgade config server to 4.2.19 (from 4.0.21)
-> upgrade mongods on all shards to 4.2.19 (from 4.0.21)
-> keep mongos at 4.0.21

 

When doing this, and executing your query from the mongos CLI, I do not see the error with FCV still set to 4.0 on all shards. If you attempt to set FCV to 4.2 at this stage, mongos will fail due to not being able to communicate with the newer 4.2 binaries until you either set FCV back to 4.0 or upgrade mongos to 4.2.

I used m (GitHub Link) to acquire the additional versions, and used the new database version binaries when restarting config/mongods (starting with secondaries, stepping down the primaries and then upgrading them, then the config server)

I am using sample data from https://www.mongodb.com/docs/manual/reference/operator/aggregation/geoNear/ and attempted the sample queries there as well as the query you provided (on that sample data) with no issues. I generated hundreds of sample data records with the same shape of that sample data as well with no issue.

I do, however, trigger the error if I specify limit as a parameter in the query manually (in the configuration mentioned above). Specifying limit at all will cause it to complain, even if set to null at this stage. I do not observe the default limit being set otherwise.

In order to look into this any further, I would need to corroborate reproducing the problem with you some more. Could you: 

  • Explain in greater detail the steps you took in your upgrade process?
  • Point out any issues in the steps I outlined above in reproducing your problem?

Regards,

Christopher

Generated at Thu Feb 08 06:02:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.