[SERVER-18965] $geoNear output limit of 16mb Created: 13/Jun/15  Updated: 06/Jul/19  Resolved: 13/Jun/15

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Geo
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Faruk Brbovic Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-22949 Remove default limit from $geoNear ag... Closed
Participants:
Case:

 Description   
Issue Status as of Apr 30, 2019

ISSUE DESCRIPTION AND IMPACT

The original implementation of the $geoNear aggregation stage used the geoNear command to perform its work.  During execution of the geo query, the incoming matching documents would be placed into a structure that was subject to a maximum size of 16MB.  If this threshold was reached, then any further documents were discarded.  This partial result set would still be returned to the client and a warning would be written to the log file. 

With the deprecation (in 4.0) and subsequent removal (in 4.2) of the geoNear command, the $geoNear aggregation stage was re-written.  This new implementation provides the same general functionality but is no longer subject to the previous 16MB limitation. 

DIAGNOSIS AND AFFECTED VERSIONS

This limitation exists for all versions of the database that use the original geoNear command, which is up to and including the 4.0 series.  An indication that the 16MB threshold has been reached is recorded in the log file with an entry that resembles the following:

2019-04-25T13:34:00.307-0700 W COMMAND  [conn9] Too many geoNear results for query { position: { $nearSphere: { type: "Point", coordinates: [ 0.0, 0.0 ] }, $maxDistance: 9.999999999999999e+22 } }, truncating output. 

REMEDIATION AND WORKAROUNDS

The fix is included in the 4.2 production release as the removed geoNear command was re-implemented as a $geoNear aggregation stage which no longer has the limitation. 

Potential workarounds prior to using 4.2 include the following:

  • Use the geoNear operators available to the find command. If any subsequent transformations to the data are required, then either:
    1) Persist the output of the find operation to a collection that can have a subsequent aggregation run against it to produce the final results, or
    2) Perform the remaining computations client-side.
  • If the results do not need to be sorted by distance, then $geoWithin may be a suitable alternative. 
Original description

Currently. Geonear command seems to have a limit of 16mb for its results. This is a major limitation in many use cases especially when large result sets are returned in aggregation pipeline. Since GeoNear is the only geo command which outputs distance calculation there is no other way to get the calculation of distance.

Any plans for increasing this limit ?

Thanks



 Comments   
Comment by Cosmin Neag [ 27/Jan/19 ]

I've got the same issue. Initially the results were fine but in time the size of my documents has increased and that impacted the result set by returning only a subset of data (16mb limitation)

Comment by Faruk Brbovic [ 13/Jun/15 ]

I have asked around stackoverflow and other forums but no solution.
Problem with the aggregation cursor solution you propose is that Geonear only outputs max 16mb to the pipeline so data doesn't even get to the cursor.

Problem with any other aggregation pipeline solution is that there is no way within the aggregation pipeline to calculate geospatial distance since there is no cos() , sin() and sqrroot functionality within pipeline and there is no way to call custom JavaScript functionality from within the pipeline.

This limitation pretty much renders GeoNear functionality useless for any serious data lookups.

Comment by Ramon Fernandez Marina [ 13/Jun/15 ]

Thanks for your report fbrbovic. Increasing the document size was proposed in SERVER-5923, but there are currently no plans to do that. The aggregation pipeline can return result sets of any size by returning a cursor, so you should be able to tweak your pipeline to get the results you need.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Generated at Thu Feb 08 03:49:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.