Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-9120

Explain that using the aggregation pipeline instead of collection.distinct() is a better equivalent to SQL "SELECT DISTINCT x FROM .."

    • Type: Icon: Task Task
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • Server_Docs_20231030
    • Affects Version/s: None
    • Component/s: manual
    • Labels:
      None
    • 0.25

      A customer who didn't know about or aggregation pipelines, or knew about it but maybe saw it as being too powerful, was trying to do the equivalent of "SELECT DISTINCT field1 FROM ...".

      The documentation he used was https://docs.mongodb.com/manual/reference/method/db.collection.distinct/. I suspect he may have also read https://docs.mongodb.com/manual/reference/sql-comparison/, which gives the example of "SELECT DISTINCT(status) FROM users" == "db.users.distinct( "status" )"

      He ran into the problem that he had hundreds of thousands of distinct values in the collection he was examining and this lead to "BSONObj size: 25448908 (0x18451CC) is invalid. Size must be between 0 and 16793600(16MB)" when he tried to use the distinct command. That is the distinct values weren't a small set, so the result size exceeded 16MB, so the packing of array result value threw an exception when it reached max BSON object size.

      Neither the db.collection.distinct page or https://docs.mongodb.com/manual/reference/command/distinct/#dbcmd.distinct for the underlying db command highlight that the single array result means a 16mb result size limit. Nor do they explain to avoid that and get a cursor instead (like a find command, which is the equivalent of the SQL SELECT command) you should use an aggregation pipeline per the example at https://docs.mongodb.com/manual/reference/operator/aggregation/group/#retrieve-distinct-values.

      I think it's time we start describing the distinct command / wrapper method as a convenience method with limited result size and point to the aggregation pipeline method ( "db.myCollection.aggregate( [ { $group : { _id : "$myField" } } ] )" ) as the fundamental way of fetching distinct values.

            Assignee:
            allison.moore@mongodb.com Allison Reinheimer Moore
            Reporter:
            akira.kurogane Akira Kurogane
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved:
              6 years, 13 weeks, 1 day ago