Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-83756

[$vectorSearch, $project, $match] not passing documents when match depends on meta field from project

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.0.0-rc0
    • Affects Version/s: 6.0.11, 6.0.12
    • Component/s: None
    • None
    • Query Integration
    • Fully Compatible
    • ALL
      1. Create a search index named vector-index with the provided index definition over a new collection.
      2. Insert the provided sample document to the indexed collection.
      3. Run the aggregation pipeline provided.
    • QI 2023-12-25, QI 2024-01-08, QI 2024-01-22, QI 2024-02-05, QI 2024-02-19

      This issue was noticed in https://jira.mongodb.org/browse/HELP-52233.

      The following pipeline does not match documents when expected in 6.0.11 and 6.0.12. The pipeline does not error, and unexpectedly does not return any documents.

      [
        {
          "$vectorSearch": {
            "index": "vector-index",
            "path": "vector",
            "queryVector": [
              1.1,
              2.1
            ],
            "numCandidates": 150,
            "limit": 10
          }
        },
        {
          "$project": {
            "vector": 0,
            "score": {
              "$meta": "vectorSearchScore"
            }
          }
        },
        {
          "$match": {
            "score": {
              "$gt": 0.19
            }
          }
        }
      ] 

      When running the same pipeline with just the first [$vectorSearch, $project] stages, documents return that do have score values greater than 0.19.

      Interestingly, it seems like this issue only occurs when the $project stage excludes a field. If the above pipeline removes the vector: 0 part of the $project stage, the pipeline returns the expected set of documents.

      Sample Document

      {
        "_id": {
          "$oid": "573a1390f29313caabcd42e8"
        },
        "fullplot": "Among the earliest existing films in American cinema - notable as the first film that presented a narrative story to tell - it depicts a group of cowboy outlaws who hold up a train and rob the passengers. They are then pursued by a Sheriff's posse. Several scenes have color included - all hand tinted.",
        "title": "The Great Train Robbery",
        "vector": [
          -0.11765024065971375,
          -0.21213145554065704,
          -0.2577494978904724,
          0.022029373794794083,
          0.18414953351020813,
          0.36586806178092957,
          0.11862823367118835,
          0.08321523666381836,
          0.29645663499832153,
          0.0022856774739921093,
          0.2824995219707489,
          0.29961642622947693,
          0.04239010438323021,
          0.4067825973033905,
          -0.1598043292760849,
          0.2148534059524536,
          0.1679493486881256,
          0.16969089210033417,
          0.3145444989204407,
          -0.07887133210897446,
          -0.18495561182498932,
          -0.09759827703237534,
          0.20752380788326263,
          0.0002660630561877042,
          0.025117194280028343,
          -0.3236020505428314,
          0.21704871952533722,
          -0.20958499610424042,
          -0.30887970328330994,
          -0.39369162917137146,
          -0.4541487991809845,
          -0.3006436824798584,
          0.2732269763946533,
          0.28519636392593384,
          -0.27733004093170166,
          0.09134820848703384,
          0.19789211452007294,
          0.16620713472366333,
          0.15451955795288086,
          -0.16677556931972504,
          -0.11001905798912048,
          0.2350681871175766,
          0.14352057874202728,
          0.08762780576944351,
          -0.14170658588409424,
          -0.2390274554491043,
          -0.06526391208171844,
          0.19815577566623688,
          0.11691268533468246,
          -0.10566674917936325,
          -0.01293251197785139,
          0.1028236523270607,
          0.0663963183760643,
          -0.15577580034732819,
          0.05088368430733681,
          -0.3613817095756531,
          0.035468459129333496,
          -0.2183608114719391,
          -0.07660580426454544,
          0.07465414702892303,
          -0.34323880076408386,
          -0.25135529041290283,
          -0.035666778683662415,
          -0.17220216989517212,
          0.0796608179807663,
          0.11376814544200897,
          -0.26657962799072266,
          -0.10785310715436935,
          0.0011672211112454534,
          0.060230884701013565,
          -0.059616390615701675,
          0.1091652512550354,
          -0.2163013517856598,
          -0.042512599378824234,
          -0.21879377961158752,
          -0.03120023012161255,
          0.26145562529563904,
          0.4099317491054535,
          -0.07657518982887268,
          -0.19645383954048157,
          0.1214444488286972,
          -0.15863333642482758,
          -0.21740862727165222,
          -0.09555281698703766,
          -0.16180285811424255,
          0.3358859717845917,
          -0.007481173612177372,
          0.0592496432363987,
          -0.45026442408561707,
          0.30858924984931946,
          -0.3005770444869995,
          -0.056945886462926865,
          0.3019435703754425,
          0.20108728110790253,
          0.21957458555698395,
          0.03857314586639404,
          0.20800574123859406,
          0.14485691487789154,
          0.1916215568780899,
          -0.0020670725498348475,
          0.3323781192302704,
          -0.3285926878452301,
          0.3763233721256256,
          -0.0013920333003625274,
          0.07155482470989227,
          -0.09482531994581223,
          -0.0708141177892685,
          -0.1718258559703827,
          -0.37243640422821045,
          -0.06825312227010727,
          0.16272546350955963,
          0.14173169434070587,
          -0.14848820865154266,
          0.15826699137687683,
          0.3839591443538666,
          -0.2125476896762848,
          0.07903873175382614,
          0.44199442863464355,
          -0.26676616072654724,
          0.09757755696773529,
          0.13383761048316956,
          0.14373788237571716,
          0.09867347031831741,
          -0.1315346658229828,
          -0.3013248145580292,
          -0.17324237525463104,
          0.21638785302639008,
          0.09092690795660019,
          -0.10350912064313889,
          -0.19982074201107025,
          -0.5378654599189758,
          -0.07663127779960632,
          0.3250696063041687,
          -0.07982280850410461,
          -0.3214959502220154,
          0.2379886358976364,
          -0.058239664882421494,
          0.30418604612350464,
          -0.2900577187538147,
          -0.22941656410694122,
          -0.08184462785720825,
          -0.023101476952433586,
          -0.153031125664711,
          0.1359335333108902,
          -0.1226024180650711,
          -0.1791592389345169,
          0.09988260269165039,
          -0.020172426477074623,
          -0.2852059304714203,
          0.45608699321746826,
          0.22607013583183289,
          -0.13433052599430084,
          -0.14698535203933716,
          -0.043917521834373474,
          -0.1707649827003479,
          -0.18633048236370087,
          -0.06523872166872025,
          0.16834892332553864,
          0.18413908779621124,
          0.2815704941749573,
          0.11103569716215134,
          0.30391865968704224,
          0.283029168844223,
          0.1416444331407547,
          -0.042800698429346085,
          -0.24971401691436768,
          0.23590309917926788,
          0.06522729992866516,
          0.08855345100164413,
          -0.05423424392938614,
          -0.2309979796409607,
          0.1957871913909912,
          -0.23140402138233185,
          0.18335404992103577,
          -0.13024133443832397,
          0.2774726152420044,
          -0.340467631816864,
          0.441804438829422,
          -0.276013046503067,
          0.12379983067512512,
          0.04399506747722626,
          -0.1829381287097931,
          -0.2591503858566284,
          -0.11711062490940094,
          -0.0632048100233078,
          -0.026645071804523468,
          0.18277552723884583,
          0.021059319376945496,
          0.3614908456802368,
          0.06206006184220314,
          -0.07163409888744354,
          -0.041581377387046814,
          -0.17677843570709229,
          -0.131730318069458,
          0.21672207117080688,
          0.38391438126564026,
          0.24665479362010956,
          0.06858744472265244,
          -0.10113944113254547,
          0.11653739959001541,
          -0.3127738833427429,
          -0.20196931064128876,
          -0.16854704916477203,
          0.20115606486797333,
          0.1781543344259262,
          -0.02193872258067131,
          0.1828240305185318,
          -0.14550761878490448,
          -0.2326435148715973,
          0.11454261094331741,
          0.297747939825058,
          0.022496163845062256,
          0.0538192093372345,
          0.3587631285190582,
          0.15320129692554474,
          -0.19066448509693146,
          0.015059860423207283,
          0.0029263466130942106,
          -0.06956949084997177,
          0.1699836105108261,
          -0.06845001876354218,
          0.0017190612852573395,
          0.3263634443283081,
          -0.07113509625196457,
          0.3319341540336609,
          0.010027533397078514,
          0.16950343549251556,
          -0.10483969748020172,
          -0.2998875379562378,
          -0.23045486211776733,
          -0.2758725881576538,
          -0.11985764652490616,
          -0.2686576545238495,
          -0.19980600476264954,
          -0.4138680100440979,
          -0.1077808067202568,
          -0.1492578089237213,
          0.13200101256370544,
          0.36830294132232666,
          -0.08478769659996033,
          0.033339790999889374,
          -0.002336981939151883,
          -0.03442233055830002,
          0.32943832874298096,
          0.03849964961409569,
          -0.24022988975048065,
          -0.10006731003522873,
          0.012672185897827148,
          0.06638535857200623,
          0.06642032414674759,
          0.07391534745693207,
          -0.24832631647586823,
          -0.22292757034301758,
          -0.00998215563595295,
          0.06770025193691254,
          0.0005948301986791193,
          0.11045613139867783,
          -0.013355172239243984,
          -0.3046444356441498,
          0.12517647445201874,
          0.21137303113937378,
          0.2320733666419983,
          0.12761768698692322,
          -0.14024242758750916,
          0.10619757324457169,
          0.004375155083835125,
          -0.2231045663356781,
          0.08461868017911911,
          -0.2950367331504822,
          -0.3721398711204529,
          0.17330145835876465,
          0.23854418098926544,
          -0.5307270288467407,
          0.012168901972472668,
          -0.008676702156662941,
          -0.11905895918607712,
          0.014401458203792572,
          -0.2830154299736023,
          -0.4244348108768463,
          -0.07601073384284973,
          -0.12438468635082245,
          -0.11757797747850418,
          0.3823363482952118,
          0.17712704837322235,
          -0.2841915786266327,
          0.09009560197591782,
          -0.21773451566696167,
          -0.35735413432121277,
          -0.019248757511377335,
          -0.05660978704690933,
          -0.4503512680530548,
          0.17601518332958221,
          -0.07592517882585526,
          -0.017740637063980103,
          0.2308356910943985,
          0.18754231929779053,
          -0.10002169758081436,
          0.19592846930027008,
          -0.08100283145904541,
          0.46763908863067627,
          -0.01018767710775137,
          -0.06349238753318787,
          -0.3370194435119629,
          -0.000698139367159456,
          0.00844316091388464,
          0.05779176577925682,
          -0.07074493914842606,
          0.242868572473526,
          0.3492843508720398,
          0.08455496281385422,
          -0.20705756545066833,
          -0.19167593121528625,
          0.13049881160259247,
          0.475999653339386,
          0.10838420689105988,
          0.014645983465015888,
          0.26804178953170776,
          -0.004578789696097374,
          -0.2304943948984146,
          -0.22115907073020935,
          -0.18102207779884338,
          0.3641524314880371,
          0.1879137009382248,
          -0.2001994103193283,
          0.22903770208358765,
          0.339185506105423,
          0.09152352064847946,
          0.3772352635860443,
          -0.359032541513443,
          -0.20067743957042694,
          -0.29734930396080017,
          0.08637548238039017,
          0.1874120682477951,
          -0.3613969087600708,
          -0.24671083688735962,
          0.3716210424900055,
          0.24372325837612152,
          -0.010151015594601631,
          0.038459960371255875,
          0.2505260705947876,
          0.11547160893678665,
          0.19965754449367523,
          0.02591020241379738,
          -0.11946342885494232,
          0.16225086152553558,
          -0.29998981952667236,
          -0.08956541121006012,
          -0.17742612957954407,
          0.04627487435936928,
          0.11556537449359894,
          -0.21252484619617462,
          0.18583796918392181,
          -0.2050047516822815,
          -0.04747660830616951,
          0.07976546138525009,
          -0.053681228309869766,
          -0.1431625485420227,
          -0.053266748785972595,
          0.2652992010116577,
          -0.3357309401035309,
          0.020267300307750702,
          -0.11011805385351181,
          -0.004514480009675026,
          -0.07781223952770233,
          0.26643383502960205,
          0.19736529886722565,
          -0.16440820693969727,
          0.03900334984064102,
          -0.07384948432445526,
          -0.23138369619846344,
          0.24710531532764435,
          -0.28062722086906433,
          -0.18227963149547577,
          0.2834342420101166,
          0.09329964965581894,
          -0.043751057237386703,
          0.021708974614739418,
          0.1445489078760147,
          -0.04934481903910637,
          -0.008936065249145031,
          -0.11593864858150482,
          -0.14795023202896118,
          0.3682711124420166,
          -0.10305444151163101
        ]
      } 

      Search Index Definition

      {
        "mappings": {
          "dynamic": false,
          "fields": {
            "title": {
              "type": "token"
            },
            "vector": {
              "dimensions": 384,
              "similarity": "cosine",
              "type": "knnVector"
            }
          }
        }
      } 

      Versions Impacted

      This issue was first raised in https://jira.mongodb.org/browse/HELP-52233.

      The user experienced the issue on 6.0.11 and in the course of investigating the issue their cluster upgraded to 6.0.12. The issue seems present in at least 6.0.11 and 6.0.12. It is unclear if this impacts other versions.

            Assignee:
            reilly.mcbride@mongodb.com Reilly McBride
            Reporter:
            evan.nixon@mongodb.com Evan Nixon
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: