Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-83756

[$vectorSearch, $project, $match] not passing documents when match depends on meta field from project

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.0.0-rc0
    • Affects Version/s: 6.0.11, 6.0.12
    • Component/s: None
    • None
    • Query Integration
    • Fully Compatible
    • ALL
      1. Create a search index named vector-index with the provided index definition over a new collection.
      2. Insert the provided sample document to the indexed collection.
      3. Run the aggregation pipeline provided.
    • QI 2023-12-25, QI 2024-01-08, QI 2024-01-22, QI 2024-02-05, QI 2024-02-19

      This issue was noticed in https://jira.mongodb.org/browse/HELP-52233.

      The following pipeline does not match documents when expected in 6.0.11 and 6.0.12. The pipeline does not error, and unexpectedly does not return any documents.

      [
        {
          "$vectorSearch": {
            "index": "vector-index",
            "path": "vector",
            "queryVector": [
              1.1,
              2.1
            ],
            "numCandidates": 150,
            "limit": 10
          }
        },
        {
          "$project": {
            "vector": 0,
            "score": {
              "$meta": "vectorSearchScore"
            }
          }
        },
        {
          "$match": {
            "score": {
              "$gt": 0.19
            }
          }
        }
      ] 

      When running the same pipeline with just the first [$vectorSearch, $project] stages, documents return that do have score values greater than 0.19.

      Interestingly, it seems like this issue only occurs when the $project stage excludes a field. If the above pipeline removes the vector: 0 part of the $project stage, the pipeline returns the expected set of documents.

      Sample Document

      {
        "_id": {
          "$oid": "573a1390f29313caabcd42e8"
        },
        "fullplot": "Among the earliest existing films in American cinema - notable as the first film that presented a narrative story to tell - it depicts a group of cowboy outlaws who hold up a train and rob the passengers. They are then pursued by a Sheriff's posse. Several scenes have color included - all hand tinted.",
        "title": "The Great Train Robbery",
        "vector": [
          -0.11765024065971375,
          -0.21213145554065704,
          -0.2577494978904724,
          0.022029373794794083,
          0.18414953351020813,
          0.36586806178092957,
          0.11862823367118835,
          0.08321523666381836,
          0.29645663499832153,
          0.0022856774739921093,
          0.2824995219707489,
          0.29961642622947693,
          0.04239010438323021,
          0.4067825973033905,
          -0.1598043292760849,
          0.2148534059524536,
          0.1679493486881256,
          0.16969089210033417,
          0.3145444989204407,
          -0.07887133210897446,
          -0.18495561182498932,
          -0.09759827703237534,
          0.20752380788326263,
          0.0002660630561877042,
          0.025117194280028343,
          -0.3236020505428314,
          0.21704871952533722,
          -0.20958499610424042,
          -0.30887970328330994,
          -0.39369162917137146,
          -0.4541487991809845,
          -0.3006436824798584,
          0.2732269763946533,
          0.28519636392593384,
          -0.27733004093170166,
          0.09134820848703384,
          0.19789211452007294,
          0.16620713472366333,
          0.15451955795288086,
          -0.16677556931972504,
          -0.11001905798912048,
          0.2350681871175766,
          0.14352057874202728,
          0.08762780576944351,
          -0.14170658588409424,
          -0.2390274554491043,
          -0.06526391208171844,
          0.19815577566623688,
          0.11691268533468246,
          -0.10566674917936325,
          -0.01293251197785139,
          0.1028236523270607,
          0.0663963183760643,
          -0.15577580034732819,
          0.05088368430733681,
          -0.3613817095756531,
          0.035468459129333496,
          -0.2183608114719391,
          -0.07660580426454544,
          0.07465414702892303,
          -0.34323880076408386,
          -0.25135529041290283,
          -0.035666778683662415,
          -0.17220216989517212,
          0.0796608179807663,
          0.11376814544200897,
          -0.26657962799072266,
          -0.10785310715436935,
          0.0011672211112454534,
          0.060230884701013565,
          -0.059616390615701675,
          0.1091652512550354,
          -0.2163013517856598,
          -0.042512599378824234,
          -0.21879377961158752,
          -0.03120023012161255,
          0.26145562529563904,
          0.4099317491054535,
          -0.07657518982887268,
          -0.19645383954048157,
          0.1214444488286972,
          -0.15863333642482758,
          -0.21740862727165222,
          -0.09555281698703766,
          -0.16180285811424255,
          0.3358859717845917,
          -0.007481173612177372,
          0.0592496432363987,
          -0.45026442408561707,
          0.30858924984931946,
          -0.3005770444869995,
          -0.056945886462926865,
          0.3019435703754425,
          0.20108728110790253,
          0.21957458555698395,
          0.03857314586639404,
          0.20800574123859406,
          0.14485691487789154,
          0.1916215568780899,
          -0.0020670725498348475,
          0.3323781192302704,
          -0.3285926878452301,
          0.3763233721256256,
          -0.0013920333003625274,
          0.07155482470989227,
          -0.09482531994581223,
          -0.0708141177892685,
          -0.1718258559703827,
          -0.37243640422821045,
          -0.06825312227010727,
          0.16272546350955963,
          0.14173169434070587,
          -0.14848820865154266,
          0.15826699137687683,
          0.3839591443538666,
          -0.2125476896762848,
          0.07903873175382614,
          0.44199442863464355,
          -0.26676616072654724,
          0.09757755696773529,
          0.13383761048316956,
          0.14373788237571716,
          0.09867347031831741,
          -0.1315346658229828,
          -0.3013248145580292,
          -0.17324237525463104,
          0.21638785302639008,
          0.09092690795660019,
          -0.10350912064313889,
          -0.19982074201107025,
          -0.5378654599189758,
          -0.07663127779960632,
          0.3250696063041687,
          -0.07982280850410461,
          -0.3214959502220154,
          0.2379886358976364,
          -0.058239664882421494,
          0.30418604612350464,
          -0.2900577187538147,
          -0.22941656410694122,
          -0.08184462785720825,
          -0.023101476952433586,
          -0.153031125664711,
          0.1359335333108902,
          -0.1226024180650711,
          -0.1791592389345169,
          0.09988260269165039,
          -0.020172426477074623,
          -0.2852059304714203,
          0.45608699321746826,
          0.22607013583183289,
          -0.13433052599430084,
          -0.14698535203933716,
          -0.043917521834373474,
          -0.1707649827003479,
          -0.18633048236370087,
          -0.06523872166872025,
          0.16834892332553864,
          0.18413908779621124,
          0.2815704941749573,
          0.11103569716215134,
          0.30391865968704224,
          0.283029168844223,
          0.1416444331407547,
          -0.042800698429346085,
          -0.24971401691436768,
          0.23590309917926788,
          0.06522729992866516,
          0.08855345100164413,
          -0.05423424392938614,
          -0.2309979796409607,
          0.1957871913909912,
          -0.23140402138233185,
          0.18335404992103577,
          -0.13024133443832397,
          0.2774726152420044,
          -0.340467631816864,
          0.441804438829422,
          -0.276013046503067,
          0.12379983067512512,
          0.04399506747722626,
          -0.1829381287097931,
          -0.2591503858566284,
          -0.11711062490940094,
          -0.0632048100233078,
          -0.026645071804523468,
          0.18277552723884583,
          0.021059319376945496,
          0.3614908456802368,
          0.06206006184220314,
          -0.07163409888744354,
          -0.041581377387046814,
          -0.17677843570709229,
          -0.131730318069458,
          0.21672207117080688,
          0.38391438126564026,
          0.24665479362010956,
          0.06858744472265244,
          -0.10113944113254547,
          0.11653739959001541,
          -0.3127738833427429,
          -0.20196931064128876,
          -0.16854704916477203,
          0.20115606486797333,
          0.1781543344259262,
          -0.02193872258067131,
          0.1828240305185318,
          -0.14550761878490448,
          -0.2326435148715973,
          0.11454261094331741,
          0.297747939825058,
          0.022496163845062256,
          0.0538192093372345,
          0.3587631285190582,
          0.15320129692554474,
          -0.19066448509693146,
          0.015059860423207283,
          0.0029263466130942106,
          -0.06956949084997177,
          0.1699836105108261,
          -0.06845001876354218,
          0.0017190612852573395,
          0.3263634443283081,
          -0.07113509625196457,
          0.3319341540336609,
          0.010027533397078514,
          0.16950343549251556,
          -0.10483969748020172,
          -0.2998875379562378,
          -0.23045486211776733,
          -0.2758725881576538,
          -0.11985764652490616,
          -0.2686576545238495,
          -0.19980600476264954,
          -0.4138680100440979,
          -0.1077808067202568,
          -0.1492578089237213,
          0.13200101256370544,
          0.36830294132232666,
          -0.08478769659996033,
          0.033339790999889374,
          -0.002336981939151883,
          -0.03442233055830002,
          0.32943832874298096,
          0.03849964961409569,
          -0.24022988975048065,
          -0.10006731003522873,
          0.012672185897827148,
          0.06638535857200623,
          0.06642032414674759,
          0.07391534745693207,
          -0.24832631647586823,
          -0.22292757034301758,
          -0.00998215563595295,
          0.06770025193691254,
          0.0005948301986791193,
          0.11045613139867783,
          -0.013355172239243984,
          -0.3046444356441498,
          0.12517647445201874,
          0.21137303113937378,
          0.2320733666419983,
          0.12761768698692322,
          -0.14024242758750916,
          0.10619757324457169,
          0.004375155083835125,
          -0.2231045663356781,
          0.08461868017911911,
          -0.2950367331504822,
          -0.3721398711204529,
          0.17330145835876465,
          0.23854418098926544,
          -0.5307270288467407,
          0.012168901972472668,
          -0.008676702156662941,
          -0.11905895918607712,
          0.014401458203792572,
          -0.2830154299736023,
          -0.4244348108768463,
          -0.07601073384284973,
          -0.12438468635082245,
          -0.11757797747850418,
          0.3823363482952118,
          0.17712704837322235,
          -0.2841915786266327,
          0.09009560197591782,
          -0.21773451566696167,
          -0.35735413432121277,
          -0.019248757511377335,
          -0.05660978704690933,
          -0.4503512680530548,
          0.17601518332958221,
          -0.07592517882585526,
          -0.017740637063980103,
          0.2308356910943985,
          0.18754231929779053,
          -0.10002169758081436,
          0.19592846930027008,
          -0.08100283145904541,
          0.46763908863067627,
          -0.01018767710775137,
          -0.06349238753318787,
          -0.3370194435119629,
          -0.000698139367159456,
          0.00844316091388464,
          0.05779176577925682,
          -0.07074493914842606,
          0.242868572473526,
          0.3492843508720398,
          0.08455496281385422,
          -0.20705756545066833,
          -0.19167593121528625,
          0.13049881160259247,
          0.475999653339386,
          0.10838420689105988,
          0.014645983465015888,
          0.26804178953170776,
          -0.004578789696097374,
          -0.2304943948984146,
          -0.22115907073020935,
          -0.18102207779884338,
          0.3641524314880371,
          0.1879137009382248,
          -0.2001994103193283,
          0.22903770208358765,
          0.339185506105423,
          0.09152352064847946,
          0.3772352635860443,
          -0.359032541513443,
          -0.20067743957042694,
          -0.29734930396080017,
          0.08637548238039017,
          0.1874120682477951,
          -0.3613969087600708,
          -0.24671083688735962,
          0.3716210424900055,
          0.24372325837612152,
          -0.010151015594601631,
          0.038459960371255875,
          0.2505260705947876,
          0.11547160893678665,
          0.19965754449367523,
          0.02591020241379738,
          -0.11946342885494232,
          0.16225086152553558,
          -0.29998981952667236,
          -0.08956541121006012,
          -0.17742612957954407,
          0.04627487435936928,
          0.11556537449359894,
          -0.21252484619617462,
          0.18583796918392181,
          -0.2050047516822815,
          -0.04747660830616951,
          0.07976546138525009,
          -0.053681228309869766,
          -0.1431625485420227,
          -0.053266748785972595,
          0.2652992010116577,
          -0.3357309401035309,
          0.020267300307750702,
          -0.11011805385351181,
          -0.004514480009675026,
          -0.07781223952770233,
          0.26643383502960205,
          0.19736529886722565,
          -0.16440820693969727,
          0.03900334984064102,
          -0.07384948432445526,
          -0.23138369619846344,
          0.24710531532764435,
          -0.28062722086906433,
          -0.18227963149547577,
          0.2834342420101166,
          0.09329964965581894,
          -0.043751057237386703,
          0.021708974614739418,
          0.1445489078760147,
          -0.04934481903910637,
          -0.008936065249145031,
          -0.11593864858150482,
          -0.14795023202896118,
          0.3682711124420166,
          -0.10305444151163101
        ]
      } 

      Search Index Definition

      {
        "mappings": {
          "dynamic": false,
          "fields": {
            "title": {
              "type": "token"
            },
            "vector": {
              "dimensions": 384,
              "similarity": "cosine",
              "type": "knnVector"
            }
          }
        }
      } 

      Versions Impacted

      This issue was first raised in https://jira.mongodb.org/browse/HELP-52233.

      The user experienced the issue on 6.0.11 and in the course of investigating the issue their cluster upgraded to 6.0.12. The issue seems present in at least 6.0.11 and 6.0.12. It is unclear if this impacts other versions.

            Assignee:
            reilly.mcbride@mongodb.com Reilly McBride (Inactive)
            Reporter:
            evan.nixon@mongodb.com Evan Nixon
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: