$project silently drops root-level fields after $lookup + $unwind when multiple nested documents contain a type field with different value types

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: 8.0.20
    • Component/s: None
    • ALL
    • Hide

      js
       {{// Collection: Events (root document structure){type: "PURCHASE_CREATED", // stringcontact_id: "uuid-1",transaction:

      { id: "txn-uuid-1" }

      ,venue:

      { // THIS FIELD GETS DROPPEDid: "venue-uuid-1",name: "Some Venue"}

      ,products: [{type: { // nested object type — triggers the bugid: "product-type-uuid",name: "Generic Products"}}]}// Collection: RewardEvents

      {id: "txn-uuid-1", // joins on transaction.idtype: "REWARD_EVENT", // string — conflict with root typetotal_award: Decimal128("3.00"),total_spend: Decimal128("0.00"),total_open_spend: Decimal128("0.00"),total_commerce_spend: Decimal128("0.00"),total_crm_wallet_spend: Decimal128("0.00")}

      // Collection: ContactProfiles{contact_id: "uuid-1", // joins on contact_idtype: "CONTACT_UPDATED", // string — conflict with root typeprofile: {id: "uuid-1",name: "Some Name",type: "PERSON" // nested string type}}}}
      Pipeline:
       
       
      js
       {{[{ $match:

      { "transaction.id": "txn-uuid-1" }

      },{$lookup: {from: "RewardEvents",localField: "transaction.id",foreignField: "id",as: "RewardEvents"}},

      { $unwind: "$RewardEvents" }

      ,{$lookup: {from: "ContactProfiles",localField: "contact_id",foreignField: "contact_id",as: "ContactProfiles"}},

      { $unwind: "$ContactProfiles" }

      ,{$project: {type: 1,contact_id: 1,merchant: 1,submited_date: 1,transaction: 1,venue: 1, // silently dropped"RewardEvents.total_award": 1,"RewardEvents.total_spend": 1,"RewardEvents.total_open_spend": 1,"RewardEvents.total_commerce_spend": 1,"RewardEvents.total_crm_wallet_spend": 1,"ContactProfiles.profile": 1,_id: 0}}]}}

      Expected Result:
       
       
      js
       {{{type: "PURCHASE_CREATED",contact_id: "uuid-1",venue:

      { id: "venue-uuid-1", name: "Some Venue" }

      , // presenttransaction:

      { ... },RewardEvents: { total_award: 3.00, ... },ContactProfiles: { profile: { ... }

      }}}}
      Actual Result:
       
       
      js
       {{{type: "PURCHASE_CREATED",contact_id: "uuid-1",// venue is completely absent — no error throwntransaction:

      { ... },RewardEvents: { total_award: 3.00, ... },ContactProfiles: { profile: { ... }

      }}}}

      Additional Observations:

      • The bug is silent — no error or warning is produced
      • venue is confirmed present in the document immediately before $project (verified by removing $project from the pipeline)
      • Replacing $project with $unset (inverse approach) correctly returns venue
      • The bug appears to be triggered by the combination of:
        • type field existing at root level, inside RewardEvents, inside ContactProfiles, AND as a nested object inside a products array element
        • Projecting nested lookup fields using dot notation (e.g. "RewardEvents.total_award": 1) in the same $project as root-level fields
      • Tried all of the following without success: explicit "$venue" expression syntax, $addFields before $project, removing conflicting type fields via $$REMOVE before projecting

      Workaround: Use $unset to remove unwanted fields instead of $project to include desired fields.

      Show
      js  {{// Collection: Events (root document structure){type: "PURCHASE_CREATED", // stringcontact_id: "uuid-1",transaction: { id: "txn-uuid-1" } ,venue: { // THIS FIELD GETS DROPPEDid: "venue-uuid-1",name: "Some Venue"} ,products: [{type: { // nested object type — triggers the bugid: "product-type-uuid",name: "Generic Products"}}] }// Collection: RewardEvents {id: "txn-uuid-1", // joins on transaction.idtype: "REWARD_EVENT", // string — conflict with root typetotal_award: Decimal128("3.00"),total_spend: Decimal128("0.00"),total_open_spend: Decimal128("0.00"),total_commerce_spend: Decimal128("0.00"),total_crm_wallet_spend: Decimal128("0.00")} // Collection: ContactProfiles{contact_id: "uuid-1", // joins on contact_idtype: "CONTACT_UPDATED", // string — conflict with root typeprofile: {id: "uuid-1",name: "Some Name",type: "PERSON" // nested string type}}}} Pipeline:     js  {{[{ $match: { "transaction.id": "txn-uuid-1" } },{$lookup: {from: "RewardEvents",localField: "transaction.id",foreignField: "id",as: "RewardEvents"}}, { $unwind: "$RewardEvents" } ,{$lookup: {from: "ContactProfiles",localField: "contact_id",foreignField: "contact_id",as: "ContactProfiles"}}, { $unwind: "$ContactProfiles" } ,{$project: {type: 1,contact_id: 1,merchant: 1,submited_date: 1,transaction: 1,venue: 1, // silently dropped"RewardEvents.total_award": 1,"RewardEvents.total_spend": 1,"RewardEvents.total_open_spend": 1,"RewardEvents.total_commerce_spend": 1,"RewardEvents.total_crm_wallet_spend": 1,"ContactProfiles.profile": 1,_id: 0}}]}} Expected Result:     js  {{{type: "PURCHASE_CREATED",contact_id: "uuid-1",venue: { id: "venue-uuid-1", name: "Some Venue" } , // presenttransaction: { ... },RewardEvents: { total_award: 3.00, ... },ContactProfiles: { profile: { ... } }}}} Actual Result:     js  {{{type: "PURCHASE_CREATED",contact_id: "uuid-1",// venue is completely absent — no error throwntransaction: { ... },RewardEvents: { total_award: 3.00, ... },ContactProfiles: { profile: { ... } }}}} Additional Observations: The bug is silent — no error or warning is produced venue is confirmed present in the document immediately before $project (verified by removing $project from the pipeline) Replacing $project with $unset (inverse approach) correctly returns venue The bug appears to be triggered by the combination of: type field existing at root level, inside RewardEvents , inside ContactProfiles , AND as a nested object inside a products array element Projecting nested lookup fields using dot notation (e.g. "RewardEvents.total_award": 1 ) in the same $project as root-level fields Tried all of the following without success: explicit "$venue" expression syntax, $addFields before $project , removing conflicting type fields via $$REMOVE before projecting Workaround: Use $unset to remove unwanted fields instead of $project to include desired fields.
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When running an aggregation pipeline that performs two $lookup + $unwind stages, followed by a $project, root-level fields are silently dropped from the output. No error is thrown. The issue is triggered specifically when:

      1. The root document has a type field (string)
      2. One or both looked-up collections also have a type field (string or object)
      3. One of the root documents contains a products array where each element has a nested type field that is an object (not a string)

      The field venue (and potentially others) is consistently dropped from the $project output despite being present in the document at that stage, confirmed by running the same pipeline without the $project stage.

            Assignee:
            Unassigned
            Reporter:
            Marios Ioannou
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: