Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-775

Ability to specify union

XMLWordPrintableJSON

    • Type: Icon: Task Task
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Component/s: None
    • Labels:
      None
    • $i18n.getText("admin.common.words.hide")
      Key Status/Resolution FixVersion
      JAVA-3520 Duplicate
      CSHARP-2863 Duplicate
      $i18n.getText("admin.common.words.show")
      #scriptField, #scriptField *{ border: 1px solid black; } #scriptField{ border-collapse: collapse; } #scriptField td { text-align: center; /* Center-align text in table cells */ } #scriptField td.key { text-align: left; /* Left-align text in the Key column */ } #scriptField a { text-decoration: none; /* Remove underlines from links */ border: none; /* Remove border from links */ } /* Add green background color to cells with FixVersion */ #scriptField td.hasFixVersion { background-color: #00FF00; /* Green color code */ } /* Center-align the first row headers */ #scriptField th { text-align: center; } Key Status/Resolution FixVersion JAVA-3520 Duplicate CSHARP-2863 Duplicate

      Downstream Change Summary

      TBD

      Description of Linked Ticket

      Epic Summary

      Summary

      We will implement a new Agg stage, $union, that allows to merge results of n pipelines preserving duplicates. In order to enable merging data from multiple collections, we will also introduce an explicit stage to reference a collection, $collection.

      Motivation

      Union is a fundamental operation in relational algebra. We have several specific scenarios:

      • BIC connector for completeness with SQL.
      • TimeSeries scenario to combine data stored in per-period collections into one logical collection.
      • Combining collections in Data Lake, e.g. archival and recent data, data from different regions.

      For analytical scenarios, customers expect a complete set of fundamental operations. For example for Tableau, union and unpivot were top requested features after joins. In the future, we will be improving $lookup but delivering general and performant joins is a hard task. At the same time, union-like logic is already supported for operations that require merging results across shards in the backend.

      Documentation

      Scope Document
      Design Document

            Assignee:
            Unassigned Unassigned
            Reporter:
            backlog-server-pm Backlog - Core Eng Program Management Team
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: