[SERVER-57334] Create desugaring namespace for DocumentSourceDensify Created: 01/Jun/21  Updated: 29/Oct/23  Resolved: 18/Aug/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Joe Kanaan Assignee: Sam Mercier
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-57332 Create skeleton InternalDocumentSourc... Closed
is depended on by SERVER-57339 Add support for partitionByFields in ... Closed
is depended on by SERVER-57336 Implement densification for numeric v... Closed
is depended on by SERVER-57337 Implement densification for date valu... Closed
is depended on by SERVER-57341 Implement densification for numeric v... Closed
is depended on by SERVER-57344 Add support for partitioning and rang... Closed
Duplicate
is duplicated by SERVER-57339 Add support for partitionByFields in ... Closed
Problem/Incident
causes SERVER-59763 remove subtle typo that happens to co... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query Optimization 2021-06-28, Query Optimization 2021-07-12, Query Optimization 2021-07-26, Query Optimization 2021-08-09, QO 2021-08-23
Participants:
Linked BF Score: 159

 Description   

Implement a namespace that desugars a DocumentSourceDensify stage to a sort stage and an InternalDocumentSourceDensify. Only needs to support range: full with numeric steps (in addition to the "field" required argument), and assert on as of yet unsupported options. This ticket should include serializing InternalDocumentSourceDensify back to something that resembles the original $densify command. SERVER-53397 is a good example of this. After this ticket DocumentSourceDensify should be accessible from JSTests.

Note that we should parse the DocumentSource using IDL.



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Sam Mercier [ 03/Sep/21 ]

billy.donahue my bad that's totally a typo. not sure how that slipped in. Thanks for fixing this!

Comment by Billy Donahue [ 03/Sep/21 ]

I'm getting a fatal warning on my virtual workstation from this change.

/home/ubuntu/mongo-dev/mongo/python3-venv/bin/python3 buildscripts/scons.py VARIANT_DIR=dynamic_gcc MONGO_GIT_HASH=unknown LLVM_SYMBOLIZER=/opt/mongodbtoolchain/v3/bin/llvm-symbolizer --opt=on --dbg=on CCACHE=ccache ICECC=icecc --ninja=enabled VERBOSE=1 --link-model=dynamic --variables-files=etc/scons/mongodbtoolchain_stable_gcc.vars --modules=enterprise generate-ninja

[4/2493] export PATH='/opt/mongodbtoolchain/v3/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin';export CCACHE_NOCPP2='1';export ICECC_CARET_WORKAROUND='0';export CCACHE_PREFIX='/home/ubuntu/mongo-dev/mongo/build/scons/icecream/run-icecc.sh';/usr/bin/ccache /opt/mongodbtoolchain/v3/bin/g++ @build/dynamic_gcc/mongo/db/pipeline/document_source_densify.dyn.o.rsp
FAILED: build/dynamic_gcc/mongo/db/pipeline/document_source_densify.dyn.o
export PATH='/opt/mongodbtoolchain/v3/bin:/usr/local/bin:/opt/bin:/bin:/usr/bin';export CCACHE_NOCPP2='1';export ICECC_CARET_WORKAROUND='0';export CCACHE_PREFIX='/home/ubuntu/mongo-dev/mongo/build/scons/icecream/run-icecc.sh';/usr/bin/ccache /opt/mongodbtoolchain/v3/bin/g++ @build/dynamic_gcc/mongo/db/pipeline/document_source_densify.dyn.o.rsp
In file included from /usr/include/string.h:494,
                 from /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/cstring:42,
                 from src/mongo/base/string_data.h:33,
                 from src/mongo/db/exec/document_value/value.h:33,
                 from src/mongo/db/pipeline/document_source_densify.h:32,
                 from src/mongo/db/pipeline/document_source_densify.cpp:30:
In function 'void* memcpy(void*, const void*, size_t)',
    inlined from 'mongo::ValueStorage::ValueStorage(const mongo::ValueStorage&)' at src/mongo/db/exec/document_value/value_internal.h:169:15,
    inlined from 'mongo::Value::Value(const mongo::Value&)' at src/mongo/db/exec/document_value/value.h:133:5,
    inlined from 'constexpr mpark::detail::alt<Index, T>::alt(mpark::in_place_t, Args&& ...) [with Args = {mongo::Value&}; long unsigned int Index = 0; T = mongo::Value]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:976:46,
    inlined from 'constexpr mpark::detail::recursive_union<(mpark::detail::Trait)1, Index, T, Ts ...>::recursive_union(mpark::in_place_index_t<0>, Args&& ...) [with Args = {mongo::Value&}; long unsigned int Index = 0; T = mongo::Value; Ts = {mongo::Date_t}]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1025:364,
    inlined from 'constexpr mpark::detail::base<DestructibleTrait, Ts>::base(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}; mpark::detail::Trait DestructibleTrait = (mpark::detail::Trait)1; Ts = {mongo::Value, mongo::Date_t}]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1043:21,
    inlined from 'constexpr mpark::detail::destructor<mpark::detail::traits<mongo::Value, mongo::Date_t>, (mpark::detail::Trait)1>::destructor(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1126:188,
    inlined from 'constexpr mpark::detail::constructor<mpark::detail::traits<mongo::Value, mongo::Date_t> >::constructor(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1148:20,
    inlined from 'constexpr mpark::detail::move_constructor<mpark::detail::traits<mongo::Value, mongo::Date_t>, (mpark::detail::Trait)1>::move_constructor(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1215:188,
    inlined from 'constexpr mpark::detail::copy_constructor<mpark::detail::traits<mongo::Value, mongo::Date_t>, (mpark::detail::Trait)1>::copy_constructor(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1253:198,
    inlined from 'constexpr mpark::detail::assignment<mpark::detail::traits<mongo::Value, mongo::Date_t> >::assignment(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1271:20,
    inlined from 'constexpr mpark::detail::move_assignment<mpark::detail::traits<mongo::Value, mongo::Date_t>, (mpark::detail::Trait)1>::move_assignment(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1373:185,
    inlined from 'constexpr mpark::detail::copy_assignment<mpark::detail::traits<mongo::Value, mongo::Date_t>, (mpark::detail::Trait)1>::copy_assignment(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1413:195,
    inlined from 'constexpr mpark::detail::impl<mongo::Value, mongo::Date_t>::impl(mpark::in_place_index_t<I>, Args&& ...) [with long unsigned int I = 0; Args = {mongo::Value&}][inherited from mpark::detail::base<(mpark::detail::Trait)1, mongo::Value, mongo::Date_t>]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1431:20,
    inlined from 'constexpr mpark::variant<Ts>::variant(Arg&&) [with Arg = mongo::Value&; Decayed = mongo::Value; typename std::enable_if<(! std::is_same<Decayed, mpark::variant<Ts> >::value), int>::type <anonymous> = 0; typename std::enable_if<(! mpark::detail::is_in_place_index<Decayed>::value), int>::type <anonymous> = 0; typename std::enable_if<(! mpark::detail::is_in_place_type<Decayed>::value), int>::type <anonymous> = 0; long unsigned int I = 0; T = mongo::Value; typename std::enable_if<std::is_constructible<T, Arg>::value, int>::type <anonymous> = 0; Ts = {mongo::Value, mongo::Date_t}]' at src/third_party/variant-1.4.0/include/mpark/variant.hpp:1579:62,
    inlined from 'void boost::optional_detail::optional_base<T>::construct(Expr&&, const void*) [with Expr = mongo::Value&; T = mpark::variant<mongo::Value, mongo::Date_t>]' at src/third_party/boost/boost/optional/optional.hpp:670:7,
    inlined from 'void boost::optional_detail::optional_base<T>::assign_expr(Expr&&, const ExprPtr*) [with Expr = mongo::Value&; ExprPtr = mongo::Value; T = mpark::variant<mongo::Value, mongo::Date_t>]' at src/third_party/boost/boost/optional/optional.hpp:367:12,
    inlined from 'typename boost::enable_if<boost::optional_detail::is_optional_val_init_candidate<T, Expr>, boost::optional<T>&>::type boost::optional<T>::operator=(Expr&&) [with Expr = mongo::Value&; T = mpark::variant<mongo::Value, mongo::Date_t>]' at src/third_party/boost/boost/optional/optional.hpp:1000:9,
    inlined from 'mongo::DocumentSource::GetNextResult mongo::DocumentSourceInternalDensify::handleNeedGenExplicit(mongo::Document, mongo::Value, NumericBounds)' at src/mongo/db/pipeline/document_source_densify.cpp:509:50:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:34:33: error: 'void* __builtin_memcpy(void*, const void*, long unsigned int)' accessing 16 bytes at offsets 328 and 328 overlaps 16 bytes at offset 328 [-Werror=restrict]
cc1plus: all warnings being treated as errors
ICECC[8011] 00:27:33: Compiled on 10.122.0.245

ValueStorage copy constructor is doing a memcpy of some memory block onto itself.
That's disallowed by memcpy, which demands __restrict (nonoverlapping) memory range arguments.

inlined from 'mongo::ValueStorage::ValueStorage(const mongo::ValueStorage&)' at src/mongo/db/exec/document_value/value_internal.h:169:15,

The top of the error stack is:

inlined from 'mongo::DocumentSource::GetNextResult mongo::DocumentSourceInternalDensify::handleNeedGenExplicit(mongo::Document, mongo::Value, NumericBounds)' at src/mongo/db/pipeline/document_source_densify

509             _current = stdx::get<Value>(*_current), _range.getStep();

So yeah, it looks like we're doing a variant get<Value> on the variant referred to by _current.

Actually it kind of looks like this was a total typo, as it is simply a restatement of the argument list that went into the addValues call on the previous line.

508             _current = addValues(stdx::get<Value>(*_current), _range.getStep());

Is that what happened? Line 509 doesn't really make much sense on its own. It's using a comma operator for no reason, etc... I'm pretty sure it is a corruption of line 508.

(I've given this problem its own ticket as SERVER-59763)

Comment by Githook User [ 19/Aug/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-57334 document_source_densify_test.cpp initializer waits for server parameters before setting feature flag
Branch: master
https://github.com/mongodb/mongo/commit/08a5d7c54876ada6d7ac42bcab54cf65be423001

Comment by Githook User [ 19/Aug/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-57334 fix mac os x compile
Branch: master
https://github.com/mongodb/mongo/commit/b34cdd55029c517c9ac28a091f5765c3f490171f

Comment by Githook User [ 18/Aug/21 ]

Author:

{'name': 'samontea', 'email': 'merciers.merciers@gmail.com', 'username': 'samontea'}

Message: SERVER-57334 Create desugaring namespace for DocumentSourceDensify
Branch: master
https://github.com/mongodb/mongo/commit/ca088c270c1b511c342a1366d4b296b3c98774f0

Generated at Thu Feb 08 05:41:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.