[JAVA-3228] Watch with project pipeline didn't work correctly. Created: 15/Mar/19  Updated: 27/Oct/23  Resolved: 04/May/19

Status: Closed
Project: Java Driver
Component/s: Change Streams
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Han Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
backported by SERVER-41559 Can not fetch changed array elements ... Backlog

 Description   

I am using mongo java drivers as below:

 

<dependency>
      <groupId>org.mongodb</groupId>
      <artifactId>mongo-java-driver</artifactId>
       <version>3.10.1</version>
    </dependency>
    <dependency>
      <groupId>org.mongodb</groupId>
      <artifactId>mongodb-driver-reactivestreams</artifactId>
      <version>1.11.0</version>
    </dependency>
 

When I define a simple pipeline as:

List<Bson> pipeline = Arrays.asList(
                Aggregates.project(Projections.fields(
                        Projections.computed("id", "$fullDocument._id")
                ))
        );

After my doing some CRUD operations, I always get these 'null' output:

ChangeStreamDocument{resumeToken={"_data": "825C8B65510000000129295A1004E2034BD33C7D40DE92CB94E139C92CB146645F696400645C8B65511B2DF1941FD846250004"}, namespace=null, fullDocument=null, documentKey=null, clusterTime=null, operationType=null, updateDescription=null}
ChangeStreamDocument{resumeToken={"_data": "825C8B65510000000229295A1004E2034BD33C7D40DE92CB94E139C92CB146645F696400645C8B65511B2DF1941FD846250004"}, namespace=null, fullDocument=null, documentKey=null, clusterTime=null, operationType=null, updateDescription=null}
ChangeStreamDocument{resumeToken={"_data": "825C8B65560000000129295A1004E2034BD33C7D40DE92CB94E139C92CB146645F696400645C8B65511B2DF1941FD846250004"}, namespace=null, fullDocument=null, documentKey=null, clusterTime=null, operationType=null, updateDescription=null}
ChangeStreamDocument{resumeToken={"_data": "825C8B65560000000229295A1004E2034BD33C7D40DE92CB94E139C92CB146645F696400645C8B65511B2DF1941FD846250004"}, namespace=null, fullDocument=null, documentKey=null, clusterTime=null, operationType=null, updateDescription=null}

Did not it should be like :

{
    "id": xxxxxxxxxx
}

Please correct me.

 

 



 Comments   
Comment by Han [ 06/Jun/19 ]

I have created one in the SERVER project.

Thanks, Ross.

Comment by Ross Lawley [ 05/Jun/19 ]

Hi jiehan2019,

The driver only returns what is provided by the server. You are welcome to introspect it and convert the data to the format as needed or alternatively create a feature request in the SERVER project.

Ross

Comment by Han [ 05/Jun/19 ]

Yang model instance is transformed into json and  then to document which is to be saved in MongoDB

Comment by Han [ 05/Jun/19 ]

Hi @Ross Lawley,

Here are outputs for corresponding steps, What I need is to manipulate the deleted array item, but it seems not workable.

1) First , here is the initial data in the database,

   there're 3 outer array items "test-0, test-1, test-2" with each has 3 inner array items "nest-test-0,nest-test-1,nest-test-3":

 

{"_id":"5cf7553cc6b365a5c72f2163","opendaylight-mdsal-binding-test:top":{"top-level-list":[{"name":"test-0","nested-list":[{"name":"nest-test-0","type":"nest-type-0"},{"name":"nest-test-1","type":"nest-type-1"},{"name":"nest-test-2","type":"nest-type-2"}],"simple":"simple-case"},{"name":"test-1","nested-list":[{"name":"nest-test-0","type":"nest-type-0"},{"name":"nest-test-1","type":"nest-type-1"},{"name":"nest-test-2","type":"nest-type-2"}],"simple":"simple-case"},{"name":"test-2","nested-list":[{"name":"nest-test-0","type":"nest-type-0"},{"name":"nest-test-1","type":"nest-type-1"},{"name":"nest-test-2","type":"nest-type-2"}],"simple":"simple-case"}]}}

2) Next , I pull one nested array item by calling 'collection.updateOne(...)' where 'Update' and 'UpdateOptions' like:

Update{fieldName='opendaylight-mdsal-binding-test:top.top-level-list.$[item0].nested-list', operator='$pull', value=Document{{name=nest-test-2}}}
UpdateOptions{upsert=true, bypassDocumentValidation=null, collation=null, arrayFilters=[And Filter{filters=[Filter{fieldName='item0.name', value=test-0}]}]}

It's clear above that I specify the '$[item0]' with 'Filter{fieldName='item0.name', value=test-0}'  to delete the inner array item 'nest-test-2' from outer array item 'test-0'.

3) After step 2), finally I recieve the change event:

ChangeStreamDocument{resumeToken={"_data": "825CF75C47000000022B022C0100296E5A10042A6A4D3FE1F64145A8D27037F0BA45BD46645F696400645CF75C47D4CE6723E929EA0B0004"}, namespace=configuration.urn:opendaylight:params:xml:ns:yang:mdsal:test:binding@2014-07-01, fullDocument=Document{{_id=5cf75c47d4ce6723e929ea0b, opendaylight-mdsal-binding-test:top=Document{{top-level-list=[Document{{name=test-0, nested-list=[Document{{name=nest-test-0, type=nest-type-0}}, Document{{name=nest-test-1, type=nest-type-1}}], simple=simple-case}}, Document{{name=test-1, nested-list=[Document{{name=nest-test-0, type=nest-type-0}}, Document{{name=nest-test-1, type=nest-type-1}}, Document{{name=nest-test-2, type=nest-type-2}}], simple=simple-case}}, Document{{name=test-2, nested-list=[Document{{name=nest-test-0, type=nest-type-0}}, Document{{name=nest-test-1, type=nest-type-1}}, Document{{name=nest-test-2, type=nest-type-2}}], simple=simple-case}}]}}}}, documentKey={"_id": {"$oid": "5cf75c47d4ce6723e929ea0b"}}, clusterTime=Timestamp{value=6698924430749335554, seconds=1559714887, inc=2}, operationType=OperationType{value='update'}, updateDescription=UpdateDescription{removedFields=[], updatedFields={"opendaylight-mdsal-binding-test:top.top-level-list.0.nested-list": [{"name": "nest-test-0", "type": "nest-type-0"}, {"name": "nest-test-1", "type": "nest-type-1"}]}}}

From the output we can see that it's an 'update' operation type and the 'updatedFields' is :

{"opendaylight-mdsal-binding-test:top.top-level-list.0.nested-list": [{"name": "nest-test-0", "type": "nest-type-0"}, {"name": "nest-test-1", "type": "nest-type-1"}]}}

Note 'updatedFields' above just provides data after '$pull', but I really need to see detail deleted data information "what I delete from where", an idealy output I would prefer is like kind of :

{"opendaylight-mdsal-binding-test:top.top-level-list.0.nested-list": [{
"array-filter": {"0": {"name": "test-0"}},
"pulled":{"name": "nest-test-2", "type": "nest-type-2"},
"current": {"name": "nest-test-0", "type": "nest-type-0"}, {"name": "nest-test-1", "type": "nest-type-1"}]}}
}

In this way , with 'array-filter' and 'pulled' fields, I could get the deleted data and notify listeners just like 'DataTreeChangeService' does:

https://github.com/opendaylight/alt-datastores/blob/master/mongodb/yongo/src/main/java/org/opendaylight/datastore/yongo/impl/YongoStream.java#L155

 

 

 

Comment by Ross Lawley [ 26/Apr/19 ]

Hi jiehan2019,

The ChangeStreamDocument is a POJO and not a flexible map like document. If you are going to manipulate the output of the Change Stream then please use the ChangeStreamIterable#withDocumentClass:

collection.watch(pipeline)
    .fullDocument(FullDocument.UPDATE_LOOKUP))
    .withDocumentClass(Document.class);

That will provide the change stream documents as provided by the server and should fix the issue you are seeing. Please let me know if that works for you.

Ross

Comment by Han [ 26/Apr/19 ]

Hi @Jeff Yemin 

Sorry for late comments when I fould this issue was closed:

yes, I called like 'collection.watch(pipeline).fullDocument(FullDocument.UPDATE_LOOKUP))',

and I have been working on my project which I need to adapt MongoDB stream to DataTreeChangeService:

https://wiki.opendaylight.org/view/Project_Proposals:Alt-datastores#DataTreeChangeService

But now it seems that I cannot resolve issues mentioned in the paragraph above correctly.

Comment by Jeffrey Yemin [ 29/Mar/19 ]

Hi jiehan2019

Do you mean to say that if you call it the way you indicate in your last comment, it works as expected?

Comment by Han [ 15/Mar/19 ]

call as:

collection.watch(pipeline)
.fullDocument(FullDocument.UPDATE_LOOKUP))

Generated at Thu Feb 08 08:59:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.