[GODRIVER-1818] BSON encoding, passing single string to $addToSet field Created: 04/Jan/21  Updated: 27/Oct/23  Resolved: 19/Jan/21

Status: Closed
Project: Go Driver
Component/s: BSON
Affects Version/s: 1.4.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andrew Hodel Assignee: Kevin Albertson
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 4.14


Attachments: File godriver1818_test.go    
Issue Links:
Duplicate
is duplicated by GODRIVER-1820 CLONE - BSON encoding, passing single... Closed
is duplicated by GODRIVER-1821 Different with $addToSet in NodeJS an... Closed
Related
related to GODRIVER-1819 BSON encoding, passing single string ... Closed

 Description   

This fails:

 

 
update := bson.D{
 
                                 {"$set", f},
 
                                 {"$addToSet", bson.E{"outsideIp", "string"}},
 
}
 
 
 
updateResult, err := collection.UpdateOne(context.TODO(), filter, update)

 

This works:

 

 
update := bson.D{
 
                                 {"$set", f},
 
}
 
 
 
updateResult, err := collection.UpdateOne(context.TODO(), filter, update)

 

 

Here is the error message:

 

 
multiple write errors: [\{write errors: [{Cannot apply $addToSet to non-array field. Field named 'key' has non-array type string}]}, \{<nil>}]

 

I only want to use $addToSet with a single string, as documented in the mongodb docs:

 

 
db.inventory.update( { _id: 1 }, { $addToSet: { tags: "accessories" } } )

 

What's going on here?



 Comments   
Comment by Kevin Albertson [ 19/Jan/21 ]

Closing since this is not a Go driver bug or feature request.

Comment by Kevin Albertson [ 19/Jan/21 ]

Hi andrewhodel@gmail.com,

Changing how $addToSet works with non-array values would require a change to how the server interprets $addToSet. Drivers to not modify update operators. Feel free to file a SERVER ticket describing the desired behavior if you'd like to pursue that change server-side.

As a tip, it should be possible to get the desired behavior using aggregation updates. Here is an example using the shell which adds the value "B":

db.test.drop()
db.test.insert({ _id: 0, outsideIp: null })
db.test.insert({ _id: 1, outsideIp: ["A"] })
db.test.insert({ _id: 2, outsideIp: ["B", "C"] })
 
// Desired behavior:
// Add "B" to the outsideIp array.
// Overwrite null value with ["B"].
// If "B" is already in the array, ignore it.
db.test.updateMany({}, [
    {
        $set: {
            outsideIp: {
                $cond: {
                    if: { $eq: ["$outsideIp", null] },
                    then: ["B"],
                    else: {
                        $setUnion: ["$outsideIp", ["B"]]
                    }
                }
            }
        }
    }
])
 
// Resulting documents:
// { "_id" : 0, "outsideIp" : [ "B" ] }
// { "_id" : 1, "outsideIp" : [ "A", "B" ] }
// { "_id" : 2, "outsideIp" : [ "B", "C" ] }

Comment by Andrew Hodel [ 07/Jan/21 ]

That is fundamentally what JSON is, JSON isn't the TCP/IP stack you know.

Comment by Andrew Hodel [ 07/Jan/21 ]

Also, the $addToSet will replace an array of objects with an array of strings then re-replace an array of strings with an array of arrays so your logic doesn't adhere between levels within the same thing (JSON).

 

That means for sure something should change.

Comment by Andrew Hodel [ 06/Jan/21 ]

I somewhat agree but truly think it is just not wanting to rewrite everything to support what has been standard for many many years.

 

I would suggest rethinking it in terms of what the industry will be capable of with it being done and without it being done.

Comment by Kevin Albertson [ 06/Jan/21 ]

Hi andrewhodel@gmail.com, I agree with your sentiment about consistency between drivers. MongoDB drivers are on the path to more consistent behavior, as drivers continue to adhere to common specifications.

Drivers should not modify the update document sent to the server. They pass along what you pass. If you were experiencing different behavior when running the same update using $addToSet with the NodeJS driver, that could indicate a bug in the NodeJS driver. It should not be changing the update document specified.

I tried the following with the NodeJS driver 3.6.3:

const { MongoClient } = require("mongodb");
const uri = "mongodb://localhost";
const client = new MongoClient(uri, { useUnifiedTopology: true });
async function run() {
try {
await client.connect();
const database = client.db('test');
const collection = database.collection('coll');
await collection.drop();
// Insert { "remoteIp": "a" }.
await collection.insertOne ({"_id": 0, "remoteIp": "a"});
await collection.updateOne ({"_id": 0}, {"$addToSet": {"remoteIp": "b"}});
} finally {
// Ensures that the client will close when you finish/error
await client.close();
}
}
run().catch(console.dir);

Similar to the Go driver, this results in a server error (a similar error if "remoteIp" is null as well):

MongoError: Cannot apply $addToSet to non-array field. Field named 'remoteIp' has non-array type string

If you have a reproducible NodeJS script that shows different behavior, please include it.

Changing the behavior of $addToSet when the stored value is a non-array / null would require the server to change handling of $addToSet. Feel free to file a feature request ticket on the SERVER project describing the desired behavior.

Note, I am closing GODRIVER-1819, GODRIVER-1820 (which appear to be clones) and GODRIVER-1821 (since the conversation has continued here) as duplicates.

Comment by Andrew Hodel [ 05/Jan/21 ]

Field is Not an Array

If you use $addToSet on a field that is not an array, the operation will fail. For example, consider a document in a collection foo that contains a non-array field colors.

 

So for sure it is just that writing a null value to a JSON string (which is javascript object notation) acts different for JSON in javascript (where JSON came from) and in the JSON that mongodb accepts, but the documentation does describe it.

 

What do you think, should mongodb own the JSON spec?

 

Also I commented on this, couldn't link your name though.  https://jira.mongodb.org/browse/GODRIVER-1821

 

It is difficult to understand how I pass JSON to the mongodb shell and it doesn't adhere to JSON.  Surely you guys accept that.

 

JSON (JavaScript Object Notation)

Comment by Andrew Hodel [ 05/Jan/21 ]

Don't you think we should be talking on this issue - https://jira.mongodb.org/browse/GODRIVER-1821

 

Because then the search engines can properly index the information if someone has the same problem?

Comment by Andrew Hodel [ 05/Jan/21 ]

Well without using the latest versions, something like automatically
launched per the latest git of the db with a web shell, we won’t have much
to go on and really if it is by the documentation it’s ok.

I am using mongodb 4.4 and I really think the nodejs driver was overwriting
a string in an existing document update when using $addToSet

I can go through it all tomorrow and write it down cleanly so you know
because I think it is important that they act the same (should be standardized from what happens in the mongo shell).

Regardless, there is also that beyond that, so many people think of storing
null rather than using $unset to clear something and mongo BSON supports
null values for fields so maybe that might need some more thought.

Recently (months) working in C, Node and Go for what it’s worth.

Andrew

On Mon, Jan 4, 2021 at 5:35 PM Kevin Albertson (Jira) <jira@mongodb.org>

Comment by Andrew Hodel [ 04/Jan/21 ]

https://jira.mongodb.org/projects/GODRIVER/issues/GODRIVER-1821?filter=allopenissues

Comment by Andrew Hodel [ 04/Jan/21 ]

No no, you don't have to look into this.

 

You were right, but look at the other issue about the differences between NodeJS and Go regarding the mongodb driver.

Comment by Kevin Albertson [ 04/Jan/21 ]

Hi andrewhodel@gmail.com, I apologize for the miscommunication. What I meant was that the error indicates the value of the "outsideIp" stored in the document being updated is a string.

Here is an example in the shell. $addToSet works as expected when "outsideIp" is an array of strings:

> db.test.coll.insert({ "_id" : 0, "outsideIp" : [ "a", "b" ] })
> db.test.coll.update({ "_id": 0}, {"$addToSet": { "outsideIp": "c" }})
WriteResult({ "nInserted" : 1 })

But, if "outsideIp" is stored as a string, I receive the same error when attempting to update:

> db.test.coll.insert({ "_id" : 1, "outsideIp" : "d" })
> db.test.coll.update({ "_id": 1}, {"$addToSet": { "outsideIp": "c" }})
WriteResult({
	"nMatched" : 0,
	"nUpserted" : 0,
	"nModified" : 0,
	"writeError" : {
		"code" : 2,
		"errmsg" : "Cannot apply $addToSet to non-array field. Field named 'outsideIp' has non-array type string"
	}
})

I am not sure what is causing the difference you are observing in behavior between the NodeJS driver and Go driver. If possible, can you include the document containing "outsideIp", or a fully reproducible example?

Comment by Andrew Hodel [ 04/Jan/21 ]

This is a problem between the nodejs driver and the go driver, why would they be different when the company mongo is making them?

 

The nodejs driver will replace the field in the document, the go driver will not.

Comment by Andrew Hodel [ 04/Jan/21 ]

Don't you realize that makes no sense, it has no reason?

Comment by Andrew Hodel [ 04/Jan/21 ]

Why would you tell me to pass a string, then when I run the code that passes a string and it returns an error about needing an array to use an array?  You would have to first explain why you said to use a string!

Comment by Andrew Hodel [ 04/Jan/21 ]

Permalink
 kevin.albertson added a comment - Jan 04 2021 10:52:55 PM GMT+0000
The error returned from the server indicates that "outsidelp" is a string. It should be an array:

Cannot apply $addToSet to non-array field. Field named 'outsideIp' has non-array type string

https://docs.mongodb.com/manual/reference/operator/update/addToSet/#add-to-array

 

 

You must be joking, I am using your code that specifies it as a string!

Comment by Kevin Albertson [ 04/Jan/21 ]

The error returned from the server indicates that "outsidelp" is a string. It should be an array:

Cannot apply $addToSet to non-array field. Field named 'outsideIp' has non-array type string

https://docs.mongodb.com/manual/reference/operator/update/addToSet/#add-to-array

Comment by Andrew Hodel [ 04/Jan/21 ]

r from r.RemoteAddr is https://golang.org/src/net/http/request.go?s=3211:11861#L93

Comment by Andrew Hodel [ 04/Jan/21 ]

This is a bug, it still fails with the same error:

 

error in wss config request when updating host: multiple write errors: [{write errors: [

{Cannot apply $addToSet to non-array field. Field named 'outsideIp' has non-array type string}

]}, {<nil>}]

 

                        update := bson.D{

                               

{"$set", f}

,

                                {"$addToSet", bson.D{{"outsideIp", r.RemoteAddr}}},

                        }

 

                        updateResult, err := collection.UpdateOne(context.TODO(), filter, update)

                        if err != nil

{                                 _ = updateResult                                 fmt.Printf("\n\nerror in wss config request when updating host: %s\n", err)                         }
Comment by Kevin Albertson [ 04/Jan/21 ]

Hi andrewhodel@gmail.com, thank you for the report!

In your first example, I believe bson.D should be used in place of bson.E, with an additional set of braces, like so:

update := bson.D{
    {"$set", f},
    {"$addToSet", bson.D{{"outsideIp", "string"}}},
}
 
updateResult, err := collection.UpdateOne(context.TODO(), filter, update)

This assumes that "outsideIp" is an array in the document being updated. The value of "$addToSet" must be a document.

I've attached godriver1818_test.go with a compilable example.

Since this isn't a bug or feature request we're closing the issue. For further assistance, please create a post in our community forum here. If you believe this is incorrect, please add a comment.

Comment by Andrew Hodel [ 04/Jan/21 ]

Here's another example:

 

This works:

 

2124                                         // add cmd to host document using $addToSet with the ws_id of the ui client and the uuidv4 specifying the unique command

2125                                         update := bson.D{

2126                                                 {"$addToSet", bson.D{

2127                                                         {"cmds", bson.D{

2128                                                                

{"cmd", msg_json.Cmd}

,

2129                                                                

{"ws_id", msg_json.Ws_Id}

,

2130                                                                

{"uuidv4", msg_json.UuidV4}

,

2131                                                         }},

2132                                                 }},

2133                                         }

 

 

 

But this doesn't:

 

2135                         update := bson.D{

2136                                

{"$set", f}

,

2137                                 {"$addToSet", bson.D{"outsideIp": r.RemoteAddr}},

2138                         }

 

So how do I add a single string to an array using $addToSet rather than an array of objects?

 

Generated at Thu Feb 08 08:37:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.