|
OK, thanks, Eric!
|
|
I recommend calling validate() then. It really only checks if BSON String-type elements are correct, and shouldn't have much of an effect on performance.
|
|
Unfortunately, no: iterator.next(true) gives me "Segmentation fault", but iterator.next().validate() works well and throws exception while checking invalid data.
|
|
Would calling next(true) work for you? This would achieve the same checking, and it's already present in the server.
|
|
Yes, I think this method will help us. Thanks!
|
|
I see. We could add a nextSafe() to BSONObjIterator which does the checking in the proposed patch, and then you could use that in your code. Would that work?
|
|
We've got it from mongo server. Maybe it was some specific network error, dunno.
validate() can help but I need to do it every time with every BSONObject. This is a performance penalty too, unfortunately.
|
|
Hi. Recently I've had some time to think about what this check will do.
For every routine call to next() when iterating over a BSONObject, the proposed patch adds a call to strnlen() over the next BSONElement. For large elements or objects with many elements, this strnlen scanning could add a material performance penalty. I'm not sure that adding the check to next() is the right way to fix the crashing issue.
For the crashes you experienced, from where did the bad BSON data originate? The test provided in this ticket hardcodes BSON values in a way that doesn't represent real-world use. Perhaps just running validate() on known bad data will be sufficient to avoid server crashes?
|
|
Great, thanks!
|
|
Thanks. We're in code freeze for the moment but as soon as 2.1 is branched, I can submit to head and then consider backporting to 2.0.7.
|
|
Done.
|
|
Thanks, I'd like to incorporate this patch into the mongodb source. Can you first sign the contributor agreement here: http://www.10gen.com/contributor
|
|
Hi Eric!
I`ve tested this issue again using source code directly from the git repository (git clone https://github.com/mongodb/mongo.git). Unfortunately, results are the same:
Program received signal SIGSEGV, Segmentation fault.
0x08049ae4 in mongo::BSONElement::type (this=0xbffff27c) at /home/mpugachev/work/mongodb/src/mongo/bson/bsonelement.h:110
110 BSONType type() const
{ return (BSONType) *reinterpret_cast< const signed char * >(data); }
(gdb) bt
#0 0x08049ae4 in mongo::BSONElement::type (this=0xbffff27c) at /home/mpugachev/work/mongodb/src/mongo/bson/bsonelement.h:110
#1 0x08049afd in mongo::BSONElement::eoo (this=0xbffff27c) at /home/mpugachev/work/mongodb/src/mongo/bson/bsonelement.h:126
#2 0x0804a025 in mongo::BSONElement::BSONElement (this=0xbffff27c, d=0x5e50142 <Address 0x5e50142 out of bounds>) at /home/mpugachev/work/mongodb/src/mongo/bson/bsonelement.h:448
#3 0x0804a403 in mongo::BSONObjIterator::next (this=0xbffff288) at /home/mpugachev/work/mongodb/src/mongo/bson/bsonobjiterator.h:70
#4 0x0804a6ff in mongo::BSONObj::getField (this=0xbffff2f8, name=...) at /home/mpugachev/work/mongodb/src/mongo/bson/bson-inl.h:266
#5 0x0804a1a7 in mongo::BSONObj::hasField (this=0xbffff2f8, name=0x804d0d8 "$err") at /home/mpugachev/work/mongodb/src/mongo/bson/bsonobj.h:227
#6 0x08049193 in main (argc=1, argv=0xbffff3c4) at crash.cpp:9
In bsonfix-nightly.patch you can find updated patch.
|
|
Hi Pugachev,
The latest sources are not for version 2.0.6; they are for candidate version 2.1.2, which can be found in the Nightly development builds. I can't apply patches to older version code such as version 2.0.6. I need to apply them to the latest code and then backport it. I believe the latest code already does more BSON validation. If someone can show the latest code hitting this same issue, I can fix it and backport it for release in 2.0.7.
|
|
Hi all!
I`ve retested this issue and got similar result with the latest sources. My test environment:
Linux iceland 2.6.38-15-generic-pae #60-Ubuntu SMP Tue May 22 11:48:17 UTC 2012 i686 i686 i386 GNU/Linux
db version v2.0.6, pdfile version 4.5
MongoDB shell version: 2.0.6
Stacktrace:
Program received signal SIGSEGV, Segmentation fault.
0x08049f4a in mongo::BSONElement::type (this=0xbffff27c) at /home/mpugachev/Downloads/mongodb-src-r2.0.6/bson/bsonelement.h:107
107 BSONType type() const
{ return (BSONType) *data; }
(gdb) bt
#0 0x08049f4a in mongo::BSONElement::type (this=0xbffff27c) at /home/mpugachev/Downloads/mongodb-src-r2.0.6/bson/bsonelement.h:107
#1 0x08049f63 in mongo::BSONElement::eoo (this=0xbffff27c) at /home/mpugachev/Downloads/mongodb-src-r2.0.6/bson/bsonelement.h:123
#2 0x0804a3f9 in mongo::BSONElement::BSONElement (this=0xbffff27c, d=0x5e51162 <Address 0x5e51162 out of bounds>) at /home/mpugachev/Downloads/mongodb-src-r2.0.6/bson/bsonelement.h:405
#3 0x0804a972 in mongo::BSONObjIterator::next (this=0xbffff288) at /home/mpugachev/Downloads/mongodb-src-r2.0.6/bson/bsonobjiterator.h:70
#4 0x0804ad57 in mongo::BSONObj::getField (this=0xbffff2f8, name=...) at /home/mpugachev/Downloads/mongodb-src-r2.0.6/bson/bson-inl.h:267
#5 0x0804a5f3 in mongo::BSONObj::hasField (this=0xbffff2f8, name=0x804d4c2 "$err") at /home/mpugachev/Downloads/mongodb-src-r2.0.6/bson/bsonobj.h:221
#6 0x08049243 in main (argc=1, argv=0xbffff3c4) at crash.cpp:9
After debugging I understand that the error was in size calculation logic (BSONElement::valuestrsize() method). To be more precise, the error was in data itself (provided in crash.cpp, we`ve got it from mongod) and this method just returned a garbage. So, the question is why mongod give us corrupted data. Maybe a network error or something, dunno.
Sergey`s patch helps to catch this event.
|
|
Hi Sergey,
Can you try the same test with crash.cpp using a Nightly build? Paste in the backtrace of the crash as you did with version 2.0 above. Since I am unable to reproduce the issue here, I'm hoping the further information will give me enough confidence that your suggested patch will fix the issue.
|
|
binary file
|
|
I don't understand why the stack trace keeps changing for you. It says that line 9 is now crashing.
The core file is not useful without the binary that created it.
|
|
core dump from crash.cpp
|
|
I added mu core file
I repeated again bt:
#0 0x08049c48 in mongo::BSONElement::type (this=0xbfdd40bc) at /home/skablov/mongodb-src-r2.0.4/bson/bsonelement.h:107
107 BSONType type() const
{ return (BSONType) *data; }
(gdb) bt
#0 0x08049c48 in mongo::BSONElement::type (this=0xbfdd40bc) at /home/skablov/mongodb-src-r2.0.4/bson/bsonelement.h:107
#1 0x08049c61 in mongo::BSONElement::eoo (this=0xbfdd40bc) at /home/skablov/mongodb-src-r2.0.4/bson/bsonelement.h:123
#2 0x0804a0c3 in mongo::BSONElement::BSONElement (this=0xbfdd40bc, d=0x5e50122 <Address 0x5e50122 out of bounds>) at /home/skablov/mongodb-src-r2.0.4/bson/bsonelement.h:403
#3 0x0804a507 in mongo::BSONObjIterator::next (this=0xbfdd40c8) at /home/skablov/mongodb-src-r2.0.4/bson/bsonobjiterator.h:68
#4 0x0804a83f in mongo::BSONObj::getField (this=0xbfdd4138, name=...) at /home/skablov/mongodb-src-r2.0.4/bson/bson-inl.h:267
#5 0x0804a27b in mongo::BSONObj::hasField (this=0xbfdd4138, name=0x804cd7c "$err") at /home/skablov/mongodb-src-r2.0.4/bson/bsonobj.h:221
#6 0x08049073 in main (argc=1, argv=0xbfdd4204) at crash1.cpp:9
(gdb) f 6
#6 0x08049073 in main (argc=1, argv=0xbfdd4204) at crash1.cpp:9
9 if ( bson.hasField("$err") ) {
(gdb) l
4
5 int main(int argc, const char* argv[])
6 {
7 ::mongo::BSONObj bson(reinterpret_cast<const char*>(buf));
8 printf("%d\n", bson.objsize());
9 if ( bson.hasField("$err") )
{
10 printf("Has Field\n");
11 }
12 return 0;
13 }
(gdb)
|
#3 0x0804902b in main (argc=1, argv=0xbf9ca594) at crash.cpp:6
|
When I look at your crash.cpp attachment, line 6 is an open brace. Was your stack trace generated using the same crash.cpp?
|
|
You are right I have compiled with wrong header, I' ve recompiled and now I've got the following backtrace
#0 0x0804a92d in mongo::BSONObj::toString (this=0x804cd52, isArray=false, full=255) at /home/.../mongodb-src-r2.0.4/bson/bson-inl.h:444
444 if ( isEmpty() ) return "{}";
(gdb) bt
#0 0x0804a92d in mongo::BSONObj::toString (this=0x804cd52, isArray=false, full=255) at /home/.../mongodb-src-r2.0.4/bson/bson-inl.h:444
#1 0x0804ab7b in mongo::BSONObj::toString (this=0xbf9ca48c, s=..., isArray=false, full=255) at /home/.../mongodb-src-r2.0.4/bson/bson-inl.h:462
#2 0x0804a52b in mongo::BSONElement::embeddedObject (this=0xb766aff4) at /home/.../mongodb-src-r2.0.4/bson/bson-inl.h:183
#3 0x0804902b in main (argc=1, argv=0xbf9ca594) at crash.cpp:6
(gdb) f 0
#0 0x0804a92d in mongo::BSONObj::toString (this=0x804cd52, isArray=false, full=255) at /home/.../mongodb-src-r2.0.4/bson/bson-inl.h:444
444 if ( isEmpty() ) return "{}";
(gdb) l
439 s.insert( it.next() );
440 return s;
441 }
442
443 inline string BSONObj::toString( bool isArray, bool full ) const {
444 if ( isEmpty() ) return "{}";
445 StringBuilder s;
446 toString(s, isArray, full);
447 return s.str();
448 }
(gdb)
|
|
Please ensure that the headers you are using match the libraries you are linking. In the stack trace I see a reference to bsoninlines.h, which hasn't existed since October 2010. Are you sure you're using headers from 2.0.4?
|
|
#0 0x0804a92d in mongo::BSONObjIterator::more (this=0xbf9ca458) at /usr/include/mongo/bson/../bson/bsonobjiterator.h:55
#1 0x0804ab7b in mongo::BSONObj::getField (this=0xbf9ca4c4, name=...) at /usr/include/mongo/bson/../bson/bsoninlines.h:81
#2 0x0804a52b in mongo::BSONObj::hasField (this=0xbf9ca4c4, name=0x804cd52 "$err") at /usr/include/mongo/bson/../bson/bsonobj.h:154
#3 0x0804902b in main (argc=1, argv=0xbf9ca594) at crash.cpp:9
|
|
Can you send the stack trace you yet?
|
|
Hello, Please tell me what version should I use not to get core dump
I compiled (g++ -o crash crash.cpp -I/home/skablov/mongodb-src-r2.0.4) crash.cpp with mongodb-src-r2.0.4 mongodb-mongo-r2.1.0-1582-g5d3433e.tar.gz mongodb-mongo-r2.0.0-rc2-1631-g524af38.tar.gz
In all cases I get Segmentation fault.
Please tell me where the core of the problem is
or may be I have done something in the wrong way.
|
|
Hi Sergey:
Thank you for your submission. I tried out crash.cpp but it just prints "4102" for me. Even when I run it with Valgrind, it doesn't detect any memory corruption errors. I am running this on Fedora Linux but I don't think that should make a difference.
Can you try running crash.cpp again on the code pulled from the head of master? If you can still reproduce the crash, I can look into it further. Thanks!
|
Generated at Thu Feb 08 03:06:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.