[SERVER-1082] index on compount with array fails without array Created: 03/May/10  Updated: 12/Jul/16  Resolved: 04/May/10

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 1.4.3, 1.5.2

Type: Bug Priority: Major - P3
Reporter: Eliot Horowitz (Inactive) Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File log.error.gz    
Participants:

 Description   

see jstests/index_arr1.js



 Comments   
Comment by Eliot Horowitz (Inactive) [ 04/May/10 ]

There is a new official build up that should have this as well.

Comment by Narayan Newton [ 04/May/10 ]

I'm leaving this building indexes, but both my test case has passed and its farther along in index build than its been before.

Thanks Eliot.

-N

Comment by Narayan Newton [ 04/May/10 ]

Am currently fighting with boost, but I built a version of Mongod on an older boost and it looks like this does fix things. I'll confirm further with a newer boost shortly. The automated build seems to have failed on a replication unit test.

-N

Comment by Eliot Horowitz (Inactive) [ 04/May/10 ]

Think i fixed that case as well.
Pushed a build - will be about 90 minutes

http://buildbot.mongodb.org/waterfall?show=V1.4%20Linux%2064-bit

Comment by auto [ 04/May/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: another case for SERVER-1082
http://github.com/mongodb/mongo/commit/4e8cafb9e909fe9a71ba12fc1863251cfa7455b0

Comment by auto [ 04/May/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: another case for SERVER-1082
http://github.com/mongodb/mongo/commit/5d4d3c74f1b9e33fc5b0c6d8d3910eab572a7bfe

Comment by Narayan Newton [ 04/May/10 ]

Reproduced on 1.5.1:

> db.fields_current.node.ensureIndex(

{ status: 1, "@fields.ex_node_group.value": 1, "@fields.current.value": 1, "@fields.promoted_media.fid": 1 }

);
!e.eoo()

Tue May 4 18:41:08 connection accepted from 127.0.0.1:55704 #3
Tue May 4 18:42:25 building new index on

{ status: 1.0, @fields.ex_node_group.value: 1.0, @fields.current.value: 1.0, @fields.promoted_media.fid: 1.0 }

for examiner_migration.fields_current.node
Tue May 4 18:42:25 Buildindex examiner_migration.fields_current.node idxNo:1 { _id: ObjectId('4be0a2d196b388ebbbcbdb00'), ns: "examiner_migration.fields_current.node", key:

{ status: 1.0, @fields.ex_node_group.value: 1.0, @fields.current.value: 1.0, @fields.promoted_media.fid: 1.0 }

, name: "status_1_@fields.ex_node_group.value_1_@fields.current.value_1_@fields..." }
Tue May 4 18:42:26 examiner_migration.system.indexes Assertion failure !e.eoo() db/../bson/bsonobjbuilder.h 65
0x503dc6 0x50e334 0x4bcfcf 0x5834f0 0x58402a 0x653ccc 0x655b1a 0x65703c 0x658c25 0x63a79e 0x63fb1c 0x6ea3b2 0x6fd800 0x2aaaaacce617 0x2aaaab748c2d
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x503dc6]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x50e334]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo14BSONObjBuilder8appendAsERKNS_11BSONElementEPKc+0x14f) [0x4bcfcf]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZNK5mongo9IndexSpec8_getKeysESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjERSt3setIS9_NS_22BSONObjCmpDefaultOrderESaIS9_EE+0x530) [0x5834f0]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZNK5mongo9IndexSpec7getKeysERKNS_7BSONObjERSt3setIS1_NS_22BSONObjCmpDefaultOrderESaIS1_EE+0x2ea) [0x58402a]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x62c) [0x653ccc]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod [0x655b1a]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0xa4c) [0x65703c]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x55) [0x658c25]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x21e) [0x63a79e]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x179c) [0x63fb1c]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(_ZN5mongo10connThreadEv+0x232) [0x6ea3b2]
/usr/local/mongodb-linux-x86_64-2010-05-04/bin/mongod(thread_proxy+0x80) [0x6fd800]
/lib64/libpthread.so.0 [0x2aaaaacce617]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748c2d]
Tue May 4 18:42:26 Caught Assertion in insert , continuing
Tue May 4 18:42:26 insert examiner_migration.system.indexes exception assertion db/../bson/bsonobjbuilder.h:65 665ms

Tue May 4 18:12:25 Mongo DB : starting : pid = 11865 port = 29019 dbpath = data2 master = 0 slave = 0 64-bit
****
WARNING: This is development version of MongoDB. Not recommended for production.
****
Tue May 4 18:12:25 db version v1.5.2-pre-, pdfile version 4.5
Tue May 4 18:12:25 git version: ffc097e429cd58aea0063be168b91e229b4f9942
Tue May 4 18:12:25 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue May 4 18:12:25 waiting for connections on port 29019

Comment by Narayan Newton [ 04/May/10 ]

Attached is the section of our log containing this entire session, as some of the indexes built correctly and some didn't depending on the index. Rather than looking for the pattern myself and missing something, I figured I'd just give you the entire session.

-N

Comment by Karoly Negyesi [ 04/May/10 ]

After I updated to the nightly I started reindexing because my testing have shown that the upgrade did not help but reindexing did. I then left for the night with a little script that ran reindex on every collection. That apparently backfired.

Comment by Narayan Newton [ 04/May/10 ]

I'm afraid we maybe having some issues with this fix, after upgrading to the nightly release with these commits, we are seeing assertions on index build. The following is an except of this behavior:

Tue May 4 17:10:59 insert examiner_0428.system.indexes exception assertion db/jsobj.h:1196 247ms
Tue May 4 17:10:59 building new index on

{ status: 1, @fields.ex_node_group.value: 1, @fields.current.value: 1, @fields.promoted_media.fid: 1 }

for examiner_0428.fields_current.node
Tue May 4 17:10:59 Buildindex examiner_0428.fields_current.node idxNo:9 { _id: ObjId(4be08d638ead0ecf33210000), ns: "examiner_0428.fields_current.node", key:

{ status: 1, @fields.ex_node_group.value: 1, @fields.current.value: 1, @fields.promoted_media.fid: 1 }

, name: "ex_skybox_pageviews", background: false }
Tue May 4 17:10:59 examiner_0428.system.indexes Assertion failure !e.eoo() db/jsobj.h 1196
0x4f6696 0x4feda4 0x4b837b 0x658a91 0x659cda 0x607ffb 0x609523 0x60a8c2 0x60c4e6 0x5f5f9c 0x5f8887 0x6aa034 0x6bc730 0x2aaaaacce617 0x2aaaab748c2d
mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4f6696]
mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x4feda4]
mongod(_ZN5mongo14BSONObjBuilder8appendAsERKNS_11BSONElementEPKc+0x14b) [0x4b837b]
mongod(_ZNK5mongo9IndexSpec8_getKeysESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjERSt3setIS9_NS_22BSONObjCmpDefaultOrderESaIS9_EE+0x411) [0x658a91]
mongod(_ZNK5mongo9IndexSpec7getKeysERKNS_7BSONObjERSt3setIS1_NS_22BSONObjCmpDefaultOrderESaIS1_EE+0x2ea) [0x659cda]
mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x56b) [0x607ffb]
mongod [0x609523]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x8a2) [0x60a8c2]
mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x60c4e6]
mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x26c) [0x5f5f9c]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERK11sockaddr_in+0x14e7) [0x5f8887]
mongod(_ZN5mongo10connThreadEv+0x244) [0x6aa034]
mongod(thread_proxy+0x80) [0x6bc730]
/lib64/libpthread.so.0 [0x2aaaaacce617]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748c2d]
Tue May 4 17:10:59 Caught Assertion in insert , continuing
Tue May 4 17:10:59 insert examiner_0428.system.indexes exception assertion db/jsobj.h:1196 184ms
Tue May 4 17:10:59 end connection 10.192.198.15:52586
Tue May 4 17:20:31 connection accepted from 10.192.198.15:33785 #2330

Here is the banner from this version so you can see the git string:

Tue May 4 02:43:26 Mongo DB : starting : pid = 8319 port = 27017 dbpath = /ebs/lib/mongodb/data/ master = 0 slave = 0 64-bit
Tue May 4 02:43:26 db version v1.4.3-pre-, pdfile version 4.5
Tue May 4 02:43:26 git version: fe2d4d93f2599cf2b955e880be2fb64d8dca9097
Tue May 4 02:43:26 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue May 4 02:43:26 waiting for connections on port 27017
Tue May 4 02:43:26 web admin interface listening on port 28017

Downgrading to 1.4.2 removed these assertions on IndexBuild.

Comment by Doug Green [ 04/May/10 ]

We ran this for a day, and it broke our index creations. I think that Karoly removed all indexes, so that this nightly build could recreate them. But it would only create some indexes before seeing these errors in the log. We backed out the nightly and go back to the previous version, and index creation is working fine.

Tue May 4 17:10:58 building new index on

{ uid: 1, @fields.ex_node_group.value: 1, @fields.examiner_title.tid: 1, @fields.tag_summary.tid: 1, status: 1, created: -1 }

for examiner_0428.fields_current.node
Tue May 4 17:10:58 Buildindex examiner_0428.fields_current.node idxNo:9 { _id: ObjId(4be08d628ead0ecf33200000), ns: "examiner_0428.fields_current.node", key:

{ uid: 1, @fields.ex_node_group.value: 1, @fields.examiner_title.tid: 1, @fields.tag_summary.tid: 1, status: 1, created: -1 }

, name: "author_nodes_per_examiner_title_per_tag", background: false }
Tue May 4 17:10:59 examiner_0428.system.indexes Assertion failure !e.eoo() db/jsobj.h 1196
0x4f6696 0x4feda4 0x4b837b 0x658a91 0x659cda 0x607ffb 0x609523 0x60a8c2 0x60c4e6 0x5f5f9c 0x5f8887 0x6aa034 0x6bc730 0x2aaaaacce617 0x2aaaab748c2d
mongod(_ZN5mongo12sayDbContextEPKc+0xe6) [0x4f6696]
mongod(_ZN5mongo8assertedEPKcS1_j+0x154) [0x4feda4]
mongod(_ZN5mongo14BSONObjBuilder8appendAsERKNS_11BSONElementEPKc+0x14b) [0x4b837b]
mongod(_ZNK5mongo9IndexSpec8_getKeysESt6vectorIPKcSaIS3_EES1_INS_11BSONElementESaIS6_EERKNS_7BSONObjERSt3setIS9_NS_22BSONObjCmpDefaultOrderESaIS9_EE+0x411) [0x658a91]
mongod(_ZNK5mongo9IndexSpec7getKeysERKNS_7BSONObjERSt3setIS1_NS_22BSONObjCmpDefaultOrderESaIS1_EE+0x2ea) [0x659cda]
mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0x56b) [0x607ffb]
mongod [0x609523]
mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x8a2) [0x60a8c2]
mongod(_ZN5mongo11DataFileMgr6insertEPKcRNS_7BSONObjEb+0x66) [0x60c4e6]
mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x26c) [0x5f5f9c]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERK11sockaddr_in+0x14e7) [0x5f8887]
mongod(_ZN5mongo10connThreadEv+0x244) [0x6aa034]
mongod(thread_proxy+0x80) [0x6bc730]
/lib64/libpthread.so.0 [0x2aaaaacce617]
/lib64/libc.so.6(clone+0x6d) [0x2aaaab748c2d]
Tue May 4 17:10:59 Caught Assertion in insert , continuing
Tue May 4 17:10:59 insert examiner_0428.system.indexes exception assertion db/jsobj.h:1196 247ms

Comment by Karoly Negyesi [ 04/May/10 ]

Thank you SO MUCH for prompt discovery and fixing.

Comment by auto [ 04/May/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: fix array index when empty array SERVER-1082
http://github.com/mongodb/mongo/commit/fe2d4d93f2599cf2b955e880be2fb64d8dca9097

Comment by auto [ 04/May/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: fix array index when empty array SERVER-1082
Note: this only increase index keys, not decrease or change, so old indexes won't cause corruption
http://github.com/mongodb/mongo/commit/c540e566de6c1e2a478f62beb66efbf65a5fba56

Comment by Eliot Horowitz (Inactive) [ 03/May/10 ]

yeah - that's wrong - but doesn't impact the case.

Comment by Karoly Negyesi [ 03/May/10 ]

please fix in 1.4 if possible too.

Also... that commit might be broken, you are inserting two _id: 1 items?

Comment by auto [ 03/May/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: broken test for SERVER-1082
http://github.com/mongodb/mongo/commit/98e28e523c5110c48e4e6e861780f6d20a669c86

Generated at Thu Feb 08 02:56:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.