[SERVER-10572] V2.2 Linux 32-bit ABORT during sharding/sync3.js Created: 19/Aug/13  Updated: 10/Dec/14  Resolved: 25/Nov/13

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 2.2.5
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Matt Kangas Assignee: Greg Studer
Resolution: Duplicate Votes: 0
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

buildbot V2.2 Linux 32-bit


Issue Links:
Duplicate
duplicates SERVER-10688 sync6.js - verify failed in Distribut... Closed
Participants:

 Description   

V2.2 Linux 32 build 317 (Mon Aug 19, rev d62637911)

mongostest_smokesharding::sharding/sync3.js

http://buildlogs.mongodb.org/V2.2%20Linux%2032-bit/builds/317/test/sharding/sync3.js

 m30000| Mon Aug 19 08:52:26 distributed lock 'testdistlockwithsync/bs-linux32.10gen.cc:30000:1376916738:2011189613' acquired, ts : 5212150ae1fe5491b1ea72f2
 m30000| Mon Aug 19 08:52:26 [conn7] CMD fsync: sync:1 lock:0
 m30000| Mon Aug 19 08:52:26 [conn7] fsync from getlasterror
 m30000| Mon Aug 19 08:52:26   Assertion failure indUpdate["state"].numberInt() > 0 src/mongo/client/distlock.cpp 796
 m30000| 0x876a15a 0x8740bb5 0x872b0be 0x81fef11 0x820cce3 0x81803af 0x87b542e 0xdb2542 0x2e8b6e 
 m30000| Mon Aug 19 08:52:26 distributed lock 'testdistlockwithsync/bs-linux32.10gen.cc:30000:1376916738:2011189613' unlocked. 
 m30000| Mon Aug 19 08:52:26 [conn7] CMD fsync: sync:1 lock:0
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x876a15a]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo10logContextEPKc+0xa5) [0x8740bb5]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x12e) [0x872b0be]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo15DistributedLock8lock_tryERKSsbPNS_7BSONObjE+0xd131) [0x81fef11]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo20TestDistLockWithSync9runThreadEv+0x63) [0x820cce3]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5boost6detail11thread_dataIPFvvEE3runEv+0xf) [0x81803af]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod [0x87b542e]
 m30000|  /lib/i686/nosegneg/libpthread.so.0 [0xdb2542]
 m30000|  /lib/i686/nosegneg/libc.so.6(clone+0x5e) [0x2e8b6e]
 m30000| Mon Aug 19 08:52:26 [conn7] fsync from getlasterror
 m30000| Mon Aug 19 08:52:26 [conn6] fsync from getlasterror
 m30000| Mon Aug 19 08:52:26 terminate() called, printing stack (if implemented for platform):
 m30000| 0x876a15a 0x8170e14 0x7bb0a5 0x7bb0e2 0x7bb21a 0x81fdaad 0x820cce3 0x81803af 0x87b542e 0xdb2542 0x2e8b6e 
 m30002| Mon Aug 19 08:52:26 [conn7] CMD fsync: sync:1 lock:0
 m30000| Mon Aug 19 08:52:26 [conn7] CMD fsync: sync:1 lock:0
 m30000| Mon Aug 19 08:52:26 [conn7] fsync from getlasterror
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x876a15a]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo11myterminateEv+0x54) [0x8170e14]
 m30000|  /usr/lib/libstdc++.so.6 [0x7bb0a5]
 m30000|  /usr/lib/libstdc++.so.6 [0x7bb0e2]
 m30000|  /usr/lib/libstdc++.so.6 [0x7bb21a]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo15DistributedLock8lock_tryERKSsbPNS_7BSONObjE+0xbccd) [0x81fdaad]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo20TestDistLockWithSync9runThreadEv+0x63) [0x820cce3]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5boost6detail11thread_dataIPFvvEE3runEv+0xf) [0x81803af]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod [0x87b542e]
 m30000|  /lib/i686/nosegneg/libpthread.so.0 [0xdb2542]
 m30000|  /lib/i686/nosegneg/libc.so.6(clone+0x5e) [0x2e8b6e]
 m30000| Mon Aug 19 08:52:26 Got signal: 6 (Aborted).
 m30000| 
 m30000| Mon Aug 19 08:52:26 Backtrace:
 m30000| 0x876a15a 0x817189a 0x9f4420 0x9f4402 0x23c8a0 0x23e271 0x8170e19 0x7bb0a5 0x7bb0e2 0x7bb21a 0x81fdaad 0x820cce3 0x81803af 0x87b542e 0xdb2542 0x2e8b6e 
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x876a15a]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo10abruptQuitEi+0x3ba) [0x817189a]
 m30000|  [0x9f4420]
 m30000|  [0x9f4402]
 m30000|  /lib/i686/nosegneg/libc.so.6(gsignal+0x50) [0x23c8a0]
 m30000|  /lib/i686/nosegneg/libc.so.6(abort+0x101) [0x23e271]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo11myterminateEv+0x59) [0x8170e19]
 m30000|  /usr/lib/libstdc++.so.6 [0x7bb0a5]
 m30000|  /usr/lib/libstdc++.so.6 [0x7bb0e2]
 m30000|  /usr/lib/libstdc++.so.6 [0x7bb21a]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo15DistributedLock8lock_tryERKSsbPNS_7BSONObjE+0xbccd) [0x81fdaad]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5mongo20TestDistLockWithSync9runThreadEv+0x63) [0x820cce3]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod(_ZN5boost6detail11thread_dataIPFvvEE3runEv+0xf) [0x81803af]
 m30000|  /mnt/slaves/Linux_32bit_V2.2/mongo/mongod [0x87b542e]
 m30000|  /lib/i686/nosegneg/libpthread.so.0 [0xdb2542]
 m30000|  /lib/i686/nosegneg/libc.so.6(clone+0x5e) [0x2e8b6e]



 Comments   
Comment by sam.helman@10gen.com [ 25/Nov/13 ]

This appears to have showed up again -

https://mci.10gen.com/ui/task/mongodb_mongo_master_linux_32_95586f19acca1258a819930e206100efb1ce4000_13_11_25_01_14_05_sharding_linux_32

Direct log link:

http://buildlogs.mongodb.org/mci_0.9_linux_32/builds/9439/test/sharding_0/sync3.js

Comment by Matt Kangas [ 19/Aug/13 ]

Is this the same issue Tad noticed in Builder V2.4 Linux RHEL 32-bit Build #128?

V2.4 Linux RHEL 32-bit Build #128 (Tue Jul 2, rev a6462d21e)

http://buildbot.mongodb.org:8081/builders/V2.4%20Linux%20RHEL%2032-bit/builds/128/steps/test_8/logs/stdio
http://buildlogs.mongodb.org/V2.4%20Linux%20RHEL%2032-bit/builds/128/test/sharding/sync3.js

MongoDB shell version: 2.4.5-rc1-pre-
null
Tue Jul  2 09:51:50.099 shell: started program /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod --port 30000 --dbpath /data/db/sync30 --noprealloc --smallfiles --oplogSize 40 --nohttpinterface --setParameter enableTestCommands=1
 m30000| note: noprealloc may hurt performance in many applications
 m30000| Tue Jul  2 09:51:50.139 
 m30000| Tue Jul  2 09:51:50.139 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
 m30000| Tue Jul  2 09:51:50.139 
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] MongoDB starting : pid=5753 port=30000 dbpath=/data/db/sync30 32-bit host=bs-rhel-57-32-1
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] 
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] **       32 bit builds are limited to less than 2GB of data (or less with --journal).
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] **       Note that journaling defaults to off for 32 bit and is currently off.
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] **       See http://dochub.mongodb.org/core/32bit
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] 
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] db version v2.4.5-rc1-pre-
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] git version: a6462d21ea9487d061eba788333003711793411f
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] build info: Linux bs-rhel-57-32-1 2.6.18-274.el5xen #1 SMP Fri Jul 8 18:18:40 EDT 2011 i686 BOOST_LIB_VERSION=1_49
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] allocator: system
 m30000| Tue Jul  2 09:51:50.153 [initandlisten] options: { dbpath: "/data/db/sync30", nohttpinterface: true, noprealloc: true, oplogSize: 40, port: 30000, setParameter: [ "enableTestCommands=1" ], smallfiles: true }
 m30000| Tue Jul  2 09:51:50.169 [FileAllocator] allocating new datafile /data/db/sync30/local.ns, filling with zeroes...
 m30000| Tue Jul  2 09:51:50.169 [FileAllocator] creating directory /data/db/sync30/_tmp
 m30000| Tue Jul  2 09:51:50.216 [FileAllocator] done allocating datafile /data/db/sync30/local.ns, size: 16MB,  took 0.041 secs
 m30000| Tue Jul  2 09:51:50.216 [FileAllocator] allocating new datafile /data/db/sync30/local.0, filling with zeroes...
 m30000| Tue Jul  2 09:51:50.258 [FileAllocator] done allocating datafile /data/db/sync30/local.0, size: 16MB,  took 0.041 secs
 m30000| Tue Jul  2 09:51:50.260 [initandlisten] waiting for connections on port 30000
 
...
 
 m30000| Tue Jul  2 09:52:00.482 lock update lost, lock 'testdistlockwithsync/bs-rhel-57-32-1:30000:1372773110:1645260249' not propagated.
 m30000| Tue Jul  2 09:52:00.482 distributed lock 'testdistlockwithsync/bs-rhel-57-32-1:30000:1372773110:1645260249' was not acquired.
 m30000| Tue Jul  2 09:52:00.483 lock update won, completing lock propagation for 'testdistlockwithsync/bs-rhel-57-32-1:30000:1372773110:1645260249'
 m30000| Tue Jul  2 09:52:00.484 [conn4] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.488 [conn4] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.495 distributed lock 'testdistlockwithsync/bs-rhel-57-32-1:30000:1372773110:1645260249' acquired, ts : 51d2db008d8b01485b8b5c36
 m30000| Tue Jul  2 09:52:00.506 [conn4] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.508 [conn4] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.509 [conn6] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.510   Assertion failure indUpdate[LocksType::state()].numberInt() > 0 src/mongo/client/distlock.cpp 886
 m30000| Tue Jul  2 09:52:00.510 [conn8] fsync from getlasterror
 m30001| Tue Jul  2 09:52:00.511 [conn5] fsync from getlasterror
 m30001| Tue Jul  2 09:52:00.512 [conn4] fsync from getlasterror
 m30001| Tue Jul  2 09:52:00.512 [conn8] fsync from getlasterror
 m30000| 0x8a0e27d 0x89e3535 0x89c8bde 0x8374bc4 0x8384493 0x82d02df 0x8a5c62e 0x3a3869 0x30d4de 
 m30002| Tue Jul  2 09:52:00.514 [conn5] fsync from getlasterror
 m30002| Tue Jul  2 09:52:00.515 [conn4] fsync from getlasterror
 m30002| Tue Jul  2 09:52:00.515 [conn7] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.515 distributed lock 'testdistlockwithsync/bs-rhel-57-32-1:30000:1372773110:1645260249' unlocked. 
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x2d) [0x8a0e27d]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo10logContextEPKc+0xa5) [0x89e3535]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x12e) [0x89c8bde]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo15DistributedLock8lock_tryERKSsbPNS_7BSONObjEd+0x11064) [0x8374bc4]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo20TestDistLockWithSync9runThreadEv+0x73) [0x8384493]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5boost6detail11thread_dataIPFvvEE3runEv+0xf) [0x82d02df]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod [0x8a5c62e]
 m30000|  /lib/i686/nosegneg/libpthread.so.0 [0x3a3869]
 m30000|  /lib/i686/nosegneg/libc.so.6(clone+0x5e) [0x30d4de]
 m30000| Tue Jul  2 09:52:00.516 [conn8] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.517 [conn6] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.522 terminate() called, printing stack (if implemented for platform):
 m30000| 0x8a0e27d 0x82be0e4 0x8325b5 0x8325f2 0x832731 0x837334e 0x8384493 0x82d02df 0x8a5c62e 0x3a3869 0x30d4de 
 m30001| Tue Jul  2 09:52:00.523 [conn8] fsync from getlasterror
 m30001| Tue Jul  2 09:52:00.524 [conn4] fsync from getlasterror
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x2d) [0x8a0e27d]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo11myterminateEv+0x54) [0x82be0e4]
 m30000|  /usr/lib/libstdc++.so.6 [0x8325b5]
 m30000|  /usr/lib/libstdc++.so.6 [0x8325f2]
 m30000|  /usr/lib/libstdc++.so.6 [0x832731]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo15DistributedLock8lock_tryERKSsbPNS_7BSONObjEd+0xf7ee) [0x837334e]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo20TestDistLockWithSync9runThreadEv+0x73) [0x8384493]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5boost6detail11thread_dataIPFvvEE3runEv+0xf) [0x82d02df]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod [0x8a5c62e]
 m30000|  /lib/i686/nosegneg/libpthread.so.0 [0x3a3869]
 m30000|  /lib/i686/nosegneg/libc.so.6(clone+0x5e) [0x30d4de]
 m30000| Tue Jul  2 09:52:00.524 Got signal: 6 (Aborted).
 m30000| 
 m30002| Tue Jul  2 09:52:00.526 [conn7] fsync from getlasterror
 m30002| Tue Jul  2 09:52:00.526 [conn4] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.528 [conn8] fsync from getlasterror
 m30001| Tue Jul  2 09:52:00.528 [conn8] fsync from getlasterror
 m30002| Tue Jul  2 09:52:00.529 [conn7] fsync from getlasterror
 m30000| Tue Jul  2 09:52:00.530 Backtrace:
 m30000| 0x8a0e27d 0x82bf43a 0xa39420 0xa39402 0x262040 0x263a21 0x82be0e9 0x8325b5 0x8325f2 0x832731 0x837334e 0x8384493 0x82d02df 0x8a5c62e 0x3a3869 0x30d4de 
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo15printStackTraceERSo+0x2d) [0x8a0e27d]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo10abruptQuitEi+0x3ba) [0x82bf43a]
 m30000|  [0xa39420]
 m30000|  [0xa39402]
 m30000|  /lib/i686/nosegneg/libc.so.6(gsignal+0x50) [0x262040]
 m30000|  /lib/i686/nosegneg/libc.so.6(abort+0x101) [0x263a21]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo11myterminateEv+0x59) [0x82be0e9]
 m30000|  /usr/lib/libstdc++.so.6 [0x8325b5]
 m30000|  /usr/lib/libstdc++.so.6 [0x8325f2]
 m30000|  /usr/lib/libstdc++.so.6 [0x832731]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo15DistributedLock8lock_tryERKSsbPNS_7BSONObjEd+0xf7ee) [0x837334e]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5mongo20TestDistLockWithSync9runThreadEv+0x73) [0x8384493]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod(_ZN5boost6detail11thread_dataIPFvvEE3runEv+0xf) [0x82d02df]
 m30000|  /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/mongod [0x8a5c62e]
 m30000|  /lib/i686/nosegneg/libpthread.so.0 [0x3a3869]
 m30000|  /lib/i686/nosegneg/libc.so.6(clone+0x5e) [0x30d4de]
 m30000| 
Tue Jul  2 09:52:00.535 DBClientCursor::init call() failed
 m30001| Tue Jul  2 09:52:00.535 [conn3] end connection 127.0.0.1:38386 (8 connections now open)
 m30001| Tue Jul  2 09:52:00.536 [conn5] end connection 127.0.0.1:38392 (7 connections now open)
 m30001| Tue Jul  2 09:52:00.536 [conn7] end connection 127.0.0.1:38398 (6 connections now open)
 m30001| Tue Jul  2 09:52:00.536 [conn4] end connection 127.0.0.1:38391 (5 connections now open)
 m30001| Tue Jul  2 09:52:00.536 [conn6] end connection 127.0.0.1:38397 (4 connections now open)
 m30001| Tue Jul  2 09:52:00.536 [conn8] end connection 127.0.0.1:38399 (3 connections now open)
 m30001| Tue Jul  2 09:52:00.537 [conn9] end connection 127.0.0.1:38411 (2 connections now open)
 m30002| Tue Jul  2 09:52:00.536 [conn3] end connection 127.0.0.1:41755 (8 connections now open)
 m30002| Tue Jul  2 09:52:00.536 [conn6] end connection 127.0.0.1:41768 (8 connections now open)
 m30002| Tue Jul  2 09:52:00.536 [conn8] end connection 127.0.0.1:41770 (6 connections now open)
 m30002| Tue Jul  2 09:52:00.536 [conn5] end connection 127.0.0.1:41764 (5 connections now open)
 m30002| Tue Jul  2 09:52:00.536 [conn7] end connection 127.0.0.1:41769 (4 connections now open)
 m30002| Tue Jul  2 09:52:00.537 [conn9] end connection 127.0.0.1:41776 (3 connections now open)
 m30002| Tue Jul  2 09:52:00.537 [conn4] end connection 127.0.0.1:41763 (2 connections now open)
Tue Jul  2 09:52:00.590 JavaScript execution failed: Error: error doing query: failed at src/mongo/shell/query.js:L78
failed to load: /data/buildslaves/Linux_RHEL_32bit_V2.4/mongo/jstests/sharding/sync3.js

Generated at Thu Feb 08 03:23:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.