[SERVER-22996] mongos (nightly build) segfault with invalid config option Created: 07/Mar/16  Updated: 17/May/16  Resolved: 17/May/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.3.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Bernie Hackett Assignee: Misha Tyulenev
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File stacktrace.txt    
Issue Links:
Duplicate
duplicates SERVER-22320 Remove SYNC option and SyncClusterCon... Closed
is duplicated by SERVER-23087 mongos crash after "SYNC config serve... Closed
Related
related to SERVER-22318 Remove code that supports SCCC Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:
Linked BF Score: 0

 Description   

It's not entirely clear what the invalid configuration option is, but seems to be related to config server configuration. Regardless mongos shouldn't segfault.

2016-03-07T22:23:44.718+0000 E SHARDING [mongosMain] Error initializing sharding system: UnsupportedFormat: SYNC config server connection string is not allowed.



 Comments   
Comment by Misha Tyulenev [ 11/Mar/16 ]

Fixed: https://jira.mongodb.org/browse/SERVER-22320
Could you please confirm - that the latest master does not segfault if you're specifying the config servers in SCCC format?

Comment by Rui Zhang (Inactive) [ 10/Mar/16 ]

Yes, with Spencer's format, it works.

Comment by Misha Tyulenev [ 10/Mar/16 ]

rui.zhang please clarify are you getting this segfault by just trying to start mongos with the SYNC connection format?
If you change tests to use --configdb format specified by Spencer it should work.

Comment by Spencer Brody (Inactive) [ 09/Mar/16 ]

Yeah, I think you need to change that to

--configdb <replset name>/$IPconfig1:27017,$IPconfig2:27017,$IPconfig3:27017

Comment by Rui Zhang (Inactive) [ 08/Mar/16 ]

let me give it a try, I am using following format for mongos

mongos ...    --configdb $IPconfig1:27017,$IPconfig2:27017,$IPconfig3:27017

Comment by Spencer Brody (Inactive) [ 08/Mar/16 ]

Clearly the fact that this is segfaulting instead of exiting cleanly with an error message is a problem.

The configuration problem triggering the segfault, however, is that you're specifying the config servers in SCCC format (listing the 3 config servers comma separated with no replset name), whereas now you need to specify the config servers with a replica set connection string (ie <replset name>/<confisvr1>,<configsvr2>,....)

Comment by Rui Zhang (Inactive) [ 08/Mar/16 ]

This seems not related to legacy config server, we are using CSRS in sys-perf tests. Here is log

config servers are CSRS

[ec2-user@ip-10-2-1-81 ~]$ ps -aef | grep mongod
ec2-user  2911     1  1 22:58 ?        00:00:01 /home/ec2-user/latest/bin/mongod --port 27017 --replSet configSvrRS --dbpath /home/ec2-user/data/dbs --configsvr --fork --logpath /home/ec2-user/data/logs/m
ongod.log --storageEngine=wiredTiger

stack trace for mongos

[ec2-user@ip-10-2-1-99 ~]$ cat data/logs/mongos.log
2016-03-08T22:59:49.028+0000 I CONTROL  [main]
2016-03-08T22:59:49.029+0000 I CONTROL  [main] ** NOTE: This is a development version (3.3.2-123-ge4bd8ce) of MongoDB.
2016-03-08T22:59:49.029+0000 I CONTROL  [main] **       Not recommended for production.
2016-03-08T22:59:49.029+0000 I CONTROL  [main]
2016-03-08T22:59:49.029+0000 I SHARDING [mongosMain] MongoS version 3.3.2-123-ge4bd8ce starting: pid=3555 port=27017 64-bit host=ip-10-2-1-99 (--help for usage)
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain] db version v3.3.2-123-ge4bd8ce
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain] git version: e4bd8cef2b34b96f07986d8523333dd3c6b29f1a
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain] allocator: tcmalloc
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain] modules: none
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain] build environment:
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain]     distarch: x86_64
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain]     target_arch: x86_64
2016-03-08T22:59:49.029+0000 I CONTROL  [mongosMain] options: { processManagement: { fork: true }, sharding: { chunkSize: 64, configDB: "10.2.1.81:27017,10.2.1.82:27017,10.2.1.83:27017" }, systemLog: { de
stination: "file", path: "/home/ec2-user/data/logs/mongos.log" } }
2016-03-08T22:59:49.029+0000 E SHARDING [mongosMain] Error initializing sharding system: UnsupportedFormat: SYNC config server connection string is not allowed.
2016-03-08T22:59:49.029+0000 F -        [mongosMain] Invalid access at address: 0x18
2016-03-08T22:59:49.031+0000 F -        [mongosMain] Got signal: 11 (Segmentation fault).
 
 0xbea022 0xbe8f49 0xbe92c8 0x7f038503f100 0x7f0385039bd0 0xb20940 0x63dd5d 0x640f30 0x7f0384c8fb15 0x63d8d9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"7EA022","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"7E8F49"},{"b":"400000","o":"7E92C8"},{"b":"7F0385030000","o":"F100"},{"b":"7F0385030000","o":"9BD0","s":"p
thread_mutex_lock"},{"b":"400000","o":"720940","s":"_ZN5mongo20ClusterCursorManager8shutdownEv"},{"b":"400000","o":"23DD5D","s":"_ZN5mongo11exitCleanlyENS_8ExitCodeE"},{"b":"400000","o":"240F30","s":"main
"},{"b":"7F0384C6E000","o":"21B15","s":"__libc_start_main"},{"b":"400000","o":"23D8D9"}],"processInfo":{ "mongodbVersion" : "3.3.2-123-ge4bd8ce", "gitVersion" : "e4bd8cef2b34b96f07986d8523333dd3c6b29f1a",
 "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.14.35-28.38.amzn1.x86_64", "version" : "#1 SMP Wed Mar 11 22:50:37 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2,
"b" : "400000" }, { "b" : "7FFEEC22F000", "elfType" : 3 }, { "b" : "7F0385C6C000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7F0385A68000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "
b" : "7F0385764000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F0385462000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7F038524C000", "path" : "/lib64/libgcc_s.so.1", "
elfType" : 3 }, { "b" : "7F0385030000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7F0384C6E000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7F0385E74000", "path" : "/lib64/l
d-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xbea022]
 mongos(+0x7E8F49) [0xbe8f49]
 mongos(+0x7E92C8) [0xbe92c8]
 libpthread.so.0(+0xF100) [0x7f038503f100]
 libpthread.so.0(pthread_mutex_lock+0x0) [0x7f0385039bd0]
 mongos(_ZN5mongo20ClusterCursorManager8shutdownEv+0x30) [0xb20940]
 mongos(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x5D) [0x63dd5d]
 mongos(main+0x10) [0x640f30]
 libc.so.6(__libc_start_main+0xF5) [0x7f0384c8fb15]
 mongos(+0x23D8D9) [0x63d8d9]
-----  END BACKTRACE  -----
2016-03-08T22:59:49.031+0000 F -        [mongosMain] /proc/self/maps:
00400000-01028000 r-xp 00000000 ca:01 413190                             /home/ec2-user/latest/bin/mongos
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 01228000-0127d000 rw-p 00c28000 ca:01 413190                             /home/ec2-user/latest/bin/mongos
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 0127d000-012cc000 rw-p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 01c38000-02738000 rw-p 00000000 00:00 0                                  [heap]
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 02738000-02838000 rw-p 00000000 00:00 0                                  [heap]
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f037df46000-7f037df47000 ---p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f037df47000-7f037e747000 rw-p 00000000 00:00 0                          [stack:3556]
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f037e747000-7f0384c6e000 r--p 00000000 ca:01 410811                     /usr/lib/locale/locale-archive
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0384c6e000-7f0384e25000 r-xp 00000000 ca:01 396466                     /lib64/libc-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0384e25000-7f0385025000 ---p 001b7000 ca:01 396466                     /lib64/libc-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385025000-7f0385029000 r--p 001b7000 ca:01 396466                     /lib64/libc-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385029000-7f038502b000 rw-p 001bb000 ca:01 396466                     /lib64/libc-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f038502b000-7f0385030000 rw-p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385030000-7f0385046000 r-xp 00000000 ca:01 396492                     /lib64/libpthread-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385046000-7f0385246000 ---p 00016000 ca:01 396492                     /lib64/libpthread-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385246000-7f0385247000 r--p 00016000 ca:01 396492                     /lib64/libpthread-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385247000-7f0385248000 rw-p 00017000 ca:01 396492                     /lib64/libpthread-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385248000-7f038524c000 rw-p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f038524c000-7f0385261000 r-xp 00000000 ca:01 396417                     /lib64/libgcc_s-4.8.2-20140120.so.1
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385261000-7f0385461000 ---p 00015000 ca:01 396417                     /lib64/libgcc_s-4.8.2-20140120.so.1
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385461000-7f0385462000 rw-p 00015000 ca:01 396417                     /lib64/libgcc_s-4.8.2-20140120.so.1
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385462000-7f0385563000 r-xp 00000000 ca:01 396474                     /lib64/libm-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385563000-7f0385762000 ---p 00101000 ca:01 396474                     /lib64/libm-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385762000-7f0385763000 r--p 00100000 ca:01 396474                     /lib64/libm-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385763000-7f0385764000 rw-p 00101000 ca:01 396474                     /lib64/libm-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385764000-7f038584a000 r-xp 00000000 ca:01 396685                     /usr/lib64/libstdc++.so.6.0.19
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f038584a000-7f0385a49000 ---p 000e6000 ca:01 396685                     /usr/lib64/libstdc++.so.6.0.19
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385a49000-7f0385a51000 r--p 000e5000 ca:01 396685                     /usr/lib64/libstdc++.so.6.0.19
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385a51000-7f0385a53000 rw-p 000ed000 ca:01 396685                     /usr/lib64/libstdc++.so.6.0.19
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385a53000-7f0385a68000 rw-p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385a68000-7f0385a6b000 r-xp 00000000 ca:01 396472                     /lib64/libdl-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385a6b000-7f0385c6a000 ---p 00003000 ca:01 396472                     /lib64/libdl-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385c6a000-7f0385c6b000 r--p 00002000 ca:01 396472                     /lib64/libdl-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385c6b000-7f0385c6c000 rw-p 00003000 ca:01 396472                     /lib64/libdl-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385c6c000-7f0385c73000 r-xp 00000000 ca:01 396496                     /lib64/librt-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385c73000-7f0385e72000 ---p 00007000 ca:01 396496                     /lib64/librt-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385e72000-7f0385e73000 r--p 00006000 ca:01 396496                     /lib64/librt-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385e73000-7f0385e74000 rw-p 00007000 ca:01 396496                     /lib64/librt-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0385e74000-7f0385e95000 r-xp 00000000 ca:01 396459                     /lib64/ld-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0386082000-7f0386089000 r--s 00000000 ca:01 1558                       /usr/lib64/gconv/gconv-modules.cache
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0386089000-7f038608f000 rw-p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0386094000-7f0386095000 rw-p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0386095000-7f0386096000 r--p 00021000 ca:01 396459                     /lib64/ld-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0386096000-7f0386097000 rw-p 00022000 ca:01 396459                     /lib64/ld-2.17.so
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7f0386097000-7f0386098000 rw-p 00000000 00:00 0
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7ffeeb8f9000-7ffeeb91a000 rw-p 00000000 00:00 0                          [stack]
2016-03-08T22:59:49.031+0000 F -        [mongosMain] 7ffeeb92f000-7ffeeb931000 r-xp 00000000 00:00 0                          [vdso]
2016-03-08T22:59:49.031+0000 F -        [mongosMain] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Comment by Bernie Hackett [ 07/Mar/16 ]

I see. I thought we added support for CSRS in mongo-orchestration, but looks like it was never merged.

https://github.com/10gen/mongo-orchestration/pull/192

Comment by Spencer Brody (Inactive) [ 07/Mar/16 ]

Likely fallout from SERVER-22318

Comment by Bernie Hackett [ 07/Mar/16 ]

Stacktrace attached. It surpasses Jira's character count limit.

Generated at Thu Feb 08 04:02:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.