[SERVER-34746] Segmentation fault when shard is started with --shardsvr before being added to a shard Created: 30/Apr/18  Updated: 29/Oct/23  Resolved: 01/May/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.6.4
Fix Version/s: 3.6.5

Type: Bug Priority: Major - P3
Reporter: Mariano Benítez Mulet Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-36520 Segmentation fault when add member fr... Closed
is duplicated by SERVER-37119 try to connect version 3.6.4 with c... Closed
Problem/Incident
is caused by SERVER-32677 Segmentation fault converting Replica... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Set up in docker a replica set with:

mongo1:
    container_name: mongo1
    image: mongo:3.6.3
    command: mongod --shardsvr --replSet RS1 --dbpath /data/db --port 27017
    ports:
      - 27017:27017
    expose:
      - 27017
    environment:
      TERM: xterm
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/docker/mongo-rs/mongo1:/data/db
  mongo2:
    container_name: mongo2
    image: mongo:3.6.3
    command: mongod --shardsvr --replSet RS1 --dbpath /data/db --port 27027
    ports:
      - 27027:27027
    expose:
      - 27027
    environment:
      TERM: xterm
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/docker/mongo-rs/mongo2:/data/db
  mongo3:
    container_name: mongo3
    image: mongo:3.6.3
    command: mongod --shardsvr --replSet RS1 --dbpath /data/db --port 27037
    ports:
      - 27037:27037
    expose:
      - 27037
    environment:
      TERM: xterm
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/docker/mongo-rs/mongo3:/data/db 

Replica set in configured with:

config = {
    "_id" : "RS1",
    "members" : [
        {
            "_id" : 0,
            "host" : "mongo1:27017"
        },
        {
            "_id" : 1,
            "host" : "mongo2:27027"
        },
        {
            "_id" : 2,
            "host" : "mongo3:27037"
        }
    ]
}
rs.initiate(config); 

Collection is set with:

db.adminCommand({
    setFeatureCompatibilityVersion: "3.6"
});
databaseName = "myDatabase";
collectionName = "people";
 
db = db.getSiblingDB(databaseName);
collection = db.getCollection(collectionName);
collection.drop();db.createCollection(collectionName, {
   validator: { $jsonSchema: {
      bsonType: "object",
      required: ["name","description","address"],
      properties: {
         name: {
            bsonType: "string",
            description: "must be a string and is required"
         },
         description: {
            bsonType: "string",
            description: "must be a string and is required"
         },
         address: {
            bsonType: "string",
            description: "must be a string and is required"
         }
      }
   }}
});
collection = db.getCollection(collectionName);
collection.createIndex({
    name: -1,
    description: "text"
});
 

After that, after a random amount of time the whole set of nodes crash.

Let me know if there's further data I can provide.

Cheers

 

Sprint: Sharding 2018-05-21
Participants:
Case:

 Description   

After some time the replica set is running in Docker all nodes of my replica set crash due to segmentation fault.

This happened to me every time I use JSON schema validation in a collection. When I don't, all nodes work fine.

Here I am pasting the logs:

mongo2            | 2018-04-30T11:10:25.905+0200 F -        [thread74] Invalid access at address: 0x18
mongo2            | 2018-04-30T11:10:25.947+0200 F -        [thread74] Got signal: 11 (Segmentation fault).
mongo2            | 
mongo2            |  0x55f86f905f11 0x55f86f905129 0x55f86f905796 0x7fe038fea890 0x7fe038fe5234 0x55f86f0e0306 0x55f86f0e5091 0x55f86e7741fe 0x55f86e774520 0x55f86f1e3b8a 0x55f86f1e5038 0x55f86e75b5a2 0x55f86f3e01fa 0x55f86f3e0bbc 0x55f86f3e0f94 0x55f86f4569f9 0x55f86f456c41 0x55f86e75a4ad 0x55f86fa14fd0 0x7fe038fe3064 0x7fe038d1862d
mongo2            | ----- BEGIN BACKTRACE -----
mongo2            | {"backtrace":[{"b":"55F86D6F4000","o":"2211F11","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55F86D6F4000","o":"2211129"},{"b":"55F86D6F4000","o":"2211796"},{"b":"7FE038FDB000","o":"F890"},{"b":"7FE038FDB000","o":"A234","s":"pthread_mutex_lock"},{"b":"55F86D6F4000","o":"19EC306","s":"_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE"},{"b":"55F86D6F4000","o":"19F1091","s":"_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE"},{"b":"55F86D6F4000","o":"10801FE","s":"_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE"},{"b":"55F86D6F4000","o":"1080520","s":"_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE"},{"b":"55F86D6F4000","o":"1AEFB8A","s":"_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE"},{"b":"55F86D6F4000","o":"1AF1038","s":"_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE"},{"b":"55F86D6F4000","o":"10675A2"},{"b":"55F86D6F4000","o":"1CEC1FA","s":"_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_"},{"b":"55F86D6F4000","o":"1CECBBC","s":"_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_"},{"b":"55F86D6F4000","o":"1CECF94","s":"_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m"},{"b":"55F86D6F4000","o":"1D629F9","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"55F86D6F4000","o":"1D62C41","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"55F86D6F4000","o":"10664AD"},{"b":"55F86D6F4000","o":"2320FD0"},{"b":"7FE038FDB000","o":"8064"},{"b":"7FE038C30000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55F86D6F4000", "elfType" : 3, "buildId" : "F9BA3BD99A1212BD7BCFA9EDA7EB7CC05FEBB67C" }, { "b" : "7FFDB03DF000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7FE03A178000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7FE039F17000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7FE039B1B000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FD6376149047833953B0269E84DE181CA45DBE90" }, { "b" : "7FE039913000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7FE03970F000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7FE03940E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7FE0391F8000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7FE038FDB000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7FE038C30000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7FE03A38F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
mongo2            |  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55f86f905f11]
mongo2            |  mongod(+0x2211129) [0x55f86f905129]
mongo2            |  mongod(+0x2211796) [0x55f86f905796]
mongo2            |  libpthread.so.0(+0xF890) [0x7fe038fea890]
mongo2            |  libpthread.so.0(pthread_mutex_lock+0x4) [0x7fe038fe5234]
mongo2            |  mongod(_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE+0x46) [0x55f86f0e0306]
mongo2            |  mongod(_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE+0x41) [0x55f86f0e5091]
mongo2            |  mongod(_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE+0x10E) [0x55f86e7741fe]
mongo2            |  mongod(_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE+0x20) [0x55f86e774520]
mongo2            |  mongod(_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE+0x12A) [0x55f86f1e3b8a]
mongo2            |  mongod(_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE+0x28) [0x55f86f1e5038]
mongo2            |  mongod(+0x10675A2) [0x55f86e75b5a2]
mongo2            |  mongod(_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_+0x7A) [0x55f86f3e01fa]
mongo2            |  mongod(_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_+0x3AC) [0x55f86f3e0bbc]
mongo2            |  mongod(_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m+0x164) [0x55f86f3e0f94]
mongo2            |  mongod(_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code+0x389) [0x55f86f4569f9]
mongo2            |  mongod(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x55f86f456c41]
mongo2            |  mongod(+0x10664AD) [0x55f86e75a4ad]
mongo2            |  mongod(+0x2320FD0) [0x55f86fa14fd0]
mongo2            |  libpthread.so.0(+0x8064) [0x7fe038fe3064]
mongo2            |  libc.so.6(clone+0x6D) [0x7fe038d1862d]
mongo2            | -----  END BACKTRACE  -----
 

mongo3            | 2018-04-30T11:10:26.083+0200 I NETWORK  [conn10] end connection 172.19.0.8:34336 (9 connections now open)
mongo3            | 2018-04-30T11:10:26.083+0200 I NETWORK  [conn5] end connection 172.19.0.8:34290 (8 connections now open)
mongo3            | 2018-04-30T11:10:26.083+0200 I NETWORK  [conn11] end connection 172.19.0.8:34338 (7 connections now open)
mongo3            | 2018-04-30T11:10:26.084+0200 I NETWORK  [conn15] end connection 172.19.0.8:34346 (6 connections now open)
mongo2 exited with code 139
mongo3            | 2018-04-30T11:10:26.945+0200 I NETWORK  [listener] connection accepted from 172.19.0.2:42980 #75 (7 connections now open)
mongo3            | 2018-04-30T11:10:26.955+0200 F -        [thread76] Invalid access at address: 0x18
mongo3            | 2018-04-30T11:10:26.982+0200 I NETWORK  [conn75] end connection 172.19.0.2:42980 (6 connections now open)
mongo3            | 2018-04-30T11:10:26.985+0200 F -        [thread76] Got signal: 11 (Segmentation fault).
mongo3            | 
mongo3            |  0x561674202f11 0x561674202129 0x561674202796 0x7f7ac94db890 0x7f7ac94d6234 0x5616739dd306 0x5616739e2091 0x5616730711fe 0x561673071520 0x561673ae0b8a 0x561673ae2038 0x5616730585a2 0x561673cdd1fa 0x561673cddbbc 0x561673cddf94 0x561673d539f9 0x561673d53c41 0x5616730574ad 0x561674311fd0 0x7f7ac94d4064 0x7f7ac920962d
mongo3            | ----- BEGIN BACKTRACE -----
mongo3            | {"backtrace":[{"b":"561671FF1000","o":"2211F11","s":"_ZN5mongo15printStackTraceERSo"},{"b":"561671FF1000","o":"2211129"},{"b":"561671FF1000","o":"2211796"},{"b":"7F7AC94CC000","o":"F890"},{"b":"7F7AC94CC000","o":"A234","s":"pthread_mutex_lock"},{"b":"561671FF1000","o":"19EC306","s":"_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE"},{"b":"561671FF1000","o":"19F1091","s":"_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE"},{"b":"561671FF1000","o":"10801FE","s":"_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE"},{"b":"561671FF1000","o":"1080520","s":"_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE"},{"b":"561671FF1000","o":"1AEFB8A","s":"_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE"},{"b":"561671FF1000","o":"1AF1038","s":"_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE"},{"b":"561671FF1000","o":"10675A2"},{"b":"561671FF1000","o":"1CEC1FA","s":"_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_"},{"b":"561671FF1000","o":"1CECBBC","s":"_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_"},{"b":"561671FF1000","o":"1CECF94","s":"_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m"},{"b":"561671FF1000","o":"1D629F9","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"561671FF1000","o":"1D62C41","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"561671FF1000","o":"10664AD"},{"b":"561671FF1000","o":"2320FD0"},{"b":"7F7AC94CC000","o":"8064"},{"b":"7F7AC9121000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "561671FF1000", "elfType" : 3, "buildId" : "F9BA3BD99A1212BD7BCFA9EDA7EB7CC05FEBB67C" }, { "b" : "7FFFBA33C000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7F7ACA669000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7F7ACA408000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7F7ACA00C000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FD6376149047833953B0269E84DE181CA45DBE90" }, { "b" : "7F7AC9E04000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7F7AC9C00000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7F7AC98FF000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7F7AC96E9000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7F7AC94CC000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7F7AC9121000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7F7ACA880000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
mongo3            |  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x561674202f11]
mongo3            |  mongod(+0x2211129) [0x561674202129]
mongo3            |  mongod(+0x2211796) [0x561674202796]
mongo3            |  libpthread.so.0(+0xF890) [0x7f7ac94db890]
mongo3            |  libpthread.so.0(pthread_mutex_lock+0x4) [0x7f7ac94d6234]
mongo3            |  mongod(_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE+0x46) [0x5616739dd306]
mongo3            |  mongod(_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE+0x41) [0x5616739e2091]
mongo3            |  mongod(_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE+0x10E) [0x5616730711fe]
mongo3            |  mongod(_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE+0x20) [0x561673071520]
mongo3            |  mongod(_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE+0x12A) [0x561673ae0b8a]
mongo3            |  mongod(_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE+0x28) [0x561673ae2038]
mongo3            |  mongod(+0x10675A2) [0x5616730585a2]
mongo3            |  mongod(_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_+0x7A) [0x561673cdd1fa]
mongo3            |  mongod(_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_+0x3AC) [0x561673cddbbc]
mongo3            |  mongod(_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m+0x164) [0x561673cddf94]
mongo3            |  mongod(_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code+0x389) [0x561673d539f9]
mongo3            |  mongod(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x561673d53c41]
mongo3            |  mongod(+0x10664AD) [0x5616730574ad]
mongo3            |  mongod(+0x2320FD0) [0x561674311fd0]
mongo3            |  libpthread.so.0(+0x8064) [0x7f7ac94d4064]
mongo3            |  libc.so.6(clone+0x6D) [0x7f7ac920962d]
mongo3            | -----  END BACKTRACE  -----
 

mongo1            | 2018-04-30T11:10:26.992+0200 F -        [thread71] Invalid access at address: 0x18
mongo1            | 2018-04-30T11:10:27.005+0200 F -        [thread71] Got signal: 11 (Segmentation fault).
mongo1            | 
mongo1            |  0x558e136f6f11 0x558e136f6129 0x558e136f6796 0x7fccace4b890 0x7fccace46234 0x558e12ed1306 0x558e12ed6091 0x558e125651fe 0x558e12565520 0x558e12fd4b8a 0x558e12fd6038 0x558e1254c5a2 0x558e131d11fa 0x558e131d1bbc 0x558e131d1f94 0x558e132479f9 0x558e13247c41 0x558e1254b4ad 0x558e13805fd0 0x7fccace44064 0x7fccacb7962d
mongo1            | ----- BEGIN BACKTRACE -----
mongo1            | {"backtrace":[{"b":"558E114E5000","o":"2211F11","s":"_ZN5mongo15printStackTraceERSo"},{"b":"558E114E5000","o":"2211129"},{"b":"558E114E5000","o":"2211796"},{"b":"7FCCACE3C000","o":"F890"},{"b":"7FCCACE3C000","o":"A234","s":"pthread_mutex_lock"},{"b":"558E114E5000","o":"19EC306","s":"_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE"},{"b":"558E114E5000","o":"19F1091","s":"_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE"},{"b":"558E114E5000","o":"10801FE","s":"_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE"},{"b":"558E114E5000","o":"1080520","s":"_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE"},{"b":"558E114E5000","o":"1AEFB8A","s":"_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE"},{"b":"558E114E5000","o":"1AF1038","s":"_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE"},{"b":"558E114E5000","o":"10675A2"},{"b":"558E114E5000","o":"1CEC1FA","s":"_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_"},{"b":"558E114E5000","o":"1CECBBC","s":"_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_"},{"b":"558E114E5000","o":"1CECF94","s":"_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m"},{"b":"558E114E5000","o":"1D629F9","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"558E114E5000","o":"1D62C41","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"558E114E5000","o":"10664AD"},{"b":"558E114E5000","o":"2320FD0"},{"b":"7FCCACE3C000","o":"8064"},{"b":"7FCCACA91000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "558E114E5000", "elfType" : 3, "buildId" : "F9BA3BD99A1212BD7BCFA9EDA7EB7CC05FEBB67C" }, { "b" : "7FFDA834D000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7FCCADFD9000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7FCCADD78000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7FCCAD97C000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FD6376149047833953B0269E84DE181CA45DBE90" }, { "b" : "7FCCAD774000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7FCCAD570000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7FCCAD26F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7FCCAD059000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7FCCACE3C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7FCCACA91000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7FCCAE1F0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
mongo1            |  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x558e136f6f11]
mongo1            |  mongod(+0x2211129) [0x558e136f6129]
mongo1            |  mongod(+0x2211796) [0x558e136f6796]
mongo1            |  libpthread.so.0(+0xF890) [0x7fccace4b890]
mongo1            |  libpthread.so.0(pthread_mutex_lock+0x4) [0x7fccace46234]
mongo1            |  mongod(_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE+0x46) [0x558e12ed1306]
mongo1            |  mongod(_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE+0x41) [0x558e12ed6091]
mongo1            |  mongod(_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE+0x10E) [0x558e125651fe]
mongo1            |  mongod(_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE+0x20) [0x558e12565520]
mongo1            |  mongod(_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE+0x12A) [0x558e12fd4b8a]
mongo1            |  mongod(_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE+0x28) [0x558e12fd6038]
mongo1            |  mongod(+0x10675A2) [0x558e1254c5a2]
mongo1            |  mongod(_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_+0x7A) [0x558e131d11fa]
mongo1            |  mongod(_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_+0x3AC) [0x558e131d1bbc]
mongo1            |  mongod(_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m+0x164) [0x558e131d1f94]
mongo1            |  mongod(_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code+0x389) [0x558e132479f9]
mongo1            |  mongod(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x558e13247c41]
mongo1            |  mongod(+0x10664AD) [0x558e1254b4ad]
mongo1            |  mongod(+0x2320FD0) [0x558e13805fd0]
mongo1            |  libpthread.so.0(+0x8064) [0x7fccace44064]
mongo1            |  libc.so.6(clone+0x6D) [0x7fccacb7962d]
mongo1            | -----  END BACKTRACE  -----
 

 

The error happens after a variable amount of time, independently from the activity happening in the replica set.

 



 Comments   
Comment by Mariano Benítez Mulet [ 02/May/18 ]

Yes, I already did that and it all seems to be working smoothly, thanks.

Comment by Kaloian Manassiev [ 02/May/18 ]

Hi pachico,

We don't apply patches to released versions, instead we are going to release this fix as part of 3.6.5.

Just to clarify - the scenario under which you are experiencing this crash, where you are using a -shardsvr node directly, outside of a sharded cluster, is not an intended use, so if this bug is blocking you I would suggest that you remove the -shardsvr parameter so the configuration becomes just a regular replica set.

Best regards,
-Kal.

Comment by Mariano Benítez Mulet [ 02/May/18 ]

Sorry everyone for the late reply.

I see that you have already applied a fix.

In case you're still interested, I was using PHP 7.1, the latest MongoDB PHP extension and PHP library to do a replace op:

$collection->replaceOne(
            ['_id' => $id],
            [
                'name' => $name,
                'description' => $description,
                'address' => $address
            ],
            [
                'upsert' => true,
                'writeConcern' => new WriteConcern(2, 5)
            ]
        );
 

I see there is no 3.6.5 tag in your docker hub: https://hub.docker.com/r/library/mongo/tags/ but 3.6.4 is one day old.

Should I wait for the 3.6.5 or have you applied the patch to the 3.6.4 image?

Thanks a lot again and congratulations for your incredibly fast support.

Cheers

 

 

 

Comment by Blake Oler [ 01/May/18 ]

Fixed by placing the 4.0 solution in the 3.6 branch. See SERVER-32677.

Commit: https://github.com/mongodb/mongo/commit/9e4b78f198fa6a0bca75fd1012d8437d98a3c825

Comment by Kaloian Manassiev [ 30/Apr/18 ]

pachico, thank you again for the report!

This crash is definitely a bug on the MongoD side and we will be working on a fix. In the mean time, your usage scenario, where shard nodes have been started with --shardsvr and are used directly without having been added to a cluster through addShard is not something we expect to happen in practice and don't have tests for it.

So as a workaround until we fix this issue, either remove the --shardsvr parameter from the replica set or add it to an existing cluster and connect to it through mongos.

Best regards,
-Kal.

Comment by Kaloian Manassiev [ 30/Apr/18 ]

Thanks spencer.jackson for the detailed report and the repro!

Looks like this was introduced by blake.oler's commit to fix SERVER-32677.

blake.oler, your change deferred the creation of the LogicalSessionCache until sharding initialization, which opens a window where session requests sent against a --shardsvr server will not find the cache created and will hit the segmentation fault. I think we should revert your 3.6 change and solve it in 3.6 the same way it was solved in the 4.0 branch.

Comment by Spencer Jackson [ 30/Apr/18 ]

AuthorizationSessions are automatically created for Clients. Given both that, and that the condition on line 56 explicitly checks for the existence of the authSession before accessing it, It seems unlikely that the authSession could be NULL.

I can reproduce this behavior by running a replset with --shardsvr, creating a logical session, and performing an operation in it. I did not need to have a jsonSchema validated collection, or an index.

MongoDB Enterprise rs1:SECONDARY> var session = db._mongo.startSession();      
MongoDB Enterprise rs1:SECONDARY> session.getDatabase("test").people.insert({nam
e: "foo", description: "foo", address: "bar"})  

While testing in this mode, I observed segmentation faults while interacting with the LogicalSessionCache. It appears to be NULL. Because this appears to be logical sessions related, I am reassigning back to Sharding.

2018-04-30T14:21:56.134-0400 I ASIO     [NetworkInterfaceASIO-RS-0] Successfully connected to localhost:27018, took 3ms (2 connections now open to localhost:27018)Thread 73 "conn2" received signal SIGSEGV, Segmentation fault.                 
[Switching to Thread 0x7fffc6a54700 (LWP 9553)]
0x000055555a6dc831 in mongo::initializeOperationSessionInfo (opCtx=0x7fffecb33ca0, requestBody=..., requiresAuth=true, isReplSetMemberOrMongos=true, supportsDocLocking=true) at src/mongo/db/initialize_operation_session_info.cpp:69         
69              lsc->vivify(opCtx, opCtx->getLogicalSessionId().get());        
(gdb) p lsc
$2 = (mongo::LogicalSessionCache *) 0x0
 

pachico I notice that the servers are starting with --shardsvr, do you have any mongos processess in your cluster?  If no, removing –shardsvr prevented this from happening in my local testing.

Comment by Kaloian Manassiev [ 30/Apr/18 ]

pachico, would you happen to know what is the operation which gets executed, which causes this crash to happen? Also, which language driver are you using and what version?

 

Thank you in advance.

 

-Kal.

 

Comment by Kaloian Manassiev [ 30/Apr/18 ]

The parsed stack trace looks plausible, but it is not 100% correct because there is no code path which would lead from Client::lock to initializeOperationSessionInfo.

However, line 56 of initialize_operation_session_info.cpp is the authorization session check, so passing this on to the Platforms team, specifically spencer.jackson.

Comment by Mariano Benítez Mulet [ 30/Apr/18 ]

I can confirm this happens every time now.

mongo2    | 2018-04-30T16:48:06.193+0200 F -        [conn16] Got signal: 11 (Segmentation fault).
mongo2    | 
mongo2    |  0x56546875f2b1 0x56546875e4c9 0x56546875eb36 0x7fea079b0890 0x565468036593 0x5654671542c9 0x565467157d64 0x565467158aa0 0x56546716526a 0x565467160c27 0x565467164061 0x5654680632a2 0x56546715fa8f 0x565467161fd5 0x5654671628cb 0x565467160cad 0x565467164061 0x565468063805 0x56546861b584 0x7fea079a9064 0x7fea076de62d
mongo2    | ----- BEGIN BACKTRACE -----
mongo2    | {"backtrace":[{"b":"56546653B000","o":"22242B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"56546653B000","o":"22234C9"},{"b":"56546653B000","o":"2223B36"},{"b":"7FEA079A1000","o":"F890"},{"b":"56546653B000","o":"1AFB593","s":"_ZN5mongo30initializeOperationSessionInfoEPNS_16OperationContextERKNS_7BSONObjEbbb"},{"b":"56546653B000","o":"C192C9"},{"b":"56546653B000","o":"C1CD64"},{"b":"56546653B000","o":"C1DAA0","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"56546653B000","o":"C2A26A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"56546653B000","o":"C25C27","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"56546653B000","o":"C29061"},{"b":"56546653B000","o":"1B282A2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"56546653B000","o":"C24A8F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"56546653B000","o":"C26FD5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"56546653B000","o":"C278CB","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"56546653B000","o":"C25CAD","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"56546653B000","o":"C29061"},{"b":"56546653B000","o":"1B28805"},{"b":"56546653B000","o":"20E0584"},{"b":"7FEA079A1000","o":"8064"},{"b":"7FEA075F6000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.4", "gitVersion" : "d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "56546653B000", "elfType" : 3, "buildId" : "60A0D10C334A77CEE1D3027577BCC7851570B768" }, { "b" : "7FFFCE10A000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7FEA08B3E000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7FEA088DD000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7FEA084E1000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FD6376149047833953B0269E84DE181CA45DBE90" }, { "b" : "7FEA082D9000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7FEA080D5000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7FEA07DD4000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7FEA07BBE000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7FEA079A1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7FEA075F6000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7FEA08D55000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
mongo2    |  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56546875f2b1]
mongo2    |  mongod(+0x22234C9) [0x56546875e4c9]
mongo2    |  mongod(+0x2223B36) [0x56546875eb36]
mongo2    |  libpthread.so.0(+0xF890) [0x7fea079b0890]
mongo2    |  mongod(_ZN5mongo30initializeOperationSessionInfoEPNS_16OperationContextERKNS_7BSONObjEbbb+0x293) [0x565468036593]
mongo2    |  mongod(+0xC192C9) [0x5654671542c9]
mongo2    |  mongod(+0xC1CD64) [0x565467157d64]
mongo2    |  mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2B0) [0x565467158aa0]
mongo2    |  mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x56546716526a]
mongo2    |  mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x565467160c27]
mongo2    |  mongod(+0xC29061) [0x565467164061]
mongo2    |  mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x5654680632a2]
mongo2    |  mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x56546715fa8f]
mongo2    |  mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x565467161fd5]
mongo2    |  mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x5654671628cb]
mongo2    |  mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x565467160cad]
mongo2    |  mongod(+0xC29061) [0x565467164061]
mongo2    |  mongod(+0x1B28805) [0x565468063805]
mongo2    |  mongod(+0x20E0584) [0x56546861b584]
mongo2    |  libpthread.so.0(+0x8064) [0x7fea079a9064]
mongo2    |  libc.so.6(clone+0x6D) [0x7fea076de62d]
mongo2    | -----  END BACKTRACE  -----
 

Comment by Mariano Benítez Mulet [ 30/Apr/18 ]

I pruned the entire system and started from scratch.

I created the new replica set using 3.6.4 and now the primary crashes as soon as I attempt to write for the first time:

mongo1    | 2018-04-30T16:35:11.564+0200 F -        [conn23] Invalid access at address: 0
mongo1    | 2018-04-30T16:35:11.585+0200 F -        [conn23] Got signal: 11 (Segmentation fault).
mongo1    | 
mongo1    |  0x561973f2a2b1 0x561973f294c9 0x561973f29b36 0x7f0f2dcf6890 0x561973801593 0x56197291f2c9 0x561972922d64 0x561972923aa0 0x56197293026a 0x56197292bc27 0x56197292f061 0x56197382e2a2 0x56197292aa8f 0x56197292cfd5 0x56197292d8cb 0x56197292bcad 0x56197292f061 0x56197382e805 0x561973de6584 0x7f0f2dcef064 0x7f0f2da2462d
mongo1    | ----- BEGIN BACKTRACE -----
mongo1    | {"backtrace":[{"b":"561971D06000","o":"22242B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"561971D06000","o":"22234C9"},{"b":"561971D06000","o":"2223B36"},{"b":"7F0F2DCE7000","o":"F890"},{"b":"561971D06000","o":"1AFB593","s":"_ZN5mongo30initializeOperationSessionInfoEPNS_16OperationContextERKNS_7BSONObjEbbb"},{"b":"561971D06000","o":"C192C9"},{"b":"561971D06000","o":"C1CD64"},{"b":"561971D06000","o":"C1DAA0","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"561971D06000","o":"C2A26A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"561971D06000","o":"C25C27","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"561971D06000","o":"C29061"},{"b":"561971D06000","o":"1B282A2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"561971D06000","o":"C24A8F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"561971D06000","o":"C26FD5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"561971D06000","o":"C278CB","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"561971D06000","o":"C25CAD","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"561971D06000","o":"C29061"},{"b":"561971D06000","o":"1B28805"},{"b":"561971D06000","o":"20E0584"},{"b":"7F0F2DCE7000","o":"8064"},{"b":"7F0F2D93C000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.4", "gitVersion" : "d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "561971D06000", "elfType" : 3, "buildId" : "60A0D10C334A77CEE1D3027577BCC7851570B768" }, { "b" : "7FFC79540000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7F0F2EE84000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7F0F2EC23000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7F0F2E827000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FD6376149047833953B0269E84DE181CA45DBE90" }, { "b" : "7F0F2E61F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7F0F2E41B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7F0F2E11A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7F0F2DF04000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7F0F2DCE7000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7F0F2D93C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7F0F2F09B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
mongo1    |  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x561973f2a2b1]
mongo1    |  mongod(+0x22234C9) [0x561973f294c9]
mongo1    |  mongod(+0x2223B36) [0x561973f29b36]
mongo1    |  libpthread.so.0(+0xF890) [0x7f0f2dcf6890]
mongo1    |  mongod(_ZN5mongo30initializeOperationSessionInfoEPNS_16OperationContextERKNS_7BSONObjEbbb+0x293) [0x561973801593]
mongo1    |  mongod(+0xC192C9) [0x56197291f2c9]
mongo1    |  mongod(+0xC1CD64) [0x561972922d64]
mongo1    |  mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2B0) [0x561972923aa0]
mongo1    |  mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x56197293026a]
mongo1    |  mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x56197292bc27]
mongo1    |  mongod(+0xC29061) [0x56197292f061]
mongo1    |  mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x56197382e2a2]
mongo1    |  mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x56197292aa8f]
mongo1    |  mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x56197292cfd5]
mongo1    |  mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x56197292d8cb]
mongo1    |  mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x56197292bcad]
mongo1    |  mongod(+0xC29061) [0x56197292f061]
mongo1    |  mongod(+0x1B28805) [0x56197382e805]
mongo1    |  mongod(+0x20E0584) [0x561973de6584]
mongo1    |  libpthread.so.0(+0x8064) [0x7f0f2dcef064]
mongo1    |  libc.so.6(clone+0x6D) [0x7f0f2da2462d]
mongo1    | -----  END BACKTRACE  -----
 
 

Comment by Mariano Benítez Mulet [ 30/Apr/18 ]

I upgraded to 3.6.4 and started the replica set.

As soon as I launch a write operation the primary dies:

mongo3            | 2018-04-30T14:39:40.823+0200 F -        [conn19] Invalid access at address: 0
mongo3            | 2018-04-30T14:39:40.841+0200 F -        [conn19] Got signal: 11 (Segmentation fault).
mongo3            | 
mongo3            |  0x55d96a2172b1 0x55d96a2164c9 0x55d96a216b36 0x7fcb1675b890 0x55d969aee593 0x55d968c0c2c9 0x55d968c0fd64 0x55d968c10aa0 0x55d968c1d26a 0x55d968c18c27 0x55d968c1c061 0x55d969b1b2a2 0x55d968c17a8f 0x55d968c19fd5 0x55d968c1a8cb 0x55d968c18cad 0x55d968c1c061 0x55d969b1b805 0x55d96a0d3584 0x7fcb16754064 0x7fcb1648962d
mongo3            | ----- BEGIN BACKTRACE -----
mongo3            | {"backtrace":[{"b":"55D967FF3000","o":"22242B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55D967FF3000","o":"22234C9"},{"b":"55D967FF3000","o":"2223B36"},{"b":"7FCB1674C000","o":"F890"},{"b":"55D967FF3000","o":"1AFB593","s":"_ZN5mongo30initializeOperationSessionInfoEPNS_16OperationContextERKNS_7BSONObjEbbb"},{"b":"55D967FF3000","o":"C192C9"},{"b":"55D967FF3000","o":"C1CD64"},{"b":"55D967FF3000","o":"C1DAA0","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55D967FF3000","o":"C2A26A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55D967FF3000","o":"C25C27","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55D967FF3000","o":"C29061"},{"b":"55D967FF3000","o":"1B282A2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55D967FF3000","o":"C24A8F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55D967FF3000","o":"C26FD5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55D967FF3000","o":"C278CB","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55D967FF3000","o":"C25CAD","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55D967FF3000","o":"C29061"},{"b":"55D967FF3000","o":"1B28805"},{"b":"55D967FF3000","o":"20E0584"},{"b":"7FCB1674C000","o":"8064"},{"b":"7FCB163A1000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.4", "gitVersion" : "d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-112-generic", "version" : "#135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55D967FF3000", "elfType" : 3, "buildId" : "60A0D10C334A77CEE1D3027577BCC7851570B768" }, { "b" : "7FFE053F2000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "FA97F4849697BBE252BA1F7FB2316979E93E61DE" }, { "b" : "7FCB178E9000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7FCB17688000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7FCB1728C000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FD6376149047833953B0269E84DE181CA45DBE90" }, { "b" : "7FCB17084000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7FCB16E80000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7FCB16B7F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7FCB16969000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7FCB1674C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7FCB163A1000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7FCB17B00000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
mongo3            |  mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55d96a2172b1]
mongo3            |  mongod(+0x22234C9) [0x55d96a2164c9]
mongo3            |  mongod(+0x2223B36) [0x55d96a216b36]
mongo3            |  libpthread.so.0(+0xF890) [0x7fcb1675b890]
mongo3            |  mongod(_ZN5mongo30initializeOperationSessionInfoEPNS_16OperationContextERKNS_7BSONObjEbbb+0x293) [0x55d969aee593]
mongo3            |  mongod(+0xC192C9) [0x55d968c0c2c9]
mongo3            |  mongod(+0xC1CD64) [0x55d968c0fd64]
mongo3            |  mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2B0) [0x55d968c10aa0]
mongo3            |  mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55d968c1d26a]
mongo3            |  mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55d968c18c27]
mongo3            |  mongod(+0xC29061) [0x55d968c1c061]
mongo3            |  mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x55d969b1b2a2]
mongo3            |  mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x55d968c17a8f]
mongo3            |  mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x55d968c19fd5]
mongo3            |  mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x55d968c1a8cb]
mongo3            |  mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55d968c18cad]
mongo3            |  mongod(+0xC29061) [0x55d968c1c061]
mongo3            |  mongod(+0x1B28805) [0x55d969b1b805]
mongo3            |  mongod(+0x20E0584) [0x55d96a0d3584]
mongo3            |  libpthread.so.0(+0x8064) [0x7fcb16754064]
mongo3            |  libc.so.6(clone+0x6D) [0x7fcb1648962d]
mongo3            | -----  END BACKTRACE  -----
 

 

It the storage not compatible between 3.6.3 and 3.6.4 or should I start with no previous data stored?

 

Comment by Daniel Pasette (Inactive) [ 30/Apr/18 ]

Thanks for the report. 
This looks like a duplicate of SERVER-32677.  Can you try v3.6.4?

Generated at Thu Feb 08 04:37:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.