[SERVER-52970] Segmentation Fault Created: 20/Nov/20  Updated: 24/Nov/20  Resolved: 24/Nov/20

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: 4.0.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Roman Blachman Assignee: Edwin Zhou
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File _usr_bin_mongod.114.crash    
Operating System: ALL
Participants:

 Description   

Segmentation fault with 3 node replica-set

 

2020-11-20T16:37:33.008+0000 F -        [Replication] Invalid access at address: 0x55a418c295f0
2020-11-20T16:37:33.034+0000 F -        [Replication] Got signal: 11 (Segmentation fault).
 0x55a4193fe141 0x55a4193fd359 0x55a4193fd9c6 0x7f0f0711b890 0x55a418c295f0 0x55a418c68b3f 0x55a418e36039 0x55a418e36281 0x55a418e403fe 0x55a418c622fd 0x55a418c3c948 0x55a41950d710 0x7f0f071106db 0x7f0f06e3988f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55A416FE2000","o":"241C141","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55A416FE2000","o":"241B359"},{"b":"55A416FE2000","o":"241B9C6"},{"b":"7F0F07109000","o":"12890"},{"b":"55A416FE2000","o":"1C475F0"},{"b":"55A416FE2000","o":"1C86B3F","s":"_ZN4asio6detail18completion_handlerISt8functionIFvvEEE11do_completeEPvPNS0_19scheduler_operationERKSt10error_codem"},{"b":"55A416FE2000","o":"1E54039","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"55A416FE2000","o":"1E54281","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"55A416FE2000","o":"1E5E3FE","s":"_ZN4asio10io_context3runEv"},{"b":"55A416FE2000","o":"1C802FD","s":"_ZN5mongo9transport18TransportLayerASIO11ASIOReactor3runEv"},{"b":"55A416FE2000","o":"1C5A948","s":"_ZN5mongo8executor18NetworkInterfaceTL4_runEv"},{"b":"55A416FE2000","o":"252B710"},{"b":"7F0F07109000","o":"76DB"},{"b":"7F0F06D18000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.9", "gitVersion" : "fc525e2d9b0e4bceff5c2201457e564362909765", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-45-generic", "version" : "#48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55A416FE2000", "elfType" : 3, "buildId" : "33CAA38B52AFD2F0B2E7D8911E2B46B6C740A2C7" }, { "b" : "7FFEB2B8D000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "025AD7BDF9DF8C7251192246537C451A988B98AE" }, { "b" : "7F0F085E7000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "CFC1F864E5F85EDA8E37B3BB238EC438D6D524A5" }, { "b" : "7F0F083CC000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F0F07F54000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "A409BCD92DCDD3298E92C359876AC800FDD499AC" }, { "b" : "7F0F07CEA000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "9787FF1FFFE8D82268D89F6BFA3AF07B3E9D140F" }, { "b" : "7F0F07AE6000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F0F078DE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F0F07540000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F0F07328000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "92E0BE1929D28508CF9C6D5754C7EB48C12255B3" }, { "b" : "7F0F07109000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F0F06D18000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F0F08866000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F0F06AF3000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F0F068D6000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F0F066BA000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F0F064AC000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F0F06261000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F0F0600F000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "E8A505FFF184C54BB6652424E44B852FBE27AACD" }, { "b" : "7F0F05E01000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "51B4F4B81488AADD19CC5817B9ABBB5B94BDE07F" }, { "b" : "7F0F05BE4000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F0F05866000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "62A44D226F3A74CF720305A52D3E876E214C248A" }, { "b" : "7F0F05501000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "F67AAC89381F6ACFE9979B24D4002668C4F70ABE" }, { "b" : "7F0F052CD000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F0F05097000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F0F04E16000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F0F04B40000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F0F0490E000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F0F0470A000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "C0CB7E35A4566A443F99DFBC1A54D3A0677C8A10" }, { "b" : "7F0F044FF000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F0F042E4000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F0F040A3000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F0F03D74000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F0F03B61000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F0F0395D000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F0F03754000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F0F034C7000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F0F03225000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F0F02FEF000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F0F02DD9000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F0F02BD1000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "C114D2C23BD2F3B1705F37FBF9CA06163C8B89A6" }, { "b" : "7F0F029A8000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F0F02799000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F0F0254F000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F0F02246000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "9FD7B69263297314C83BB205234E182BD4D8B86A" }, { "b" : "7F0F0200E000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55a4193fe141]
 mongod(+0x241B359) [0x55a4193fd359]
 mongod(+0x241B9C6) [0x55a4193fd9c6]
 libpthread.so.0(+0x12890) [0x7f0f0711b890]
 mongod(+0x1C475F0) [0x55a418c295f0]
 mongod(_ZN4asio6detail18completion_handlerISt8functionIFvvEEE11do_completeEPvPNS0_19scheduler_operationERKSt10error_codem+0x7F) [0x55a418c68b3f]
 mongod(_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code+0x389) [0x55a418e36039]
 mongod(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x55a418e36281]
 mongod(_ZN4asio10io_context3runEv+0x3E) [0x55a418e403fe]
 mongod(_ZN5mongo9transport18TransportLayerASIO11ASIOReactor3runEv+0x3D) [0x55a418c622fd]
 mongod(_ZN5mongo8executor18NetworkInterfaceTL4_runEv+0xB8) [0x55a418c3c948]
 mongod(+0x252B710) [0x55a41950d710]
 libpthread.so.0(+0x76DB) [0x7f0f071106db]
 libc.so.6(clone+0x3F) [0x7f0f06e3988f]
-----  END BACKTRACE  -----

_usr_bin_mongod.114.crash



 Comments   
Comment by Edwin Zhou [ 24/Nov/20 ]

Hi rblachman@preempt.com,

Unfortunately, at this time we're unable to determine the cause of the behavior given the details you've provided. Thank you for your continued monitoring of this seg fault. We will appreciate additional diagnostic data you can collect so we can work to reproduce this issue.

Best,

Edwin

 

Comment by Roman Blachman [ 24/Nov/20 ]

Hello Edwin,
I don't have access to the diagnostic data anymore, since the VM was destroyed.
I will continue monitoring and see if this reproduces, and then collect the diagnostic.data.

Anything else we can do from the above details?

Thanks,
Roman

Comment by Edwin Zhou [ 23/Nov/20 ]

Hi rblachman@preempt.com,

Would you please archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described here) and attach it to this ticket? Would you also please attach the logs leading up to the segmentation fault?

Kind regards,
Edwin

Comment by Roman Blachman [ 20/Nov/20 ]

After mongod is restarted the following logs are printed:

 

2020-11-20T17:17:58.797+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2020-11-20T17:17:58.802+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] MongoDB starting : pid=32604 port=27017 dbpath=/var/lib/mongodb 64-bit host=management
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] db version v4.0.9
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] git version: fc525e2d9b0e4bceff5c2201457e564362909765
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] modules: none
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] build environment:
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten]     distarch: x86_64
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2020-11-20T17:17:58.806+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017, ssl: { CAFile: "/data/*.preempt.internal.crt", PEMKeyFile: "/data/server.pem", clusterFile: "/data/server.pem", mode: "requireSSL" } }, replication: { replSetName: "rs0" }, security: { authorization: "enabled", clusterAuthMode: "x509" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/var/log/mongodb/mongod.log" } }
2020-11-20T17:17:58.808+0000 W STORAGE  [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2020-11-20T17:17:58.818+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-11-20T17:17:58.818+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2020-11-20T17:17:58.818+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=15572M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-11-20T17:17:59.783+0000 I STORAGE  [initandlisten] WiredTiger message [1605892679:783391][32604:0x7f444fbe7a40], txn-recover: Main recovery loop: starting at 202/48679808 to 203/256
2020-11-20T17:17:59.783+0000 I STORAGE  [initandlisten] WiredTiger message [1605892679:783970][32604:0x7f444fbe7a40], txn-recover: Recovering log 202 through 203
2020-11-20T17:17:59.894+0000 I STORAGE  [initandlisten] WiredTiger message [1605892679:894505][32604:0x7f444fbe7a40], file:collection-12--194487718431950959.wt, txn-recover: Recovering log 203 through 203
2020-11-20T17:17:59.979+0000 I STORAGE  [initandlisten] WiredTiger message [1605892679:979645][32604:0x7f444fbe7a40], file:collection-12--194487718431950959.wt, txn-recover: Set global recovery timestamp: 5fb7f0c700000172
2020-11-20T17:18:00.239+0000 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1605890247, 370)
2020-11-20T17:18:00.239+0000 I STORAGE  [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1605890247, 370) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1605890247, 370)
2020-11-20T17:18:00.719+0000 I STORAGE  [initandlisten] Starting OplogTruncaterThread local.oplog.rs
2020-11-20T17:18:00.720+0000 I STORAGE  [initandlisten] The size storer reports that the oplog contains 17739217 records totaling to 11261947776 bytes
2020-11-20T17:18:00.720+0000 I STORAGE  [initandlisten] Sampling from the oplog between Nov 20 08:49:35:3 and Nov 20 16:37:32:2 to determine where to place markers for truncation
2020-11-20T17:18:00.720+0000 I STORAGE  [initandlisten] Taking 990 samples and assuming that each section of oplog contains approximately 179124 records totaling to 113718950 bytes
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 10:36:52:197
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:25:02:5739
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:29:57:78
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:34:01:241
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:36:50:143
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:39:56:315
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:44:07:7543
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:47:17:268
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:49:53:83
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:53:48:241
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:56:51:366
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 12:59:46:227
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:03:23:139
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:05:17:15883
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:08:53:282
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:11:18:166
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:14:15:2755
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:17:12:128
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:19:37:13328
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:23:35:286
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:25:36:255
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:29:11:4519
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:32:14:321
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:34:56:19303
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:38:13:358
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:40:08:7511
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:44:08:693
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:46:46:38
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:49:19:11573
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:53:07:218
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:55:13:195
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 13:59:00:358
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:01:23:342
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:04:24:75
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:07:18:122
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:09:43:11903
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:12:57:465
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:14:53:11159
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:18:26:284
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:20:16:7
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:24:06:1167
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:26:25:16
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:29:15:11485
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:32:13:171
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:34:34:375
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:38:07:119
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:39:45:108
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:43:33:95
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:45:34:85
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:48:44:75
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:50:35:76
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:54:03:9645
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:56:47:222
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 14:59:21:51
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:02:08:256
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:04:23:679
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:05:52:40
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:08:28:78
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:09:49:95
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:11:36:244
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:14:05:5768
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:15:04:19
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:17:03:188
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:19:06:10742
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:19:58:250
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:22:18:340
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:24:22:153
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:26:03:150
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:28:21:79
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:29:36:11967
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:31:28:25
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:33:46:166
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:34:48:21133
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:37:24:551
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:39:14:7744
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:41:13:556
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:44:03:5998
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:45:02:8115
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:47:35:270
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:49:31:181
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:51:11:32
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:53:56:313
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:55:09:405
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:57:33:76
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 15:59:18:4801
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:01:07:327
2020-11-20T17:18:00.856+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:03:37:190
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:04:40:388
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:08:50:29
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:10:58:262
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:14:10:4108
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:17:07:243
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:19:46:9995
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:23:00:224
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:25:19:315
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:29:05:1119
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:31:16:38
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:34:13:7774
2020-11-20T17:18:00.857+0000 I STORAGE  [initandlisten] Placing a marker at optime Nov 20 16:37:24:82
2020-11-20T17:18:02.518+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
2020-11-20T17:18:02.520+0000 I REPL     [initandlisten] Did not find local voted for document at startup.
2020-11-20T17:18:02.522+0000 I REPL     [initandlisten] Rollback ID is 1
2020-11-20T17:18:02.523+0000 I REPL     [initandlisten] Recovering from stable timestamp: Timestamp(1605890247, 370) (top of oplog: { ts: Timestamp(1605890252, 2), t: 1 }, appliedThrough: { ts: Timestamp(1605890247, 370), t: 1 }, TruncateAfter: Timestamp(0, 0))
2020-11-20T17:18:02.523+0000 I REPL     [initandlisten] Starting recovery oplog application at the stable timestamp: Timestamp(1605890247, 370)
2020-11-20T17:18:02.523+0000 I REPL     [initandlisten] Replaying stored operations from { : Timestamp(1605890247, 370) } (exclusive) to { : Timestamp(1605890252, 2) } (inclusive).
2020-11-20T17:18:02.531+0000 I CONTROL  [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured
2020-11-20T17:18:02.531+0000 I NETWORK  [initandlisten] waiting for connections on port 27017 ssl
2020-11-20T17:18:02.554+0000 I REPL     [replexec-0] New replica set config in use: { _id: "rs0", version: 3, protocolVersion: 1, writeConcernMajorityJournalDefault: true, members: [ { _id: 0, host: "10.6.0.74:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "10.6.0.75:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 2, host: "10.6.0.76:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5fb7198854038a3bbb1d7272') } }
2020-11-20T17:18:02.554+0000 I REPL     [replexec-0] This node is 10.6.0.75:27017 in the config
2020-11-20T17:18:02.554+0000 I REPL     [replexec-0] transition to STARTUP2 from STARTUP
2020-11-20T17:18:02.554+0000 I REPL     [replexec-0] Starting replication storage threads
2020-11-20T17:18:02.555+0000 I ASIO     [Replication] Connecting to 10.6.0.74:27017
2020-11-20T17:18:02.555+0000 I ASIO     [Replication] Connecting to 10.6.0.76:27017
2020-11-20T17:18:02.555+0000 I REPL     [replexec-0] transition to RECOVERING from STARTUP2
2020-11-20T17:18:02.555+0000 I REPL     [replexec-0] Starting replication fetcher thread
2020-11-20T17:18:02.555+0000 I REPL     [replexec-0] Starting replication applier thread
2020-11-20T17:18:02.555+0000 I REPL     [replexec-0] Starting replication reporter thread
2020-11-20T17:18:02.555+0000 I REPL     [rsSync-0] Starting oplog application
2020-11-20T17:18:02.556+0000 I REPL     [rsSync-0] transition to SECONDARY from RECOVERING
2020-11-20T17:18:02.556+0000 I REPL     [rsSync-0] Resetting sync source to empty, which was :27017
2020-11-20T17:18:02.568+0000 I REPL     [replexec-1] Member 10.6.0.76:27017 is now in state SECONDARY
2020-11-20T17:18:02.569+0000 I REPL     [replexec-2] Member 10.6.0.74:27017 is now in state PRIMARY
2020-11-20T17:18:02.653+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:55380 #5 (1 connection now open)
2020-11-20T17:18:02.662+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:55382 #6 (2 connections now open)
2020-11-20T17:18:02.673+0000 I NETWORK  [conn5] received client metadata from 127.0.0.1:55380 conn5: { driver: { name: "mongo-java-driver", version: "3.6.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-45-generic" }, platform: "Java/AdoptOpenJDK/11.0.8+10", application: { name: "admin" } }
2020-11-20T17:18:02.673+0000 E NETWORK  [conn6] SSL peer certificate validation failed: certificate signature failure
2020-11-20T17:18:02.674+0000 I NETWORK  [conn6] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: certificate signature failure. Ending connection from 127.0.0.1:55382 (connection id: 6)
2020-11-20T17:18:02.674+0000 I NETWORK  [conn6] end connection 127.0.0.1:55382 (1 connection now open)
2020-11-20T17:18:02.695+0000 I ACCESS   [conn5] Successfully authenticated as principal admin on admin
2020-11-20T17:18:03.008+0000 I FTDC     [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2020-11-20T17:18:03.050+0000 I NETWORK  [listener] connection accepted from 10.6.0.74:47152 #7 (2 connections now open)
2020-11-20T17:18:03.062+0000 I NETWORK  [conn7] received client metadata from 10.6.0.74:47152 conn7: { driver: { name: "NetworkInterfaceTL", version: "4.0.9" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "18.04" } }
2020-11-20T17:18:03.065+0000 I ACCESS   [conn7]  authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=*,O=Preempt Security,L=Ramat Gan,ST=Israel,C=IL", $db: "$external" }
2020-11-20T17:18:03.556+0000 I REPL     [rsBackgroundSync] sync source candidate: 10.6.0.76:27017
2020-11-20T17:18:03.556+0000 I ASIO     [RS] Connecting to 10.6.0.76:27017
2020-11-20T17:18:03.566+0000 I REPL     [rsBackgroundSync] Changed sync source from empty to 10.6.0.76:27017
2020-11-20T17:18:03.566+0000 I ASIO     [RS] Connecting to 10.6.0.76:27017
2020-11-20T17:18:03.675+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:55388 #10 (3 connections now open)
2020-11-20T17:18:03.684+0000 E NETWORK  [conn10] SSL peer certificate validation failed: certificate signature failure
2020-11-20T17:18:03.684+0000 I NETWORK  [conn10] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: certificate signature failure. Ending connection from 127.0.0.1:55388 (connection id: 10)
2020-11-20T17:18:03.684+0000 I NETWORK  [conn10] end connection 127.0.0.1:55388 (2 connections now open)

Generated at Thu Feb 08 05:29:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.