[SERVER-21550] Invariant failure details src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 732 Created: 19/Nov/15  Updated: 10/Feb/16  Resolved: 10/Feb/16

Status: Closed
Project: Core Server
Component/s: Admin, Storage
Affects Version/s: 3.0.5, 3.0.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Peter Salhofer Assignee: Kelsey Schubert
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Kind of impossible. Was't even aware that the server went down or was rebooted.

Participants:

 Description   

This is a copy of https://jira.mongodb.org/browse/SERVER-20304 but this time on Ubuntu



 Comments   
Comment by Kelsey Schubert [ 23/Dec/15 ]

Hi psalhofer,

We are still need additional information to diagnose this problem:

  • What filesystem are you using for your dbpath?
  • Can you provide any additional details on how the server was rebooted?

Thank you,
Thomas

Comment by Ramon Fernandez Marina [ 19/Nov/15 ]

From SERVER-20304:


Just upgrade to version 3.0.7 and now the log looks as follows:

2015-11-19T09:26:33.346+0100 I CONTROL  ***** SERVER RESTARTED *****
2015-11-19T09:26:33.405+0100 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2015-11-19T09:26:33.425+0100 I JOURNAL  [initandlisten] journal dir=/var/lib/mongodb/journal
2015-11-19T09:26:33.425+0100 I JOURNAL  [initandlisten] recover begin
2015-11-19T09:26:33.425+0100 I JOURNAL  [initandlisten] info no lsn file in journal/ directory
2015-11-19T09:26:33.425+0100 I JOURNAL  [initandlisten] recover lsn: 0
2015-11-19T09:26:33.425+0100 I JOURNAL  [initandlisten] recover /var/lib/mongodb/journal/j._0
2015-11-19T09:26:33.426+0100 I JOURNAL  [initandlisten] recover cleaning up
2015-11-19T09:26:33.426+0100 I JOURNAL  [initandlisten] removeJournalFiles
2015-11-19T09:26:33.621+0100 I JOURNAL  [initandlisten] recover done
2015-11-19T09:26:33.621+0100 I JOURNAL  [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2015-11-19T09:26:39.143+0100 I JOURNAL  [durability] Durability thread started
2015-11-19T09:26:39.144+0100 I JOURNAL  [journal writer] Journal writer thread started
2015-11-19T09:26:39.146+0100 I CONTROL  [initandlisten] MongoDB starting : pid=28783 port=27017 dbpath=/var/lib/mongodb 64-bit host=tobis
2015-11-19T09:26:39.147+0100 I CONTROL  [initandlisten] db version v3.0.7
2015-11-19T09:26:39.147+0100 I CONTROL  [initandlisten] git version: 6ce7cbe8c6b899552dadd907604559806aa2e9bd
2015-11-19T09:26:39.147+0100 I CONTROL  [initandlisten] build info: Linux ip-10-229-88-125 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-11-19T09:26:39.147+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2015-11-19T09:26:39.147+0100 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2015-11-19T09:26:39.205+0100 I -        [initandlisten] Invariant failure _name == nsToDatabaseSubstring(ns) src/mongo/db/catalog/database.cpp 419
2015-11-19T09:26:39.216+0100 I CONTROL  [initandlisten] 
 0xf5c3e9 0xefb231 0xedfd52 0x91c662 0x91c75f 0x91e6d3 0x921240 0x8092a5 0x7d6db9 0x7f0d427b7ec5 0x806677
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B5C3E9"},{"b":"400000","o":"AFB231"},{"b":"400000","o":"ADFD52"},{"b":"400000","o":"51C662"},{"b":"400000","o":"51C75F"},{"b":"400000","o":"51E6D3"},{"b":"400000","o":"521240"},{"b":"400000","o":"4092A5"},{"b":"400000","o":"3D6DB9"},{"b":"7F0D42796000","o":"21EC5"},{"b":"400000","o":"406677"}],"processInfo":{ "mongodbVersion" : "3.0.7", "gitVersion" : "6ce7cbe8c6b899552dadd907604559806aa2e9bd", "uname" : { "sysname" : "Linux", "release" : "3.13.0-66-generic", "version" : "#108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "03104A2519911F189AABD1EBC9931E8EBE9AD44C" }, { "b" : "7FFE798D1000", "elfType" : 3, "buildId" : "64F93A900F0AA19FFBFF98876568D341D09FF276" }, { "b" : "7F0D43DC1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F0D43B62000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "A20EFFEC993A8441FA17F2079F923CBD04079E19" }, { "b" : "7F0D43787000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F000D29917E9B6E94A35A8F02E5C62846E5916BC" }, { "b" : "7F0D4357F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F0D4337B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F0D43077000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }, { "b" : "7F0D42D71000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F0D42B5B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F0D42796000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F0D43FDF000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5c3e9]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefb231]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xedfd52]
 mongod(_ZNK5mongo8Database13getCollectionERKNS_10StringDataE+0x42) [0x91c662]
 mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE+0x1F) [0x91c75f]
 mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE+0x1E3) [0x91e6d3]
 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x150) [0x921240]
 mongod(_ZN5mongo13initAndListenEi+0xE65) [0x8092a5]
 mongod(main+0x139) [0x7d6db9]
 libc.so.6(__libc_start_main+0xF5) [0x7f0d427b7ec5]
 mongod(+0x406677) [0x806677]
-----  END BACKTRACE  -----
2015-11-19T09:26:39.216+0100 I -        [initandlisten] 
 
***aborting after invariant() failure

Comment by Ramon Fernandez Marina [ 19/Nov/15 ]

From SERVER-20304:


Hi!

I'm having exactly the same problem, however on an Ubuntu server, so it would be great if you could re-open this issue.

2015-11-19T08:23:22.645+0100 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2015-11-19T08:23:22.676+0100 I JOURNAL  [initandlisten] journal dir=/var/lib/mongodb/journal
2015-11-19T08:23:22.677+0100 I JOURNAL  [initandlisten] recover begin
2015-11-19T08:23:22.677+0100 I JOURNAL  [initandlisten] info no lsn file in journal/ directory
2015-11-19T08:23:22.677+0100 I JOURNAL  [initandlisten] recover lsn: 0
2015-11-19T08:23:22.677+0100 I JOURNAL  [initandlisten] recover /var/lib/mongodb/journal/j._0
2015-11-19T08:23:22.686+0100 I JOURNAL  [initandlisten] recover cleaning up
2015-11-19T08:23:22.686+0100 I JOURNAL  [initandlisten] removeJournalFiles
2015-11-19T08:23:22.959+0100 I JOURNAL  [initandlisten] recover done
2015-11-19T08:23:22.959+0100 I JOURNAL  [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2015-11-19T08:23:27.562+0100 I JOURNAL  [durability] Durability thread started
2015-11-19T08:23:27.563+0100 I JOURNAL  [journal writer] Journal writer thread started
2015-11-19T08:23:27.566+0100 I CONTROL  [initandlisten] MongoDB starting : pid=1231 port=27017 dbpath=/var/lib/mongodb 64-bit host=tobis
2015-11-19T08:23:27.566+0100 I CONTROL  [initandlisten] db version v3.0.5
2015-11-19T08:23:27.566+0100 I CONTROL  [initandlisten] git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3
2015-11-19T08:23:27.566+0100 I CONTROL  [initandlisten] build info: Linux ip-10-183-35-50 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-11-19T08:23:27.566+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2015-11-19T08:23:27.566+0100 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2015-11-19T08:23:27.582+0100 I -        [initandlisten] Invariant failure details src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 732
2015-11-19T08:23:27.595+0100 I CONTROL  [initandlisten] 
 0xf5ba59 0xefa911 0xedf432 0xd3e0af 0xd3e65f 0xd3edcf 0xd41fff 0x921155 0x808f61 0x7d6ad9 0x7feb11e0cec5 0x806597
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B5BA59"},{"b":"400000","o":"AFA911"},{"b":"400000","o":"ADF432"},{"b":"400000","o":"93E0AF"},{"b":"400000","o":"93E65F"},{"b":"400000","o":"93EDCF"},{"b":"400000","o":"941FFF"},{"b":"400000","o":"521155"},{"b":"400000","o":"408F61"},{"b":"400000","o":"3D6AD9"},{"b":"7FEB11DEB000","o":"21EC5"},{"b":"400000","o":"406597"}],"processInfo":{ "mongodbVersion" : "3.0.5", "gitVersion" : "8bc4ae20708dbb493cb09338d9e7be6698e4a3a3", "uname" : { "sysname" : "Linux", "release" : "3.13.0-66-generic", "version" : "#108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "695FC6828398A9DB1F99718671147885B5ED116D" }, { "b" : "7FFDBEA95000", "elfType" : 3, "buildId" : "64F93A900F0AA19FFBFF98876568D341D09FF276" }, { "b" : "7FEB13416000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7FEB131B7000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "A20EFFEC993A8441FA17F2079F923CBD04079E19" }, { "b" : "7FEB12DDC000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F000D29917E9B6E94A35A8F02E5C62846E5916BC" }, { "b" : "7FEB12BD4000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FEB129D0000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FEB126CC000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }, { "b" : "7FEB123C6000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7FEB121B0000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7FEB11DEB000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7FEB13634000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5ba59]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefa911]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xedf432]
 mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry14_insertInCacheEPNS_16OperationContextERKNS_10StringDataEPNS0_5EntryE+0x32F) [0xd3e0af]
 mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry5_initEPNS_16OperationContextE+0x4EF) [0xd3e65f]
 mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntryC1EPNS_16OperationContextERKNS_10StringDataES5_bb+0x16F) [0xd3edcf]
 mongod(_ZN5mongo12MMAPV1Engine23getDatabaseCatalogEntryEPNS_16OperationContextERKNS_10StringDataE+0x1AF) [0xd41fff]
 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x105) [0x921155]
 mongod(_ZN5mongo13initAndListenEi+0xC01) [0x808f61]
 mongod(main+0x139) [0x7d6ad9]
 libc.so.6(__libc_start_main+0xF5) [0x7feb11e0cec5]
 mongod(+0x406597) [0x806597]
-----  END BACKTRACE  -----
2015-11-19T08:23:27.595+0100 I -        [initandlisten] 
 
***aborting after invariant() failure

/etc/mongod.conf

# mongod.conf
 
# Where to store the data.
 
# Note: if you run mongodb as a non-root user (recommended) you may
# need to create and set permissions for this directory manually,
# e.g., if the parent directory isn't mutable by the mongodb user.
dbpath=/var/lib/mongodb
 
#where to log
logpath=/var/log/mongodb/mongod.log
 
logappend=true
 
#port = 27017
 
# Listen to local interface only. Comment out to listen on all interfaces. 
bind_ip = 127.0.0.1
 
# Disables write-ahead journaling
# nojournal = true
 
# Enables periodic logging of CPU utilization and I/O wait
#cpu = true
 
# Turn on/off security.  Off is currently the default
#noauth = true
#auth = true
 
# Verbose logging output.
#verbose = true
 
# Inspect all client data for validity on receipt (useful for
# developing drivers)
#objcheck = true
 
# Enable db quota management
#quota = true
 
# Set oplogging level where n is
#   0=off (default)
#   1=W
#   2=R
#   3=both
#   7=W+some reads
#diaglog = 0
 
# Ignore query hints
#nohints = true
 
# Enable the HTTP interface (Defaults to port 28017).
#httpinterface = true
 
# Turns off server-side scripting.  This will result in greatly limited
# functionality
#noscripting = true
 
# Turns off table scans.  Any query that would do a table scan fails.
#notablescan = true
 
# Disable data file preallocation.
#noprealloc = true
 
# Specify .ns file size for new databases.
# nssize = <size>
 
# Replication Options
 
# in replicated mongo databases, specify the replica set name here
#replSet=setname
# maximum size in megabytes for replication operation log
#oplogSize=1024
# path to a key file storing authentication info for connections
# between replica set members
#keyFile=/path/to/keyfile

/etc/init/mongod.conf

# Ubuntu upstart file at /etc/init/mongod.conf
 
# Recommended ulimit values for mongod or mongos
# See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
#
limit fsize unlimited unlimited
limit cpu unlimited unlimited
limit as unlimited unlimited
limit nofile 64000 64000
limit rss unlimited unlimited
limit nproc 32000 32000
 
kill timeout 300 # wait 300s between SIGTERM and SIGKILL.
 
pre-start script
  DAEMONUSER=${DAEMONUSER:-mongodb}
  if [ ! -d /var/lib/mongod ]; then
    mkdir -p /var/lib/mongodb && chown mongodb:mongodb /var/lib/mongodb
  fi
  if [ ! -d /var/log/mongod ]; then
    mkdir -p /var/log/mongodb && chown mongodb:mongodb /var/log/mongodb
  fi
  touch /var/run/mongodb.pid
  chown $DAEMONUSER /var/run/mongodb.pid
end script
 
start on runlevel [2345]
stop on runlevel [06]
 
script
  ENABLE_MONGOD="yes"
  CONF=/etc/mongod.conf
  DAEMON=/usr/bin/mongod
  DAEMONUSER=${DAEMONUSER:-mongodb}
 
  if [ -f /etc/default/mongod ]; then . /etc/default/mongod; fi
 
  # Handle NUMA access to CPUs (SERVER-3574)
  # This verifies the existence of numactl as well as testing that the command works
  NUMACTL_ARGS="--interleave=all"
  if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
  then
    NUMACTL="$(which numactl) -- $NUMACTL_ARGS"
    DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}
  else
    NUMACTL=""
    DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}
  fi
 
  if [ "x$ENABLE_MONGOD" = "xyes" ]
  then
    exec start-stop-daemon --start \
        --chuid $DAEMONUSER \
        --pidfile /var/run/mongodb.pid \
        --make-pidfile \
        --exec $NUMACTL $DAEMON $DAEMON_OPTS
  fi
end script

Listing of dbpath:

drwxr-xr-x  3 mongodb mongodb      4096 Jul 29 08:12 .
drwxr-xr-x 41 root    root         4096 Jul 29 06:35 ..
-rw-------  1 mongodb mongodb  67108864 Nov  4 13:25 DigiCoaching.0
-rw-------  1 mongodb mongodb 134217728 Jul 28 15:26 DigiCoaching.1
-rw-------  1 mongodb mongodb  16777216 Nov  4 12:22 DigiCoaching.ns
drwxr-xr-x  2 mongodb mongodb      4096 Nov 19 08:23 journal
-rw-------  1 mongodb mongodb  67108864 Jul 29 08:12 local.0
-rw-------  1 mongodb mongodb  16777216 Jul 29 08:12 local.ns
-rwxr-xr-x  1 mongodb mongodb         5 Nov 19 08:23 mongod.lock
-rw-r--r--  1 mongodb nogroup        69 Jul 29 08:12 storage.bson

Starting mongod with -vvvvv:

/var/lib$ sudo mongod -vvvvvv -dbpath /var/lib/mongodb 
2015-11-19T08:45:11.082+0100 D SHARDING isInRangeTest passed
2015-11-19T08:45:11.083+0100 D NETWORK  [initandlisten] fd limit hard:4096 soft:1024 max conn: 819
2015-11-19T08:45:11.136+0100 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2015-11-19T08:45:11.152+0100 I JOURNAL  [initandlisten] journal dir=/var/lib/mongodb/journal
2015-11-19T08:45:11.153+0100 D COMMAND  [DataFileSync] BackgroundJob starting: DataFileSync
2015-11-19T08:45:11.153+0100 I JOURNAL  [initandlisten] recover begin
2015-11-19T08:45:11.153+0100 I JOURNAL  [initandlisten] info no lsn file in journal/ directory
2015-11-19T08:45:11.153+0100 I JOURNAL  [initandlisten] recover lsn: 0
2015-11-19T08:45:11.153+0100 I JOURNAL  [initandlisten] recover /var/lib/mongodb/journal/j._0
2015-11-19T08:45:11.159+0100 I JOURNAL  [initandlisten] recover cleaning up
2015-11-19T08:45:11.159+0100 I JOURNAL  [initandlisten] removeJournalFiles
2015-11-19T08:45:11.385+0100 D STORAGE  [initandlisten] flushing directory /var/lib/mongodb/journal
2015-11-19T08:45:11.400+0100 D JOURNAL  [initandlisten] removeJournalFiles end
2015-11-19T08:45:11.400+0100 I JOURNAL  [initandlisten] recover done
2015-11-19T08:45:11.400+0100 I JOURNAL  [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2015-11-19T08:45:15.855+0100 D STORAGE  [initandlisten] flushing directory /var/lib/mongodb/journal
2015-11-19T08:45:15.869+0100 D STORAGE  [initandlisten] flushing directory /var/lib/mongodb/journal
2015-11-19T08:45:15.872+0100 I JOURNAL  [durability] Durability thread started
2015-11-19T08:45:15.873+0100 I JOURNAL  [journal writer] Journal writer thread started
2015-11-19T08:45:15.877+0100 D STORAGE  [initandlisten] flushing directory /var/lib/mongodb
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] MongoDB starting : pid=1369 port=27017 dbpath=/var/lib/mongodb 64-bit host=tobis
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] 
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] db version v3.0.5
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] git version: 8bc4ae20708dbb493cb09338d9e7be6698e4a3a3
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] build info: Linux ip-10-183-35-50 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2015-11-19T08:45:15.877+0100 I CONTROL  [initandlisten] options: { storage: { dbPath: "/var/lib/mongodb" }, systemLog: { verbosity: 6 } }
2015-11-19T08:45:15.877+0100 D STORAGE  [initandlisten] enter repairDatabases (to check pdfile version #)
2015-11-19T08:45:15.877+0100 D STORAGE  [initandlisten]     Recovering database: local
2015-11-19T08:45:15.877+0100 D STORAGE  [initandlisten] mmf open /var/lib/mongodb/local.ns
2015-11-19T08:45:15.878+0100 D STORAGE  [initandlisten] mmf finishOpening 0x7fc4e20e0000 /var/lib/mongodb/local.ns len:16777216
2015-11-19T08:45:15.878+0100 D STORAGE  [initandlisten] mmf open /var/lib/mongodb/local.0
2015-11-19T08:45:15.878+0100 D STORAGE  [initandlisten] mmf finishOpening 0x7fc4dd0e0000 /var/lib/mongodb/local.0 len:67108864
2015-11-19T08:45:15.884+0100 D STORAGE  [initandlisten]    ***** ROLLING BACK 0 disk writes
2015-11-19T08:45:15.884+0100 D STORAGE  [initandlisten]    ***** ROLLING BACK 0 custom changes
2015-11-19T08:45:15.884+0100 D STORAGE  [initandlisten] local.startup_log: clearing plan cache - collection info cache reset
2015-11-19T08:45:15.884+0100 D STORAGE  [initandlisten] local.system.indexes: clearing plan cache - collection info cache reset
2015-11-19T08:45:15.884+0100 D STORAGE  [initandlisten] local.system.namespaces: clearing plan cache - collection info cache reset
2015-11-19T08:45:15.885+0100 D STORAGE  [initandlisten]     Recovering database: DigiCoaching
2015-11-19T08:45:15.885+0100 D STORAGE  [initandlisten] mmf open /var/lib/mongodb/DigiCoaching.ns
2015-11-19T08:45:15.885+0100 D STORAGE  [initandlisten] mmf finishOpening 0x7fc4d80e0000 /var/lib/mongodb/DigiCoaching.ns len:16777216
2015-11-19T08:45:15.885+0100 D STORAGE  [initandlisten] mmf open /var/lib/mongodb/DigiCoaching.0
2015-11-19T08:45:15.885+0100 D STORAGE  [initandlisten] mmf finishOpening 0x7fc4d30e0000 /var/lib/mongodb/DigiCoaching.0 len:67108864
2015-11-19T08:45:15.891+0100 I -        [initandlisten] Invariant failure details src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 732
2015-11-19T08:45:15.904+0100 I CONTROL  [initandlisten] 
 0xf5ba59 0xefa911 0xedf432 0xd3e0af 0xd3e65f 0xd3edcf 0xd41fff 0x921155 0x808f61 0x7d6ad9 0x7fc4e6544ec5 0x806597
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B5BA59"},{"b":"400000","o":"AFA911"},{"b":"400000","o":"ADF432"},{"b":"400000","o":"93E0AF"},{"b":"400000","o":"93E65F"},{"b":"400000","o":"93EDCF"},{"b":"400000","o":"941FFF"},{"b":"400000","o":"521155"},{"b":"400000","o":"408F61"},{"b":"400000","o":"3D6AD9"},{"b":"7FC4E6523000","o":"21EC5"},{"b":"400000","o":"406597"}],"processInfo":{ "mongodbVersion" : "3.0.5", "gitVersion" : "8bc4ae20708dbb493cb09338d9e7be6698e4a3a3", "uname" : { "sysname" : "Linux", "release" : "3.13.0-66-generic", "version" : "#108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "695FC6828398A9DB1F99718671147885B5ED116D" }, { "b" : "7FFD975A5000", "elfType" : 3, "buildId" : "64F93A900F0AA19FFBFF98876568D341D09FF276" }, { "b" : "7FC4E7B4E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7FC4E78EF000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "A20EFFEC993A8441FA17F2079F923CBD04079E19" }, { "b" : "7FC4E7514000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F000D29917E9B6E94A35A8F02E5C62846E5916BC" }, { "b" : "7FC4E730C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FC4E7108000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FC4E6E04000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }, { "b" : "7FC4E6AFE000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7FC4E68E8000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7FC4E6523000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7FC4E7D6C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf5ba59]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xefa911]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xedf432]
 mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry14_insertInCacheEPNS_16OperationContextERKNS_10StringDataEPNS0_5EntryE+0x32F) [0xd3e0af]
 mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry5_initEPNS_16OperationContextE+0x4EF) [0xd3e65f]
 mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntryC1EPNS_16OperationContextERKNS_10StringDataES5_bb+0x16F) [0xd3edcf]
 mongod(_ZN5mongo12MMAPV1Engine23getDatabaseCatalogEntryEPNS_16OperationContextERKNS_10StringDataE+0x1AF) [0xd41fff]
 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x105) [0x921155]
 mongod(_ZN5mongo13initAndListenEi+0xC01) [0x808f61]
 mongod(main+0x139) [0x7d6ad9]
 libc.so.6(__libc_start_main+0xF5) [0x7fc4e6544ec5]
 mongod(+0x406597) [0x806597]
-----  END BACKTRACE  -----
2015-11-19T08:45:15.904+0100 I -        [initandlisten] 
 
***aborting after invariant() failure

Server is standalone

Comment by Ramon Fernandez Marina [ 19/Nov/15 ]

Thanks for opening a new ticket psalhofer; I'll copy the information you provided here and here to this ticket so we can investigate.

A quick first inspection of that information seems to indicate that mongod found data corruption and shut itself down. This type of data corruption in MMAPv1 is typically caused by a failure at the storage layer, so I'd recommend you check the integrity of your storage first. Then the easier way to move forward is to resync this node from a healthy primary.

What filesystem are you using for your dbpath? Also can you provide any additional details on how the server was rebooted?

Thanks,
Ramón.

Generated at Thu Feb 08 03:57:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.