root@server:/home/server# /usr/lib/juju/mongo3.2/bin/mongod --repair --dbpath '/var/lib/juju/db'
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] MongoDB starting : pid=1611 port=27017 dbpath=/var/lib/juju/db 64-bit host=hqosjuju
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] db version v3.2.15
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] git version: e11e3c1b9c9ce3f7b4a79493e16f5e4504e01140
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] allocator: tcmalloc
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] modules: none
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] build environment:
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] distarch: x86_64
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] target_arch: x86_64
2019-02-27T20:25:02.925+0000 I CONTROL [initandlisten] options:
Unknown macro: { repair}
}
2019-02-27T20:25:02.945+0000 I - [initandlisten] Detected data files in /var/lib/juju/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-02-27T20:25:02.945+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/juju/db/mongod.lock is not empty.
2019-02-27T20:25:02.945+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-02-27T20:25:02.945+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2019-02-27T20:25:02.945+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2019-02-27T20:25:03.018+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2019-02-27T20:25:03.058+0000 I STORAGE [initandlisten] Repairing size cache
2019-02-27T20:25:03.059+0000 I STORAGE [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging.
2019-02-27T20:25:03.060+0000 I STORAGE [initandlisten] Repairing catalog metadata
2019-02-27T20:25:03.062+0000 I STORAGE [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2019-02-27T20:25:03.068+0000 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-02-27T20:25:03.068+0000 I CONTROL [initandlisten]
2019-02-27T20:25:03.068+0000 I STORAGE [initandlisten] repairDatabase admin
2019-02-27T20:25:03.069+0000 I STORAGE [initandlisten] Repairing collection admin.system.users
2019-02-27T20:25:03.069+0000 I STORAGE [initandlisten] Verify failed on uri table:collection-13-8833743074834640358. Running a salvage operation.
2019-02-27T20:25:03.069+0000 I - [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 190
2019-02-27T20:25:03.069+0000 I - [initandlisten]
***aborting after invariant() failure
2019-02-27T20:25:03.072+0000 F - [initandlisten] Got signal: 6 (Aborted).
0x12a7701 0x12a6559 0x12a6e81 0x7f02ae3fa390 0x7f02ae054428 0x7f02ae05602a 0x12208c4 0x8b6680 0x8bdb30 0x8c09b0 0xd1f0b1 0x739325 0x73c4c0 0x6f73a2 0x7f02ae03f830 0x736e99
----- BEGIN BACKTRACE -----
Unknown macro: {"backtrace"}
,{"b":"400000","o":"EA6559"},{"b":"400000","o":"EA6E81"},{"b":"7F02AE3E9000","o":"11390"},{"b":"7F02AE01F000","o":"35428","s":"gsignal"},{"b":"7F02AE01F000","o":"3702A","s":"abort"},
Unknown macro: {"b"}
,{"b":"400000","o":"4B6680","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE"},{"b":"400000","o":"4BDB30","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"4C09B0","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb"},
Unknown macro: {"b"}
,{"b":"400000","o":"339325"},{"b":"400000","o":"33C4C0"},{"b":"400000","o":"2F73A2","s":"main"},
Unknown macro: {"b"}
,{"b":"400000","o":"336E99","s":"_start"}],"processInfo":
Unknown macro: { "mongodbVersion" }
, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "154504DC4E334AF6C6D92A625384E0136F3E5837" }, { "b" : "7FFF8065D000", "elfType" : 3, "buildId" : "EFD9CD6D855DEE3F29B3F5CA8D2C6C1C2B7D7CAB" }, { "b" : "7F02B0FD9000", "path" : "/usr/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "C376C112685221C43033ED32DDD1E138B658483B" }, { "b" : "7F02B0DBF000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7F02B0BB7000", "path" : "/usr/lib/x86_64-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "CE5C34DFF6A98121F82EED45478616455801DB1D" }, { "b" : "7F02B0939000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0", "elfType" : 3, "buildId" : "9F70D8EB5739EEF251E6DE6DED289027DA61844E" }, { "b" : "7F02B0721000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0", "elfType" : 3, "buildId" : "FC0239AC1E59EB10991A2423B48986ADB010F9D3" }, { "b" : "7F02B04FB000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0", "elfType" : 3, "buildId" : "93798D047035F12B0AA8DE4DDF45DA8636519986" }, { "b" : "7F02B02F7000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0", "elfType" : 3, "buildId" : "3EBF263E88DAE32EE640C9C599A9D7DD59F3DF28" }, { "b" : "7F02B00EF000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0", "elfType" : 3, "buildId" : "34C98695297484FCDE0AD442BEB8412F1B787CE7" }, { "b" : "7F02AFDE7000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_regex.so.1.58.0", "elfType" : 3, "buildId" : "FF52EBC3EA55DD649B9A6D65D8AC7D73CF2B99D7" }, { "b" : "7F02AFBDE000", "path" : "/usr/lib/x86_64-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "96888DAFDBE35DEDCAE309DB9C0943C2C2271EB1" }, { "b" : "7F02AF961000", "path" : "/usr/lib/x86_64-linux-gnu/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "D163879ED83585A5B2B58D41CDCB78ABBBD351C5" }, { "b" : "7F02AF6F8000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "473092A9AF373FB0CAB555F9A003BC67F47756B6" }, { "b" : "7F02AF2B3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "8942CA58A3B910E883CC31E04A23DBD09729B4B0" }, { "b" : "7F02AF0AB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "69143E8B39040C964D3958490535322675F15DD3" }, { "b" : "7F02AEEA7000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "37BFC3D8F7E3B022DAC7943B1A5FACD40CEBF0AD" }, { "b" : "7F02AEB25000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "C5CA582A8E1EE50F59CAAAF84A1464C8CB3C2F3D" }, { "b" : "7F02AE81C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BAD67A84E56E73D031AE507261DA066B35949D34" }, { "b" : "7F02AE606000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F02AE3E9000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "B17C21299099640A6D863E423D99265824E7BB16" }, { "b" : "7F02AE01F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "1CA54A6E0D76188105B12E49FE6B8019BF08803A" }, { "b" : "7F02B124A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "C0ADBAD6F9A33944F2B3567C078EC472A1DAE98E" }, { "b" : "7F02ADE04000", "path" : "/usr/lib/x86_64-linux-gnu/libunwind.so.8", "elfType" : 3, "buildId" : "29F63E3B24F95E955C76E58354DB3294944105A1" }, { "b" : "7F02AD9A2000", "path" : "/usr/lib/x86_64-linux-gnu/libicui18n.so.55", "elfType" : 3, "buildId" : "F5BE69B92B7426F17A61E1A1308115A9FE73C51D" }, { "b" : "7F02AD60E000", "path" : "/usr/lib/x86_64-linux-gnu/libicuuc.so.55", "elfType" : 3, "buildId" : "463D8B610702D64AE0803C7DFCAA02CFB4C6477B" }, { "b" : "7F02AD39E000", "path" : "/lib/x86_64-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "390B2228E9A1071BB0BE285D77B6669CB37CE628" }, { "b" : "7F02AD17C000", "path" : "/lib/x86_64-linux-gnu/liblzma.so.5", "elfType" : 3, "buildId" : "15AED4855920E5A0FB8791B683EB88C7E1199260" }, { "b" : "7F02AB6C5000", "path" : "/usr/lib/x86_64-linux-gnu/libicudata.so.55", "elfType" : 3, "buildId" : "2CC92B3EC41116DD818F3DAF7DA7CBB410183DD6" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x12a7701]
mongod(+0xEA6559) [0x12a6559]
mongod(+0xEA6E81) [0x12a6e81]
libpthread.so.0(+0x11390) [0x7f02ae3fa390]
libc.so.6(gsignal+0x38) [0x7f02ae054428]
libc.so.6(abort+0x16A) [0x7f02ae05602a]
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0x114) [0x12208c4]
mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextENS_10StringDataE+0xE0) [0x8b6680]
mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextENS_10StringDataEPNS_20DatabaseCatalogEntryE+0x480) [0x8bdb30]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextENS_10StringDataEPb+0xAC0) [0x8c09b0]
mongod(ZN5mongo14repairDatabaseEPNS_16OperationContextEPNS_13StorageEngineERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbb+0xDC1) [0xd1f0b1]
mongod(+0x339325) [0x739325]
mongod(+0x33C4C0) [0x73c4c0]
mongod(main+0x732) [0x6f73a2]
libc.so.6(__libc_start_main+0xF0) [0x7f02ae03f830]
mongod(_start+0x29) [0x736e99]
----- END BACKTRACE -----
Aborted (core dumped)
Mind you, I am doing this all on a clone of the original so I wont ruin the original during this. If this fails, I need to know how to restore a copy of the DB to the same host. I have a bson/json dump from before this issue happened, however, I cannot run a restore because the database wont start up..not sure how to handle that. I have not run into any documentation or forum posts that list how to do that, which is confusing since I believe that its likely an issue others have had.