I've pushed out a custom filesystem on github (https://github.com/MicronSSD/unfs) that works with WiredTiger develop branch and MongoDB v3.4.
I would like to propose a change to remove unecessary patching to wiredtiger_open to make custom filesystem work.
Currently, the solution of patching the config string passing to wiredtiger_open() requires a hack to change wiredtiger_open() to make existing applications (such as MongoDB) work without changing the application source code.
A propose change is that wiredtiger_open() would load a filesystem plugin shared library by a given name (e.g. libcustomfs.so) and then invoke the predefined init routine from that library (e.g. wiredtiger_custom_init(const char* home)). It will be up to the custom filesystem initialization routine to handle its own parameters. WiredTiger should properly setup the necessary config string. The main thing that wiredtiger_open() needs is the name of the filesystem plugin library which can be acquired either by parameter (i.e. mongod parameter) or environment variable or some startup configuration file. If that plugin library name is not specified, then wiredtiger_open can fallback to using native filesystem.
Also, in the wiredtiger_custom_init routine, please allow the registration of all the WT_FILE_HANDLE functions once instead of each time a file is opened (i.e. via fs_open_file). This is for improving the custom filesystem performance by eliminating a bunch of registration instructions each time a file is opened. The custom functions are static and don't need to be reassigned every time. The idea of custom filesystem is for performance reason. Although you may not see its effect yet, but will be in the near future when SSD with 3D Crosspoint Technology is available.