An output type map is defined in wiredtiger.i to map non-zero return values from C-language methods to Java exceptions. This is applied uniformly to all methods that return integers.

The original authors of the compare_wrap and equals_wrap methods, however, clearly intended the result of the comparison to be returned directly. There are a couple of hints of this. There are Swig declarations to ignore/rename the methods and inside the renamed methods the logic from the output type map is duplicated.

The current state is that when a cursor comparison returns 1, a WiredTigerException is thrown with the POSIX error string for ERRNO 1. -1 throws the Wired Tiger exception string at index -1. 0 happens to work, which might be why this was missed until now. As far as I can tell, this bug has been present since the JNI was introduced.

There are a number of ways of solving this. Some of them are more brittle / less
performant than others. The route I've chosen to take is to avoid the typemap
by changing the return type to a long. It was the most minimally intrusive way
which had minimal performance impact.

I'll have a pull request on GitHub ready in a few moments. Here is a commit illustrating the bug in JUnit tests.

https://github.com/mwberry/wiredtiger/commit/4d143cb4884c50e7137271b76b2ba87b95810ee1

Other options would include:
1: Stashing the default Swig typemap for int and restoring it
at the end of the scope of the Cursor class. Any methods after
the restoration would return integers verbatim. This method is
brittle due to a reliance on the ordering of methods and Swig
declarations.
2: Boxing the int return type into a capital-I Integer in Java
3: Boxing the int into a single element int[] and then unboxing in Java
4: Some other deep Swig magick to arcane for myself.

Source Pointers:
wiredtiger.i:165 - typemap out int
wiredtiger.i:313-314 - ignore/rename compare
wiredtiger.i:315-316 - ignore/rename equals
wiredtiger.i:1189 - compare_wrap body
wiredtiger.i:1198 - equals_wrap body
wiredtiger.i:1694 - Cursor::compare body
wiredtiger.i:1704 - Curosr::equals body