When I search in the index that use custom collator and index key is variable length field, custom collator function can receive invalid WT_ITEM arg. I attached program that reproduce error. It looks like error happens because of next code in __curindex_search_near function:
Function __curindex_search has same problem.
Also, I see strange result of search_near() for same index but without custom collator. Attached program has next output:
Index def_collator has 2 keys insight: "123" and "12345". I call search_near() for the key = "1234" and get found_key = "12345" with exact = -1. Does the exact should be equal to 1? It seems that "1234" < "12345".