[JAVA-2637] access count of compound index in $indexStats java driver Created: 16/Oct/17  Updated: 27/Oct/23  Resolved: 16/Nov/17

Status: Closed
Project: Java Driver
Component/s: Query Operations
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Minor - P4
Reporter: Paul Lin Assignee: Unassigned
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS



 Description   

I met a strange phenomenon, hope someone can provide a little clue to answer.
I added a compound index to mycollection. After using db.mycollection.find().Explain() in the shell, I can see that the index appears in WiningPlan, that is, the index is used. Checking by db.mycollection.aggregate ([{$ indexStats: {}}]) after find(), I can see that the index access counter has increased. Using python with pymongo also has the same result. Only java with MongoDB Java Driver (http://mongodb.github.io/mongo-java-driver/) can not make the access counter increase. It can not be determined that java uses that index. What can I do to ensure java use the index?



 Comments   
Comment by Jeffrey Yemin [ 16/Nov/17 ]

paullin as I haven't heard back from you in answer to my most recent question, I'm closing this issue. I'll be happy to re-open it if you can provide more information.

Comment by Jeffrey Yemin [ 08/Nov/17 ]

Looking at your command line of java -cp . CompoundIndexTest, I wouldn't expect it to work as you don't include the mongo-java-driver JAR file.

As for the compilation, you do not need to include both mongo-java-driver and bson JAR files, as the mongo-java-driver JAR file is an Uber jar which contains all the class that are in bson.jar.

With regard to index use, please provide a complete working test program for us to try to reproduce the issue, as we have been unable to do so given the information you've provided so far.

Comment by Paul Lin [ 18/Oct/17 ]

It is my mistake to provide incomplete information. I did run test twice. First test runs with .modifiers(new Document("$explain", true)) and gets result of "index in winningPlan". Second one runs with .iterator() instead and gets "counter didn't changed" by checking $indexStats in shell manually.

BTW, I put downloaded bson-3.0.4.jar, mongo-java-driver-3.4.3.jar and the code you provided, saved as CompoundIndexTest.java, in same folder then compiled it in command line: javac -cp ./mongo-java-driver-3.4.3.jar:./bson-3.0.4.jar CompoundIndexTest.java
I got Error: Cannot find symbol
System.out.println(cur.toJson(JsonWriterSettings.builder().indent(true).build()));
It was solved and compiled OK with little modification:
JsonWriterSettings settings = new JsonWriterSettings( true );
System.out.println(cur.toJson(settings));

Second error occurred as I executed it in command line:
java -cp . CompoundIndexTest
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/bson/conversions/Bson
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
I can not solve this one so far.

Comment by Jeffrey Yemin [ 17/Oct/17 ]

I wouldn't expect this test to hit the index because it's running an explain rather than the actual query. This slightly modified program demonstrates that the index is accessed:

import com.mongodb.MongoClient;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import org.bson.Document;
import org.bson.conversions.Bson;
import org.bson.json.JsonWriterSettings;
 
import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
 
import static com.mongodb.client.model.Aggregates.match;
import static com.mongodb.client.model.Filters.*;
import static java.util.Arrays.asList;
 
public class CompoundIndexTest {
    public static void main(String[] args) throws ParseException {
        DateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        Date start = df.parse("2017-09-29 16:00:00");
        Date end = df.parse("2017-10-01 15:00:00");
 
        Bson statdFilter = new Document("$gte", start).append("$lte", end);
 
        MongoClient client = new MongoClient();
        MongoDatabase database = client.getDatabase("myDB");
        MongoCollection<Document> collection = database.getCollection("mycollection");
 
        // uncomment these lines the first time this is run
//        collection.drop();
//        collection.insertOne(new Document("field1", "12345").append("field2", "203").append("field3", start));
//        collection.createIndex(new Document("field1", 1).append("field2", 1).append("field3", 1));
 
        // Run the query (without $explain)
        for (Document cur : collection.find(and(eq("field1", "12345"),
                in("field2", "203", "201", "202", "204"), new Document("field3", statdFilter)))) {
            System.out.println(cur.toJson(JsonWriterSettings.builder().indent(true).build()));
        }
 
        System.out.println();
 
        // Get index stats.  The accesses.ops field increments by one after each run
        for (Document cur : collection.aggregate(asList(new Document("$indexStats", new Document()),
                match(eq("name", "field1_1_field2_1_field3_1"))))) {
             System.out.println(cur.toJson(JsonWriterSettings.builder().indent(true).build()));
        }
    }
}

Comment by Paul Lin [ 17/Oct/17 ]

I did use iterator() in my test code. here is my test code:

package com.mycompany.mongo.querytest;
 
import static com.mongodb.client.model.Filters.and;
import static com.mongodb.client.model.Filters.eq;
import static com.mongodb.client.model.Filters.in;
 
import java.text.DateFormat;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
 
import org.bson.Document;
import org.bson.conversions.Bson;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
 
import com.mongodb.MongoClient;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoCursor;
import com.mongodb.client.MongoDatabase;
 
@RunWith(SpringRunner.class)
@SpringBootTest
public class QueryUsingMongoClientTest {
	private final Logger logger = LoggerFactory.getLogger(QueryTestApplicationTests.class);
 
	@Autowired
	private MongoClient client;
 
	@Test
	public void testQuerymycollection() throws ParseException {
		DateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
		Date start = df.parse("2017-09-29 16:00:00");
		Date end = df.parse("2017-10-01 15:00:00");
 
		Bson statdFilter = new Document("$gte", start).append("$lte", end);
 
		MongoDatabase database = client.getDatabase("myDB");
 
		MongoCollection<Document> collection = database.getCollection("mycollection");
 
		logger.info("Before");
 
		MongoCursor<Document> iterator = collection.find(and(eq("field1", "12345"),
				in("field2",  "203" , "201" , "202" , "204"), new Document("field3", statdFilter))).modifiers(new Document("$explain", true)).iterator();
		while (iterator.hasNext()) {
			logger.info(iterator.next().toJson());
		}
	}
}

Hope this could help to reveal something.

Comment by Jeffrey Yemin [ 16/Oct/17 ]

I have an idea what might be happening. The Java driver has a fluent API for the find() method, and as a result the query is only executed when the application actually starts iterating the results. So:

myCollection.find(Filters.eq("_id", id)   // returns a FindIterable()

will not execute the query on the server, while

myCollection.find().iterator()   // returns a MongoCursor

will. Please confirm whether this might be the root cause. Otherwise, please provide some sample code so we can investigate further.

Generated at Thu Feb 08 08:57:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.