[CSHARP-3368] Issue with creating TTL Indexes Created: 28/Jan/21  Updated: 27/Oct/23  Resolved: 18/Feb/21

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 2.11.5, 2.11.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nikolay Papunov Assignee: James Kovacs
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Tested on Ubuntu with MongoDB 4.4.2, and Windows Server 2019 with MongoDB 4.4.1, 4.4.2 and 4.4.3


Attachments: Text File CreateIndexes.cs    

 Description   

Hello,

I am not able to create TTL Indexes from my C# code to Database 4.4.2 or 4.4.3. The code works file with version 4.4.1

C# Driver is version 2.11.6.

I can create no-TTL Indexes without any problem.

Here is the part of my code:

srUIndexes = Builders<SRUsage>.IndexKeys;
srUIndexesAdd = new List<CreateIndexModel<SRUsage>>()
    {
        new CreateIndexModel<SRUsage>(srUIndexes.Ascending(x => x.Date), new CreateIndexOptions { ExpireAfter = new TimeSpan(30, 0, 0, 0) }),
    };
await dbXen.srusages.Indexes.CreateManyAsync(srUIndexesAdd);

 
Here is the error from MongoDB server:

{"t":\{"$date":"2021-01-28T17:38:53.691+02:00"},"s":"W", "c":"NETWORK", "id":4615610, "ctx":"conn1","msg":"Failed to check socket connectivity","attr":\{"error":"The operation completed successfully."}}
{"t":\{"$date":"2021-01-28T17:38:53.691+02:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn1","msg":"Interrupted operation as its client disconnected","attr":\{"opId":19}}
{"t":\{"$date":"2021-01-28T17:38:53.691+02:00"},"s":"I", "c":"NETWORK", "id":22989, "ctx":"conn1","msg":"Error sending response to client. Ending connection from remote","attr":\{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"127.0.0.1:54579","connectionId":1}}
{"t":\{"$date":"2021-01-28T17:38:53.691+02:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn1","msg":"Connection ended","attr":\{"remote":"127.0.0.1:54579","connectionId":1,"connectionCount":1}}



 Comments   
Comment by James Kovacs [ 18/Feb/21 ]

Hi, Nikolay,

Thank you for letting us know that you found the root cause of the problem and providing us a detailed description. We have closed this ticket.

Sincerely,
James

Comment by Nikolay Papunov [ 29/Jan/21 ]

Hello Robert,

I think I found out the root cause. If there is a manually defined TTL Index in some collection and after that, you described it the same in the C# code. The code will create no indexes, and no errors will be returned. - just skipped all. If I delete all TTL indexes from my DB collections, all missing indexes (including TTL) will be created successfully. I've tested several times with success,
That's why it is working on 4.4.1 because I have never created TTL indexes manually there, but on PROD and test, I did.
PS: if I add the Index creation code in the Constructor of the DataContext (as it was before) and "not able" to create any Indexes, it is not possible to do any queries to the DB. That was the initial impact of this. I saw nothing new in the DB for more than 1 hour.

PS2: After one more test, this happened with the TTL seconds are different(even with 1-2 seconds). If the index is manually created and the seconds are exact the same, there is no issue.  In case of issue, there is no exception or skipping this particular index (collection), just skipped all defined after this "exception".

Comment by Nikolay Papunov [ 28/Jan/21 ]

I tested the code with my DB for 3 collections, and the indexes created successfully. But, if I run the code for all of the collections and doesn't works. (No one missing index is created)

If i comment the lines with TTL index definitions, everything works.

You can check my code attached.

 

 

 

 

 

Comment by Nikolay Papunov [ 28/Jan/21 ]

I have tested your code and it works. But in my real application, there is data in the DB. Some of the collections are with more than several millions documents. One of the collections is with 1.6b.

on my development computer, when i started on debug mode it works but here i am with version 4.4.1. on my test server it is 4.4.3 and still returning this error.

I'll try to reproduce the issue with our code and several collections and my real database. I'll let you know when I have news.

 

Yes, one TTL index per collection

Comment by Robert Stam [ 28/Jan/21 ]

I repeated the above test program with server versions 4.4.2 and 4.4.3 and no exceptions were thrown and the TTL index was created.

Comment by Robert Stam [ 28/Jan/21 ]

Is it possible that the collection(s) you are creating new TTL indexes on are very large?

I'm wondering whether it's just taking the server a very long time to create the new TTL index and in the meantime the client has timed out and closed the connection. That's one possible theory for why you saw the server log entries you saw.

Comment by Robert Stam [ 28/Jan/21 ]

> I forgot to mentioned that the issue appears when I try to create more than 1 TTL indexes. I have more than 10 Collections, where a such TTL index is needed.

But only one TTL index per collection right?

> With version 4.4.1, all indexes created properly.

It just so happens that I tested with 4.4.1. I will test with 4.4.2 and 4.4.3 also.

Comment by Robert Stam [ 28/Jan/21 ]

I am unable to reproduce this. I used the following short program to attempt to reproduce:

namespace ReproCSharp3368
{
    public class SRUsage
    {
        public int Id { get; set; }
        public DateTime Date { get; set; }
    }
 
    public static class Program
    {
        public static async Task Main(string[] args)
        {
            var client = new MongoClient("mongodb://localhost");
            var database = client.GetDatabase("test");
            database.DropCollection("test");
            var collection = database.GetCollection<SRUsage>("test");
 
            var srUIndexes = Builders<SRUsage>.IndexKeys;
            var srUIndexesAdd = new List<CreateIndexModel<SRUsage>>()
            {
                new CreateIndexModel<SRUsage>(srUIndexes.Ascending(x => x.Date), new CreateIndexOptions { ExpireAfter = new TimeSpan(30000) }),
            };
            await collection.Indexes.CreateManyAsync(srUIndexesAdd);
        }
    }
}

The program ran to completion and no exception was thrown.

I used the mongo shell to verify that the index was created:

MongoDB Enterprise > use test
switched to db test
MongoDB Enterprise > db.test.getIndexes()
[
        {
                "v" : 2,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_"
        },
        {
                "v" : 2,
                "key" : {
                        "Date" : 1
                },
                "name" : "Date_1",
                "expireAfterSeconds" : 2592000
        }
]
MongoDB Enterprise >

I also checked the server logs to see and no errors were logged there either:

{"t":{"$date":"2021-01-28T08:19:11.783-08:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"conn58205","msg":"createCollection","attr":{"namespace":"test.test","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"8019062e-266c-4e0d-b523-92f579154974"}},"options":{}}}
{"t":{"$date":"2021-01-28T08:19:11.800-08:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"conn58205","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"test.test","index":"_id_","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2021-01-28T08:19:11.800-08:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"conn58205","msg":"Index build: done building","attr":{"buildUUID":null,"namespace":"test.test","index":"Date_1","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}

 

 

Comment by Nikolay Papunov [ 28/Jan/21 ]

Hello,

 

I forgot to mentioned that the issue appears when I try to create more than 1 TTL indexes. I have more than 10 Collections, where a such TTL index is needed.

 

It seems that if I am trying to create just one, it probably works or sometimes works.

With version 4.4.1, all indexes created properly.

 

Br,

Nikolay

Generated at Wed Feb 07 21:45:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.