Uploaded image for project: 'Realm JavaScript SDK'
  1. Realm JavaScript SDK
  2. RJS-2205

Long running (4 minutes) writes for large datasets from a Map / Array loop

      How frequently does the bug occur?

      Always

      Description

      We have some VERY long running "write" queries, but to be fair, against large datasets. Although I don't really know what "large" means in Realm terms... I am talking when we have ~1,000,000 items to write/create (see code below). It can take upwards of 4 minutes (depending on the device). And because it is wrapped in a single write statement, our main thread in our Node/Electron app is blocked during this time (which is the real problem). I would be okay with it taking 4 minutes but cannot have our main thread blocked.

      Here are my questions:

      1) is this expected that it could be taking 4 minutes to make this large Write query? or is there something wrong with the way we structure our code below?
      2) we wrap it in a single Write statement because that is what is recommended from the docs, but does that change when you have such a large dataset being written like this? Should we instead be doing some sort of batching of the writes?
      3) is there any way to run Realm write queries asynchronously in a way that would not be blocking the main thread? I know in the past the Realm team has said there is not a way to run Realm in a Node Worker Thread, is this still true? (see here: https://github.com/realm/realm-js/issues/4434#issuecomment-1075403729 and here: https://github.com/realm/realm-js/issues/4440)

      The TLDR here is whether this is expected to take a long time for a creation of 1 million items and whether there is a better way to structure our write (should we batch it breaking it up into 10 x 100,000 writes for example?) and/or if there is a way to not block the main thread in Node by having Realm run asynchronously and/or off thread in like a Worker Thread?

      Thanks so much in advance!

      Stacktrace & log output

      Unable to find source-code formatter for language: shell. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      N/A
      

      Can you reproduce the bug?

      Always

      Reproduction Steps

      Here is the example code. For the context of this issue we are having, between the "parent" RootBlocks and their children, they total around 1.1 million RootBlocks and Blocks being created.

      try {
        client.write(() => {
          blocksMap.forEach((blockObj, blockId) => {
            //Create Root block
            let _id = new BSON.ObjectId();
            let rootTitle = blockObj.block?.title;
      
            const parentRootBlock = client.create('RootBlock', {
              _id: _id,
              string: rootTitle,
              children: [],
            });
      
            //Create direct children of root block
            blockObj.children.forEach(childBlock => {
              let _id = new BSON.ObjectId();
              const blockContent = blocksMap.get(childBlock)?.block?.string;
              const eachChild = client.create('Block', {
                _id: _id,
                string: blockContent,
                children: [],
                root: parentRootBlock,
              });
      
              //Add the child to the root block children
              parentRootBlock.children.push(eachChild);
            });
          });
        });
      } catch (error) {
        console.log(`RealmDB writeToRealm error: ${error}`);
      } finally { }
      

      Version

      10.22

      What services are you using?

      Local Database only

      Are you using encryption?

      No

      Platform OS and version(s)

      Windows 11

      Build environment

      N/A

      Cocoapods version

      No response

            Assignee:
            Unassigned Unassigned
            Reporter:
            unitosyncbot Unito Sync Bot
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: