Potential for async state corruption under Fiber-based concurrency

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: 9.1.0
    • Component/s: None
    • None
    • None
    • Ruby Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      MONGOID-5915 addressed potential state corruption when running under Fiber-based concurrency. It turns out that, under further consideration, the issue remains despite the fix that was made for that issue.

      From Claude:

      The defect

      lib/mongoid/threaded.rb:602:

      if Fiber[STORAGE_OWNER_KEY] != Fiber.current.object_id
        Fiber[STORAGE_KEY] = (Fiber[STORAGE_KEY] || {}).dup   # shallow
        Fiber[STORAGE_OWNER_KEY] = Fiber.current.object_id
      end
      

      The hash is copied, its values are not. Every nested container — stack arrays, the
      current-scope hash, the sessions map, autosave lists, modified-document sets — stays the same object in the parent fiber and in all child and sibling fibers. The doc comment at
      :598-600 claims each fiber gets a snapshot of its parent's state. That is wrong for
      nested values.

      Reproductions

      Concurrent reads

      Two sibling request fibers under a real Async scheduler. Fiber A runs `Model.unscoped`
      and yields at MongoDB I/O; fiber B, a different request, queries during that
      window.

      boot: storage keys = [:"[mongoid]:persistence_context", "sessions", "create-stack",
                            "build-stack", "without_default_scope-stack", "assign-stack"]
      boot: stack object_id = 1024
      
      fiber A stack object_id: 1024  (parent: 1024)
      fiber B stack object_id: 1024
      fiber B sees stack:      [Doc]
      fiber B without_default_scope?(Doc): true
      fiber B selector:        {}
      fiber B documents:       ["alice-data", "mallory-data"]
      

      Doc's default scope is where(tenant: 'alice'). Fiber B's selector came back empty and it read another tenant's document.

      The sessions map leaks the same way — a sibling fiber with no session of its own picks up fiber A's live Mongo::Session out of the shared map.

      The precondition is trivial. Model.new alone, with no database access, materialises
      `build-stack` and `without_default_scope-stack`, which arms every request fiber spawned afterwards.

      MONGOID-5915 is not fixed

      The 5915 repro (10 concurrent fibers, Movie.only(:title, :slug).find_by then
      Movie.find_by(...).adult) passes under the shallow dup only because that script has no open with_scope when the fibers spawn. Instrumenting Threaded.set during the racy run shows the clobbered key is current-scope, a nested hash:

      keys written: {"current-scope" => 10, "[mongoid]:persistence_context" => 10, "sessions" => 10}
      

      Leave any unrelated scope open — Other.with_scope(Other.all), which never touches Movie — and the original corruption returns:

        no open scope inside an unrelated open with_scope
      no copy 8/10 errors (pre-5915 behaviour)
      shallow {} {Mongoid::Errors::AttributeNotLoaded => 9}
      deep {} {}

      This matters for triage: it is plain data corruption in a single-tenant async app, with no
      attacker and no multi-tenancy involved.

      Why fiber state is inherited at all

      Not to mimic threads. Thread state is not inherited at all — under :thread isolation,
      storage lives in Thread.current.thread_variable_get, and a child thread gets nil back and builds a fresh Hash.

      So the thread model has two opposite properties: total isolation across threads, total
      sharing across fibers within a thread.

      The inheritance in the fiber path exists because Mongoid uses fibers as a control-flow
      device in interceptable.rb:176 (_mongoid_run_child_callbacks_with_around), which spawns a fiber per embedded child to interleave around-callbacks. The comment at threaded.rb:66-72 says so explicitly. Those fibers are not concurrency primitives and must see their parent's state.

      Secondary finding: threads inherit under :fiber

      Under :fiber isolation, Thread.new also inherits — the new thread's root fiber inherits Fiber[] storage, trips the owner mismatch, shallow-dups, and shares nested containers with the spawning fiber.

      Background threads that started clean under :thread no longer do. Separate leak path, same fix.

      What the `.dup` was actually for

      MONGOID-5915, "Running Mongoid queries via async leads to state corruption" — a user report via PR #6009. Nothing to do with callbacks. From the ticket:

      putting `Mongoid::Threaded.reset!` just inside the `concurrency.times do` block "fixes" the issue... I'm not quite sure how to fix this; maybe if we can detect that we're starting a new query... we can call reset at that point.

      So the .dup and reset! are two answers to the same question, and the .dup was chosen specifically to avoid requiring users to call reset!. That requirement was already considered and rejected as the fix once.

      The tests added with the .dup only exercise top-level scalars (Threaded.set('x', ...)), which is why the nested containers were missed.

      On Threaded.reset! as the mitigation

      It works:

      without reset!    selector={}                    docs=["alice-data","mallory-data"]  LEAK
      with reset!       selector={"tenant"=>"alice"}   docs=["alice-data"]                 clean
      

      But:

      • Nothing calls it. grep -rn "reset!" lib/ returns exactly one hit, the definition. No railtie hook, no ActiveSupport::Executor integration, no middleware. Every other reference is in specs.
      • The insecure path does not require opting in. isolation_level defaults to :rails (`config.rb:129`), which delegates ActiveSupport::IsolatedExecutionState.isolation_level (config.rb:172-186). A Rails app on Falcon sets config.active_support.isolation_level = :fiber, and Mongoid silently follows. That developer never touched a Mongoid isolation setting.
      • It does not address 5915. The corrupting fibers there are spawned within a request, so a reset at the request boundary does not help.

      reset! is necessary but not sufficient.

      The shallow dup is not a coherent contract

      Whether a callback fiber can write through to its parent depends on whether the key happened to be materialised earlier:

      callback fiber writes...        parent sees
        nested, pre-existing key  ->  ["parent", "from-callback-fiber"]   shared
        nested, NEW key           ->  []                                  isolated
        top-level key             ->  nil                                 isolated
      

      Copy policy

      Callbacks need inheritance, not sharing. A copy preserves inheritance fully — the
      child still sees the parent's persistence context, current scope, and session — and breaks only write-back. Measured with a copying storage, embedded around-callbacks are unchanged:

      shallow dup (current):          copy on inherit:
        parent before                   parent before
        child a before                  child a before
        child b before                  child b before
        child b after                   child b after
        child a after                   child a after
        parent after                    parent after
      

      Depth is the wrong way to think about it

      A uniform one-level copy ({{storage.each_with_object({})

      { |(k,v),a| a[k] = v.dup }

      }}) is not
      sufficient. It depends on where each operation mutates:

      key shape mutated at depth-1 enough?
      <name>-stack Array stack(name).push/pop — the array itself yes
      current-scope Hash{Class => Criteria} current_scope[klass] = scope yes
      [mongoid]:persistence_context Hash{object_id => Context} store[id] = ctx yes
      sessions Hash{client => Session} sessions[client] = s, delete yes
      db-override, client-override scalar replaced wholesale yes
      autosaves Hash{Class => Array} autosaves[klass].push(id)inner array no
      validations Hash{Class => Array} same no
      modified-documents Hash{Session => Set} modified_documents[session] << docinner set no

      Previous tests turn on keys with depth-1, which is why the experiments above came out clean. autosaves and validations are a genuine bug in their own right: they are recursion guards holding _id}}s, so fiber A's in-flight autosave of {{Person#123 can make fiber B skip its own autosave of the same id. That is a missed write, not just a leak.

      The rule to encode

      Copy every container Mongoid mutates in place; stop at objects that carry identity.

      Recurse through Hash, Array, Set. Never copy Mongo::Session, Mongoid::Document, Class, Criteria, PersistenceContext — that identity is what lets a callback fiber join the parent's transaction and route its own queries correctly.

            Assignee:
            Unassigned
            Reporter:
            Jamis Buck
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: