Uploaded image for project: 'Mongoid'
  1. Mongoid
  2. MONGOID-1864

Documents missing with large compound index

    • Type: Icon: Task Task
    • Resolution: Done
    • 2.4.8
    • Affects Version/s: None
    • Component/s: None
    • Labels:

      This problem might be hard to duplicate, but basically I have a unique compound index that may occasionally be too large. Unfortunately I'm not seeing any notification of this problem when the index is created, and only know it's a problem because a.) uniqueness is being violated, and b.) queries with the index are not returning all the records I expect.

      My class structure is basically like this:

      class Simulator
        has_many :profiles
      end
      
      class Profile
        field :parameter_hash
        field :proto_string
        belongs_to :simulator
        index ([[:simulator_id,  Mongo::DESCENDING], [:parameter_hash, Mongo::DESCENDING], [:proto_string, Mongo::DESCENDING]]), :unique => true
      end
      

      Now for the weirdness. I create a Profile with a very large proto_string. Let's call this record 'profile'. This record can be found with

      Profile.where(:simulator_id => profile.simulator_id).count 
      # => 1
      Profile.where(:parameter_hash =>  profile.parameter_hash).count
      # => 1
      Profile.where(:proto_string =>  profile.proto_string).count
      # => 1
      Profile.where(:parameter_hash =>  profile.parameter_hash, :proto_string =>  profile.proto_string).count
      # => 1
      

      but not with:

      Profile.where(:simulator_id => profile.simulator_id, :proto_string => profile.proto_string).count
      # => 0
      Profile.where(:simulator_id => profile.simulator_id, :parameter_hash => profile.parameter_hash).count
      # => 0
      Profile.where(:simulator_id => profile.simulator_id, :parameter_hash => profile.parameter_hash,  :proto_string => profile.proto_string).count
      # => 0
      

      My understanding is that this behavior arises because simulator_id is the first piece of my index, so basically all the queries that succeed are just not using the index. So this leads me to a couple of questions:

      During development/testing, how could I foresee/test for this problem? In production, how can adjust for this? I believe mongodb has a strategy for dealing with this by appending '.hint({$natural: 1})' to the end of a query where this is suspected. Is there anything similar I could do with mongoid?

            Assignee:
            Unassigned Unassigned
            Reporter:
            bcassell Ben Cassell
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: