Uploaded image for project: 'Perl Driver'
  1. Perl Driver
  2. PERL-265

MongoDB::Cursor silently skips over thousands of objects after failing with "recv timed out"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: 0.700.0
    • Fix Version/s: None
    • Component/s: Perl driver
    • Environment:
      Ubuntu 8.04.4 LTS, Perl 5.14.2

      Description

      Hello,

      We store a lot (millions) of files in a GridFS database. In order to iterate (and resume iterating) over a list of those files sorted rougly by an insertion date, we do this:

      # Create a cursor to iterate over files
      # ('query_timeout' is set to 19000 ms, we can't affort to wait 30 seconds)
      my $find_query = { _id => { '$gt' => MongoDB::OID->new(value => '...') } };
      my $cursor = $mongodb_fs_files_collection->query($find_query)->sort( {_id => 1} )->fields( {_id => 1, filename => 1} );
      $cursor->immortal(1);

      We were having a lot of problems with recv timed out previously (Perl driver seems to fail when MongoDB decides to create a new 2 GB data file for a minute or so), so we wrapped attempts to fetch a next object into eval{}; that also retries to read a next filename:

      my $attempt_to_read_succeeded = 0;
       
      for ( my $retry = 0 ; $retry < 3 ; ++$retry )
      {
          if ( $retry > 0 )
          {
              WARN("Retrying ($retry)...");
          }
       
          eval {
       
              # Read next
              $object = $cursor->next;
              $attempt_to_read_succeeded = 1;
          };
       
          if ( $@ )
          {
              WARN("Attempt to read next the filename didn't succeed because: $@");
          }
          else
          {
              last;
          }
      }
       
      unless ( $attempt_to_read_succeeded )
      {
          LOGDIE("Unable to read the next filename from GridFS.");
      }

      However, we found out that when an attempt to read a next object from the cursor ($object = $cursor->next;) fails with recv timed out, the cursor skips a chunk of objects in the cursor's sequence (~94,000 in our case) and then resumes returning objects just fine.

      Wat do? Is it safe to continue using the cursor (without recreating it) after it die() s? How do we prevent those recv timed out failures from happening?

        Attachments

          Activity

            People

            Assignee:
            mike.friedman@10gen.com Mike Friedman
            Reporter:
            pypt Linas Valiukas
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: