Uploaded image for project: 'Perl Driver'
  1. Perl Driver
  2. PERL-127

be able to specify 32- or 64-bit ints

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Perl driver
    • Labels:
      None
    • # Replies:
      8
    • Last comment by Customer:
      true
    • Epic Link:

      Activity

      Hide
      defc0n Mitch McCracken added a comment -

      This would be useful to have

      Show
      defc0n Mitch McCracken added a comment - This would be useful to have
      Hide
      defc0n Mitch McCracken added a comment -

      Would it be possible to tell the mongo client to use the most efficient data size depending upon the integer? For instance, if its an integer which can be represented within 2^32, use the regular 32 bit integer datatype. If its an integer that requires more than 32 bits, using the IntegerLong type. I believe the Python mongo client has the ability to do this. It'd be nice to not have to worry about it as the programmer and have the client automatically pick the best fit if I'm not concerned about making them uniform.

      Show
      defc0n Mitch McCracken added a comment - Would it be possible to tell the mongo client to use the most efficient data size depending upon the integer? For instance, if its an integer which can be represented within 2^32, use the regular 32 bit integer datatype. If its an integer that requires more than 32 bits, using the IntegerLong type. I believe the Python mongo client has the ability to do this. It'd be nice to not have to worry about it as the programmer and have the client automatically pick the best fit if I'm not concerned about making them uniform.
      Hide
      behackett Bernie Hackett added a comment - - edited

      With PyMongo you can also specify a "small long". That forces the driver to encode to a BSON int64 (8 bytes) instead of a BSON int32 (4 bytes). This became a bit more complicated with python 3.x since the python "long" type was removed. We added an int64 type that is a trivial wrapper for python int that hints the encoder to store the integer in 8 bytes instead of 4.

      Show
      behackett Bernie Hackett added a comment - - edited With PyMongo you can also specify a "small long". That forces the driver to encode to a BSON int64 (8 bytes) instead of a BSON int32 (4 bytes). This became a bit more complicated with python 3.x since the python "long" type was removed. We added an int64 type that is a trivial wrapper for python int that hints the encoder to store the integer in 8 bytes instead of 4.
      Hide
      defc0n Mitch McCracken added a comment -

      this is where the pymongo driver makes its decision on how to serialize an integer to bson:

      https://github.com/mongodb/mongo-python-driver/blob/master/bson/__init__.py#L423

      this same behavior would be nice to have in the Perl driver as well

      Show
      defc0n Mitch McCracken added a comment - this is where the pymongo driver makes its decision on how to serialize an integer to bson: https://github.com/mongodb/mongo-python-driver/blob/master/bson/__init__.py#L423 this same behavior would be nice to have in the Perl driver as well
      Hide
      david.golden David Golden added a comment -

      I agree that it should DWIM if a specific form isn't specified. I've put it in the roadmap for the v1.0.0 driver. This work in particular will probably happen in the next couple months as I tackle a number of BSON related issues together.

      Show
      david.golden David Golden added a comment - I agree that it should DWIM if a specific form isn't specified. I've put it in the roadmap for the v1.0.0 driver. This work in particular will probably happen in the next couple months as I tackle a number of BSON related issues together.
      Hide
      defc0n Mitch McCracken added a comment -

      Ran into funny issue whilst trying to upgrade from 2.6 to 2.8. Because I created indexes using the Perl driver, I had index entries such as:

      {
      "v" : 1,
      "key" :

      { "start" : NumberLong(1) }

      ,
      "name" : "start_1",
      "ns" : "interface.data"
      },

      Specifying 1 in Perl ends up being NumberLong(1). I did a mongodump of the database and tried to mongorestore it in 2.8, and I got an error like:

      restore error: error creating indexes: createIndex error: exception: bad index key pattern { start:

      { $numberLong: "1" }

      }: Index keys cannot be Objects or Arrays.

      I was able to manually edit the metadata.json files that the mongodump created and modify them to be 32-bit 1 instead and it was able to import. mongo 2.8 still is letting me create indexes via the Perl 0.705.0.0 client ensure_index() and they are still getting stored as NumberLong(1). However, get_indexes() is returning empty arrayref even though there are indexes that exist, although reading changelogs this might be a separate issue that you already fixed.

      Show
      defc0n Mitch McCracken added a comment - Ran into funny issue whilst trying to upgrade from 2.6 to 2.8. Because I created indexes using the Perl driver, I had index entries such as: { "v" : 1, "key" : { "start" : NumberLong(1) } , "name" : "start_1", "ns" : "interface.data" }, Specifying 1 in Perl ends up being NumberLong(1). I did a mongodump of the database and tried to mongorestore it in 2.8, and I got an error like: restore error: error creating indexes: createIndex error: exception: bad index key pattern { start: { $numberLong: "1" } }: Index keys cannot be Objects or Arrays. I was able to manually edit the metadata.json files that the mongodump created and modify them to be 32-bit 1 instead and it was able to import. mongo 2.8 still is letting me create indexes via the Perl 0.705.0.0 client ensure_index() and they are still getting stored as NumberLong(1). However, get_indexes() is returning empty arrayref even though there are indexes that exist, although reading changelogs this might be a separate issue that you already fixed.
      Hide
      xgen-internal-githook Githook User added a comment -

      Author:

      {u'username': u'dagolden', u'name': u'David Golden', u'email': u'dagolden@cpan.org'}

      Message: PERL-127 encode small ints to Int32; encode BigInts to Int64

      For ordinary Perl integers, they are encoded as BSON Int32 if they fit
      and BSON Int64 otherwise.

      For Math::BigInt objects, conversion now extracts the BigInt as a string
      and process with strtoll. The previous implementation violated
      encapsulation and made assumptions about the inner structure of
      Math::BigInt objects. The result will then alwasy be encoded as BSON
      Int64.
      Branch: master
      https://github.com/mongodb/mongo-perl-driver/commit/9a3cfd57f059e6931ffeedb6cffd6190714c54f7

      Show
      xgen-internal-githook Githook User added a comment - Author: {u'username': u'dagolden', u'name': u'David Golden', u'email': u'dagolden@cpan.org'} Message: PERL-127 encode small ints to Int32; encode BigInts to Int64 For ordinary Perl integers, they are encoded as BSON Int32 if they fit and BSON Int64 otherwise. For Math::BigInt objects, conversion now extracts the BigInt as a string and process with strtoll. The previous implementation violated encapsulation and made assumptions about the inner structure of Math::BigInt objects. The result will then alwasy be encoded as BSON Int64. Branch: master https://github.com/mongodb/mongo-perl-driver/commit/9a3cfd57f059e6931ffeedb6cffd6190714c54f7
      Hide
      david.golden David Golden added a comment -

      The recent commit follows the behavior of pymongo, which stores integers in the smallest possible space. Users can still force 64-bit storage by wrapping integers with Math::BigInt.

      Show
      david.golden David Golden added a comment - The recent commit follows the behavior of pymongo, which stores integers in the smallest possible space. Users can still force 64-bit storage by wrapping integers with Math::BigInt.

        People

        • Votes:
          3 Vote for this issue
          Watchers:
          6 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:
            Days since reply:
            9 weeks, 1 day ago
            Date of 1st Reply: