[GODRIVER-2046] Return clearer error when truncating a float64 to float32 without the truncate tag Created: 14/Jun/21  Updated: 02/Nov/23  Resolved: 01/Nov/23

Status: Closed
Project: Go Driver
Component/s: BSON
Affects Version/s: 1.4.7, 1.5.1, 1.5.2, 1.5.3
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Matt Hartstonge Assignee: Unassigned
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows
Alpine Linux/Docker Containers
Go 1.16.5


Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

Motivation

If attempting to unmarshal a float64 into a float32 results in truncation, and the destination does not have the truncate tag, the error errCannotTruncate is returned. This error consists of the message "float64 can only be truncated to an integer type when truncation is enabled". The error message is misleading. A float64 can be unmarshaled into a float32 if the truncate tag is present.

Scope
Update the error message to apply to this case. Consider changing to the following: "float64 can only be truncated when the truncate tag is present"

Original ticket description

We've run across an issue where the truncation logic in the BSON lib is causing an issue due to a floating point rounding error when trying to validate a float32 value when comparing float64(float32(f)) to the mongo returned float64(f).

Here's the minimum reproducible showcasing that 0.1 returns incorrectly: https://play.golang.org/p/eYANaXFKVk9

package main
 
import (
	"fmt"
)
 
func main() {
	a := float64(0.1) // What type is coming back from mongo
	b := float32(a) // What type is being conformed based on specified type in struct
	c := float64(b) // The conversion back, which causes the rounding point error going from f32 -> f64
	
	fmt.Printf("%#+v | %#+v | %#+v\n", a, b, c)
	fmt.Printf("not equal? %t\n", float64(float32(a)) != a)
}

Returns:
0.1 | 0.1 | 0.10000000149011612
not equal? true

This is occurring here: https://github.com/mongodb/mongo-go-driver/blob/49af6e279ffe534210dca95e375d372517145e44/bson/bsoncodec/default_value_decoders.go#L480



 Comments   
Comment by Githook User [ 02/Nov/23 ]

Author:

{'name': 'Steven Silvester', 'email': 'steven.silvester@ieee.org', 'username': 'blink1073'}

Message: GODRIVER-2046 [master] Return clearer error when truncating a float64 to float32 without the truncate tag (#1449)

Co-authored-by: Ernie Hershey <ernie.hershey@mongodb.com>
Branch: master
https://github.com/mongodb/mongo-go-driver/commit/6ed544fb866eb0ea2aa154e7fdd38b45a0f63cf1

Comment by Matt Hartstonge [ 22/Jun/21 ]

Yeah, that would be helpful. 👍

I bet it's down to a difference in how `mgo` was pushing in `float32`s which has thrown off all our data :/

Thanks for your help Kevin, has been great 😀

Comment by Kevin Albertson [ 22/Jun/21 ]

matt@linc-ed.com that seems like a good solution!

The default decoding behavior does give users an extra safety check to prevent them from inadvertently truncating values, so I do not think we will change the default behavior.

Is your solution satisfactory? If so, I will proceed to update the ticket description to return a clearer error when truncating a float64 to float32 without the truncate tag.

Comment by Matt Hartstonge [ 16/Jun/21 ]

Hey Kevin,

Yeah, setting the BSON tag works. 👍 but ...

We generate all our microservices directly from protoc-gen-go which doesn't currently have a mechanism (natively) to enable tag addition/mutation. We encode our database models directly into the proto files due to the generation and API interface guarantees it provides for API consumers.
We could monkey patch in the truncation tags, but that relies on no-one accidentally taking it out on the next code gen cycle 😬.

Our workaround has been to copy the default decoder, register it and return the {{float32}}s, our mongo connection logic is all wrapped in a common library so it's been extremely quick to patch it out this way across all our services:

	rb := bson.NewRegistryBuilder()
	rb.RegisterTypeDecoder(reflect.TypeOf(float32(0)), truncatingFloatDecoder{})
	clientOptions.SetRegistry(rb.Build())

// truncatingFloatDecoder is a clone of the default value decoder for float32
// from mongo-go-driver, with the truncation check removed to work around
// https://jira.mongodb.org/browse/GODRIVER-2046
type truncatingFloatDecoder struct{}
 
func (tfd truncatingFloatDecoder) floatDecodeType(ec bsoncodec.DecodeContext, vr bsonrw.ValueReader, t reflect.Type) (reflect.Value, error) {
	var f float64
	var err error
	switch vrType := vr.Type(); vrType {
	case bsontype.Int32:
		i32, err := vr.ReadInt32()
		if err != nil {
			return emptyValue, err
		}
		f = float64(i32)
	case bsontype.Int64:
		i64, err := vr.ReadInt64()
		if err != nil {
			return emptyValue, err
		}
		f = float64(i64)
	case bsontype.Double:
		f, err = vr.ReadDouble()
		if err != nil {
			return emptyValue, err
		}
	case bsontype.Boolean:
		b, err := vr.ReadBoolean()
		if err != nil {
			return emptyValue, err
		}
		if b {
			f = 1
		}
	case bsontype.Null:
		if err = vr.ReadNull(); err != nil {
			return emptyValue, err
		}
	case bsontype.Undefined:
		if err = vr.ReadUndefined(); err != nil {
			return emptyValue, err
		}
	default:
		return emptyValue, fmt.Errorf("cannot decode %v into a float32 or float64 type", vrType)
	}
 
	switch t.Kind() {
	case reflect.Float32:
		return reflect.ValueOf(float32(f)), nil
	case reflect.Float64:
		return reflect.ValueOf(f), nil
	default:
		return emptyValue, bsoncodec.ValueDecoderError{
			Name:     "FloatDecodeValue",
			Kinds:    []reflect.Kind{reflect.Float32, reflect.Float64},
			Received: reflect.Zero(t),
		}
	}
}
 
func (tfd truncatingFloatDecoder) DecodeValue(dctx bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
	if !val.CanSet() {
		return bsoncodec.ValueDecoderError{
			Name:     "FloatDecodeValue",
			Kinds:    []reflect.Kind{reflect.Float32, reflect.Float64},
			Received: val,
		}
	}
 
	elem, err := tfd.floatDecodeType(dctx, vr, val.Type())
	if err != nil {
		return err
	}
 
	val.SetFloat(elem.Float())
	return nil
}

Comment by Kevin Albertson [ 16/Jun/21 ]

Great thank you for the additional tests matt@linc-ed.com!

There is a struct tag that is documented as having the intent to allow truncation of doubles into a float32 with loss of precision:
https://pkg.go.dev/go.mongodb.org/mongo-driver@v1.5.3/bson/bsoncodec#StructTags

Modifying the example I included, that can be updated like so:

type myStruct struct {
	F float32 `bson:"f,truncate"`
}

The error message does seem misleading. If the truncate tag resolves your issue, I will repurpose this ticket to consider updating the error returned when truncating a float64 to float32 without the truncate tag.

Comment by Matt Hartstonge [ 15/Jun/21 ]

Here's an extended table test with some numbers that generally throw floating point conversion errors:

package main
 
import (
	"testing"
 
	"go.mongodb.org/mongo-driver/bson"
)
 
type myStruct struct {
	F float32
}
 
func Test2046(t *testing.T) {
	testcases := []struct {
		description string
		value       float64
	}{
		{
			description: "should unmarshal float64(0) into float32(0)",
			value:       0,
		},
 
		{
			description: "should unmarshal positive float64(0.1) into float32(0.1)",
			value:       0.1,
		},
		{
			description: "should unmarshal positive float64(0.2) into float32(0.2)",
			value:       0.2,
		},
		{
			description: "should unmarshal positive float64(0.25) into float32(0.25)",
			value:       0.25,
		},
		{
			description: "should unmarshal positive float64(0.33) into float32(0.33)",
			value:       0.33,
		},
		{
			description: "should unmarshal positive float64(0.5) into float32(0.5)",
			value:       0.5,
		},
 
		{
			description: "should unmarshal negative float64(-0.1) into float32(-0.1)",
			value:       0.1,
		},
		{
			description: "should unmarshal negative float64(-0.2) into float32(-0.2)",
			value:       0.2,
		},
		{
			description: "should unmarshal negative float64(-0.25) into float32(-0.25)",
			value:       0.25,
		},
		{
			description: "should unmarshal negative float64(-0.33) into float32(-0.33)",
			value:       0.33,
		},
		{
			description: "should unmarshal negative float64(-0.5) into float32(-0.5)",
			value:       0.5,
		},
 
		{
			description: "should unmarshal positive whole numbers float64(1.0) into float32(1.0)",
			value:       1.0,
		},
		{
			description: "should unmarshal positive numbers float64(1.1) into float32(1.1)",
			value:       1.1,
		},
		{
			description: "should unmarshal positive numbers float64(1.2) into float32(1.2)",
			value:       1.2,
		},
		{
			description: "should unmarshal positive numbers float64(1.25) into float32(1.25)",
			value:       1.25,
		},
		{
			description: "should unmarshal positive numbers float64(1.33) into float32(1.33)",
			value:       1.33,
		},
		{
			description: "should unmarshal positive numbers float64(1.5) into float32(1.5)",
			value:       1.5,
		},
 
		{
			description: "should unmarshal negative whole numbers float64(-1.0) into float32(-1.0)",
			value:       -1.0,
		},
		{
			description: "should unmarshal negative numbers float64(-1.1) into float32(-1.1)",
			value:       -1.1,
		},
		{
			description: "should unmarshal negative numbers float64(-1.2) into float32(-1.2)",
			value:       -1.2,
		},
		{
			description: "should unmarshal negative numbers float64(-1.25) into float32(-1.25)",
			value:       -1.25,
		},
		{
			description: "should unmarshal negative numbers float64(-1.33) into float32(-1.33)",
			value:       -1.33,
		},
		{
			description: "should unmarshal negative numbers float64(-1.5) into float32(-1.5)",
			value:       -1.5,
		},
	}
 
	for _, testcase := range testcases {
		t.Run(testcase.description, func(t *testing.T) {
			doc, err := bson.Marshal(bson.D{{"f", testcase.value}})
			if err != nil {
				t.Errorf("Marshal error: %v", err)
			}
 
			var dst myStruct
			err = bson.Unmarshal(doc, &dst)
			if err != nil {
				t.Errorf("Unmarshal error: %v", err)
			}
		})
	}
}
 
// The test above fails with the output
// Unmarshal error: error decoding key f: float64 can only be truncated to an integer type when truncation is enabled

Comment by Matt Hartstonge [ 15/Jun/21 ]

Hi @kevin, 

That seems to be the bson repro! Nice!
And correct - due to BSON storing single point precision as a double, the conversion fails when coming back and being compared.

 

As an aside a colleague pointed out - the error returned talks about being truncated to an integer type, whereas this error is returned when dealing with floats. I'm guessing that this would cause a breaking API change though.

Comment by Kevin Albertson [ 15/Jun/21 ]

Hi matt@linc-ed.com, thank you for the report!

To check, is the behavior described occurring when trying to unmarshal a BSON double into a struct containing a float32 field? Would the following be a reproducing case?

type myStruct struct {
	F float32
}
 
func Test2046(t *testing.T) {
	doc, err := bson.Marshal(bson.D{{"f", float64(.1)}})
	if err != nil {
		log.Fatalf("Marshal error: %v", err)
	}
	var dst myStruct
	err = bson.Unmarshal(doc, &dst)
	if err != nil {
		log.Fatalf("Unmarshal error: %v", err)
	}
}
 
// The test above fails with the output
// Unmarshal error: error decoding key f: float64 can only be truncated to an integer type when truncation is enabled

Generated at Thu Feb 08 08:37:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.