[JAVA-3372] AutomaticPojoCodec fails when encoding POJOs of code-generated Avro classes Created: 05/Aug/19  Updated: 27/Oct/23  Resolved: 11/Nov/19

Status: Closed
Project: Java Driver
Component/s: Codecs
Affects Version/s: 3.10.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: F H Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File MongoDB Avro bug.txt     Zip Archive mongodb-avro-bug.zip    

 Description   

When encoding POJOs of code-generated [Avro|https://avro.apache.org/] classes, the AutomaticPojoCodec fails with the error below. I have used the documentation to encode POJOs into MongoDB Documents, but can not make it work with the Avro classes. This part of the documentation has been added to the MongoDB java driver, to make automatic POJO encoding work:  https://mongodb.github.io/mongo-java-driver/3.10/driver/getting-started/quick-start-pojo/#creating-a-custom-codecregistry

 

My java classes are being auto-generated from Avro schemas by using the Avro Maven plugin. Essentially, you drop in a Avro schema file into your project and the Maven plugin will use Avros code generation to create the java classes. These classes are not meant to be changed by the user after code generation.

 

The error I get when encoding Avro classes is: 

 

org.bson.codecs.configuration.CodecConfigurationException: An exception occurred when encoding using the AutomaticPojoCodec.Encoding a ClientRoleRequest: '{JSON_PAYLOAD_DATA_HERE}' failed with the following exception:Failed to encode 'ClientRoleRequest'. Encoding 'classSchema' errored with: 
An exception occurred when encoding using the AutomaticPojoCodec.Encoding a RecordSchema: 
'{MY_AVRO_SCHEMA_HERE}' failed with the following exception:
Unable to get value for property 'aliases' in RecordSchemaA custom Codec or PojoCodec may need to be explicitly configured 
and registered to handle this type.A custom Codec or PojoCodec may need to be explicitly configured and registered to handle 
this type.
at org.bson.codecs.pojo.AutomaticPojoCodec.encode(AutomaticPojoCodec.java:53)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
at com.mongodb.operation.BulkWriteBatch$WriteRequestEncoder.encode(BulkWriteBatch.java:398)
at com.mongodb.operation.BulkWriteBatch$WriteRequestEncoder.encode(BulkWriteBatch.java:377)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63)
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)at com.mongodb.internal.connection.BsonWriterHelper.writeDocument(BsonWriterHelper.java:75)
at com.mongodb.internal.connection.BsonWriterHelper.writePayload(BsonWriterHelper.java:59)
at com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:146)
at com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
at com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:60)
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceiveAsync(InternalStreamConnection.java:325)
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceiveAsync(UsageTrackingInternalConnection.java:114)
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceiveAsync(DefaultConnectionPool.java:455)
at com.mongodb.internal.connection.CommandProtocolImpl.executeAsync(CommandProtocolImpl.java:78)
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.executeAsync(DefaultServer.java:215)
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocolAsync(DefaultServerConnection.java:285)
at com.mongodb.internal.connection.DefaultServerConnection.commandAsync(DefaultServerConnection.java:156)
at com.mongodb.operation.MixedBulkWriteOperation.executeCommandAsync(MixedBulkWriteOperation.java:442)
at com.mongodb.operation.MixedBulkWriteOperation.executeBatchesAsync(MixedBulkWriteOperation.java:348)
at com.mongodb.operation.MixedBulkWriteOperation.access$900(MixedBulkWriteOperation.java:70)
at com.mongodb.operation.MixedBulkWriteOperation$2$1.call(MixedBulkWriteOperation.java:237)
at com.mongodb.operation.OperationHelper.validateWriteRequests(OperationHelper.java:177)
at com.mongodb.operation.MixedBulkWriteOperation$2.call(MixedBulkWriteOperation.java:220)
at com.mongodb.operation.OperationHelper$7.onResult(OperationHelper.java:517)
at com.mongodb.operation.OperationHelper$7.onResult(OperationHelper.java:514)
at com.mongodb.internal.connection.DefaultServer$1.onResult(DefaultServer.java:109)
at com.mongodb.internal.connection.DefaultServer$1.onResult(DefaultServer.java:98)
at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:49)
at com.mongodb.internal.connection.DefaultConnectionPool$2.onResult(DefaultConnectionPool.java:213)
 
at com.mongodb.internal.connection.DefaultConnectionPool$2.onResult(DefaultConnectionPool.java:198)at com.mongodb.internal.connection.UsageTrackingInternalConnection$1.onResult(UsageTrackingInternalConnection.java:65)
at com.mongodb.internal.connection.UsageTrackingInternalConnection$1.onResult(UsageTrackingInternalConnection.java:57)
at com.mongodb.internal.connection.InternalStreamConnection$1$1.onResult(InternalStreamConnection.java:166)
at com.mongodb.internal.connection.InternalStreamConnection$1$1.onResult(InternalStreamConnection.java:153)at com.mongodb.internal.connection.InternalStreamConnectionInitializer$3.onResult(InternalStreamConnectionInitializer.java:215)at com.mongodb.internal.connection.InternalStreamConnectionInitializer$3.onResult(InternalStreamConnectionInitializer.java:209)at com.mongodb.internal.connection.CommandHelper$1.onResult(CommandHelper.java:59)at com.mongodb.internal.connection.CommandHelper$1.onResult(CommandHelper.java:53)at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:395)at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:372)at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:667)at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:634)at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:507)at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:220)at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:203)at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)at com.mongodb.internal.connection.AsynchronousSocketChannelStream$AsynchronousSocketChannelAdapter.read(AsynchronousSocketChannelStream.java:137)at com.mongodb.internal.connection.AsynchronousChannelStream.readAsync(AsynchronousChannelStream.java:105)at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:507)at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:74)at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:624)at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:609)at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:507)at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:220)at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:203)at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishRead(UnixAsynchronousSocketChannelImpl.java:430)at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:191)at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293)at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
Unable to get value for property 'aliases' in RecordSchemaA custom Codec or PojoCodec may need to be explicitly configured and registered to handle this type.at org.bson.codecs.pojo.PojoCodecImpl.encodeValue(PojoCodecImpl.java:182)at org.bson.codecs.pojo.PojoCodecImpl.encodeProperty(PojoCodecImpl.java:168)at org.bson.codecs.pojo.PojoCodecImpl.encode(PojoCodecImpl.java:105)at org.bson.codecs.pojo.AutomaticPojoCodec.encode(AutomaticPojoCodec.java:50)... 71 moreCaused by: org.bson.codecs.configuration.CodecConfigurationException: An exception occurred when encoding using the AutomaticPojoCodec.Encoding a RecordSchema: '{MY_AVRO_SCHEMA_HERE}' failed with the following exception:Unable to get value for property 'aliases' in RecordSchemaA custom Codec or PojoCodec may need to be explicitly configured and registered to handle this type.at org.bson.codecs.pojo.AutomaticPojoCodec.encode(AutomaticPojoCodec.java:53)at org.bson.codecs.pojo.PojoCodecImpl.encode(PojoCodecImpl.java:109)at org.bson.codecs.pojo.LazyPojoCodec.encode(LazyPojoCodec.java:47)at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)at org.bson.codecs.pojo.PojoCodecImpl.encodeValue(PojoCodecImpl.java:180)... 74 moreCaused by: org.bson.codecs.configuration.CodecConfigurationException: Unable to get value for property 'aliases' in RecordSchemaat org.bson.codecs.pojo.PropertyAccessorImpl.getError(PropertyAccessorImpl.java:69)at org.bson.codecs.pojo.PropertyAccessorImpl.get(PropertyAccessorImpl.java:45)at org.bson.codecs.pojo.PojoCodecImpl.encodeProperty(PojoCodecImpl.java:167)at org.bson.codecs.pojo.PojoCodecImpl.encode(PojoCodecImpl.java:105)at org.bson.codecs.pojo.AutomaticPojoCodec.encode(AutomaticPojoCodec.java:50)... 78 moreCaused by: java.lang.IllegalAccessException: Class org.bson.codecs.pojo.PropertyAccessorImpl can not access a member of class org.apache.avro.Schema$NamedSchema with modifiers "public"at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102)at java.lang.reflect.AccessibleObject.slowCheckMemberAccess(AccessibleObject.java:296)at java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:288)at java.lang.reflect.Method.invoke(Method.java:491)at org.bson.codecs.pojo.PropertyAccessorImpl.get(PropertyAccessorImpl.java:37)... 81 more



 Comments   
Comment by Tim te Beek [ 17/Nov/20 ]

Here's hoping this saves someone some time, as I had to go through a few iterations to get this right-ish for my use case:

public class AvroCustomerOrderCodec implements Codec<CustomerOrder> {
 
    @Override
    public void encode(BsonWriter writer, CustomerOrder value, EncoderContext encoderContext) {
        try {
            String record = serialize(value);
            writer.writeStartDocument();
            // Store both serialized record and schema, as old schema is needed to deserialize after schema changes
            writer.writeString("schema", value.getSchema().toString());
            writer.writeString("record", record);
            writer.writeEndDocument();
        } catch (IOException e) {
            throw new RuntimeException("Failed to serialize " + value, e);
        }
    }
 
    @Override
    public CustomerOrder decode(BsonReader reader, DecoderContext decoderContext) {
        reader.readStartDocument();
        String schema = reader.readString("schema");
        String record = reader.readString("record");
        reader.readEndDocument();
        try {
            return deserialize(schema, record);
        } catch (IOException e) {
            throw new RuntimeException("Failed to deserialize " + record, e);
        }
    }
 
    @Override
    public Class<CustomerOrder> getEncoderClass() {
        return CustomerOrder.class;
    }
 
    private static String serialize(CustomerOrder avro) throws IOException {
        // Store both serialized record and schema, as old schema is needed to deserialize after schema changes
        DatumWriter<CustomerOrder> writer = new SpecificDatumWriter<>(avro.getSchema());
        ByteArrayOutputStream stream = new ByteArrayOutputStream();
        Encoder jsonEncoder = EncoderFactory.get().jsonEncoder(avro.getSchema(), stream);
        writer.write(avro, jsonEncoder);
        jsonEncoder.flush();
        return stream.toString(StandardCharsets.UTF_8);
    }
 
    private static CustomerOrder deserialize(String schemaJson, String record) throws IOException {
        // Extract stored schema from first row; needed to deserialize after schema changes
        Schema schema = new org.apache.avro.Schema.Parser().parse(schemaJson);
        DatumReader<CustomerOrder> reader = new SpecificDatumReader<>(schema);
        Decoder decoder = DecoderFactory.get().jsonDecoder(schema, record);
        return reader.read(null, decoder);
    }
}

Comment by Ross Lawley [ 22/Oct/19 ]

I think the best approach would be try a custom convention would be simplest. However, you could create a custom AvroCodecProvider that uses an Avro schema file to generate a Codec - however, that would be much more work.

j9dy Have you tried a convention to remove the avro specific properties?

Ross

Comment by F H [ 18/Oct/19 ]

I was wondering if a solution like in the MongoDB Kafka Connector might be feasible here as well:

https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/sink/converter/AvroJsonSchemafulRecordConverter.java

Comment by Ross Lawley [ 07/Oct/19 ]

Looking at the zip file the generated class it looks like the avro metadata cannot be processed. The following propertyModels are generated automatically:

  • classSchema
  • decoder
  • encoder
  • id
  • schema
  • specificData
  • whatever

The issue is not all properties are serializable automatically. It maybe that a custom convention can be used to removed the avro based metadata (classSchema, decoder, encoder, schema and specificData)

Ross

Comment by F H [ 19/Aug/19 ]

I have a prepared a fully-working example to reproduce this issue. You can find it attached to this post.

mongodb-avro-bug.zip

Comment by Esha Bhargava [ 19/Aug/19 ]

j9dy Can you attach the Java source file for the generated Avro class?

Comment by F H [ 05/Aug/19 ]

I have added the exception text in another file here, as the Code snippet above is not really readable? Hope this helps.

Generated at Thu Feb 08 08:59:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.