This project is read-only.
1
Vote

Avro JsonSchemaBuilder: "fixed" type references fail to resolve

description

This Avro schema works with JsonSchemaBuilder:
{
    "type": "record",
    "name": "RequestHeader",
    "fields": [
        { "name": "ID", "type": { "type": "fixed", "name": "System.Guid", "size": 16 } }
    ]
}
However this one fails:
[
    {
        "type": "record",
        "name": "LogJam.RequestHeader",
        "fields": [
            { "name": "ID", "type": { "type": "fixed", "name": "System.Guid", "size": 16 } }
        ]
    },
    {
        "type": "record",
        "name": "LogJam.ApplicationEntry",
        "fields": [
            { "name": "Name", "type": [ "null", "string" ] },
            { "name": "Assembly", "type": [ "null", "string" ] },
            { "name": "Version", "type": [ "null", "string" ] },
            { "name": "MachineName", "type": [ "null", "string" ] },
            { "name": "ApplicationInstanceId", "type": "System.Guid" }
        ]
    }
]
The reference to fixed type named "System.Guid" fails, with this call stack:
Result Message: System.Collections.Generic.KeyNotFoundException : The given key was not present in the dictionary.
Result StackTrace:  
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.CreatePrimitiveTypeSchema(String type, Dictionary`2 attributes) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 410
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.ParsePrimitiveTypeFromString(String token) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 382
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.Parse(JToken token, NamedSchema parent, Dictionary`2 namedSchemas) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 105
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.ParseRecordField(JObject field, NamedSchema parent, Dictionary`2 namedSchemas, Int32 position) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 349
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.<>c__DisplayClass3.<ParseRecordType>b__2(JToken field, Int32 index) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 318
   at System.Linq.Enumerable.<SelectIterator>d__1`2.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Microsoft.Hadoop.Avro.Schema.JsonExtensions.ReadArrayProperty[T](JToken token, String propertyName, Boolean isOptional, Func`3 func) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonExtensions.cs:line 195
   at Microsoft.Hadoop.Avro.Schema.JsonExtensions.OptionalArrayProperty[T](JToken token, String propertyName, Func`3 func) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonExtensions.cs:line 78
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.ParseRecordType(JObject record, NamedSchema parent, Dictionary`2 namedSchemas) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 309
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.ParseJsonObject(JObject token, NamedSchema parent, Dictionary`2 namedSchemas) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 140
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.Parse(JToken token, NamedSchema parent, Dictionary`2 namedSchemas) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 88
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.ParseUnionType(JArray unionToken, NamedSchema parent, Dictionary`2 namedSchemas) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 180
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.Parse(JToken token, NamedSchema parent, Dictionary`2 namedSchemas) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 110
   at Microsoft.Hadoop.Avro.Schema.JsonSchemaBuilder.BuildSchema(String schema) in d:\src\codeplex\hadoopsdk\src\Microsoft.Hadoop.Avro\Schema\JsonSchemaBuilder.cs:line 71
This prevents reading Avro streams which contain such schema elements using AvroContainer.CreateGenericReader.

comments

maxluk wrote Jun 23, 2015 at 6:36 PM

Avro project has graduated to Azure github here: https://github.com/hdinsight/azure-sdk-for-net/tree/master/src/HDInsight

Can you please submit your PR there?