I want to store a uuid using avro.
below is the schema i am using
{
"namespace": "somethingImportant",
"type": "record",
"name": "GoodObject",
"fields": [
{
"name": "pk",
"type": {"type": "string", "logicalType": "uuid"}
}
]
}
i am using maven plugin 1.10.2 plugin with the below config
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.10.2</version>
<executions>
<execution>
<id>schemas</id>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
<goal>protocol</goal>
<goal>idl-protocol</goal>
</goals>
<configuration>
<stringType>String</stringType>
<customConversions>org.apache.avro.Conversions$UUIDConversion</customConversions>
<enableDecimalLogicalType>true</enableDecimalLogicalType>
<sourceDirectory>${project.basedir}/src/main/resources/</sourceDirectory>
<outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
the generated class the id field is generated as UUID, and i can do a get and set of uuid.
But uuid is stored as a string(36 bytes) which defeats the purpose. I came across another solution to define seomthing like
{
"namespace": "somethingImportant",
"type": "record",
"name": "GoodObject",
"fields": [
{
"name": "pk",
"type": {"type": "string", "logicalType": "uuid", "CustomEncoding":"UUIDAsBytesEncoding"}
}
]
}
but this does not do anything on its own. This stackoverflow answer talks about implementation, but i am not sure how we can make it work with maven plugin. This blog also talks about similar thing but i could not generate #AvroEncode(using = InstantAsStringAvroEncoding.class).
Has someone used uuid with avro so that
we can set a uuid
it gets stored as byte array
The Get method returns a uuid
I don't want to do the marshelling and unmarshelling by myself obviously because that is a custom code to maintain
Related
I have a Spring Boot application with Spring Cloud Sleuth, OpenTelemetry instrumentation and OpenTelemetry exporter OTLP.
This is a gist of dependencies:
spring-cloud-starter-sleuth without Brave, because we are using OpenTelemetry instrumentation
spring-cloud-sleuth-otel-autoconfigure which introduces OpenTelemetry instrumentation libs and provides Spring autoconfiguration
opentelemetry-exporter-otlp for sending data to apm server
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-brave</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-sleuth-otel-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/io.grpc/grpc-netty -->
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
<version>1.43.0</version>
</dependency>
I have only basic configuration in my application.yml:
spring:
sleuth:
enabled: true
otel:
config:
trace-id-ratio-based: 1.0
exporter:
otlp:
endpoint: http://localhost:8200
With this setup I successfully see some data in APM. Example screens:
However if I look into Elastic documentation, I see that their screens have additional data present: Traces.
To me, it looks like span and transaction names are missing (I only see HTTP GET instead of a name), at least they are present in the documentation images.
Anyone has an idea why is this happening and how to fix this?
This is example trace document in Elastic:
const EXAMPLE = {
"_index": "apm-7.15.2-metric-000001",
"_type": "_doc",
"_id": "AYVKCH8BxjGANUnHPDgq",
"_version": 1,
"_score": 1,
"_source": {
"_doc_count": 2,
"agent": {
"name": "opentelemetry/java"
},
"processor": {
"name": "metric",
"event": "metric"
},
"transaction.duration.histogram": {
"counts": [
1,
1
],
"values": [
1439,
10495
]
},
"metricset.name": "transaction",
"observer": {
"hostname": "0798ff612508",
"id": "6a12bcef-5e7e-45b3-aee6-f2af4e175c3f",
"ephemeral_id": "389ee9b1-d4c4-4d67-b46a-bfcaa77b7b79",
"type": "apm-server",
"version": "7.15.2",
"version_major": 7
},
"#timestamp": "2022-02-17T15:25:56.160Z",
"timeseries": {
"instance": "summary-service:HTTP GET:11ed2dc65a946e45"
},
"ecs": {
"version": "1.11.0"
},
"service": {
"name": "summary-service"
},
"event": {
"ingested": "2022-02-17T15:25:57.161730700Z",
"outcome": "success"
},
"transaction": {
"result": "HTTP 2xx",
"root": true,
"name": "HTTP GET",
"type": "request"
}
},
"fields": {
"transaction.name.text": [
"HTTP GET"
],
"_doc_count": [
2
],
"service.name": [
"summary-service"
],
"processor.name": [
"metric"
],
"observer.version_major": [
7
],
"observer.hostname": [
"0798ff612508"
],
"transaction.result": [
"HTTP 2xx"
],
"transaction.duration.histogram": [
{
"counts": [
1,
1
],
"values": [
1439,
10495
]
}
],
"transaction.type": [
"request"
],
"metricset.name": [
"transaction"
],
"observer.id": [
"6a12bcef-5e7e-45b3-aee6-f2af4e175c3f"
],
"event.ingested": [
"2022-02-17T15:25:57.161Z"
],
"#timestamp": [
"2022-02-17T15:25:56.160Z"
],
"observer.ephemeral_id": [
"389ee9b1-d4c4-4d67-b46a-bfcaa77b7b79"
],
"timeseries.instance": [
"summary-service:HTTP GET:11ed2dc65a946e45"
],
"observer.version": [
"7.15.2"
],
"ecs.version": [
"1.11.0"
],
"observer.type": [
"apm-server"
],
"transaction.root": [
true
],
"processor.event": [
"metric"
],
"transaction.name": [
"HTTP GET"
],
"agent.name": [
"opentelemetry/java"
],
"event.outcome": [
"success"
]
}
}
To me, it looks like span and transaction names are missing (I only see HTTP GET instead of a name)
No, they are not missing. The reason you are seeing name as HTTP GET is due to preference for less cardinal names and the semantic conventions for tracing data. There is a detailed explanation about the naming conventions for the HTTP spans here https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md#name. Any data generated by auto-instrumentation libraries will adhere to the semantic conventions specification. I am guessing the visualisation from the linked vendor is coming from manual instrumentation where you as an end user can give any name (although it is recommended everyone uses less cardinal values but there is not enforcement there). I don't think there is anything you can "fix" here.
I have openapi contract:
openapi: 3.0.1
info:
version: 1.0.0
paths:
/getInteractions:
post:
requestBody:
content:
application/json:
schema:
$ref: scheme/interactionsRq.json
required: true
responses:
"200":
content:
application/json:
schema:
$ref: scheme/mainRs.json
in this structure:
-resources
--GetInteractionController.yaml
--scheme
----interactionsRq.json
----interactionsRs.json
----mainRs.json
In mainRs.json i have some ref to another json like this:
"resultApi": {
"title": "result",
"type": "object",
"properties": {
"interactionList": {
"type": "array",
"minItems": 0,
"maxItems": 100,
"items": {
"$ref": "interactionsRs.json#/definitions/interactionApi"
}
}
}
And when i try to package this with openapi-generator-maven-plugin:
<plugin>
<groupId>org.openapitools</groupId>
<artifactId>openapi-generator-maven-plugin</artifactId>
<version>5.3.1</version>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<inputSpec>${project.basedir}/src/main/resources/GetInteractionController.yaml</inputSpec>
<generatorName>spring</generatorName>
<apiPackage>some.api.package</apiPackage>
<modelPackage>some.dto.package</modelPackage>
<supportingFilesToGenerate>ApiUtil.java</supportingFilesToGenerate>
<configOptions>
<delegatePattern>true</delegatePattern>
</configOptions>
</configuration>
</execution>
</executions>
</plugin>
I get a warning and build error:
[WARNING] Failed to get the schema name: ./scheme/interactionsRs.json#/definitions/interactionApi
Can openapi generate code with refs like i have? Or i need to refactor json schemas and delete this refs? Maybe concatenate this in one file or something like this
Now i understand, that this is an issue with my JSON scheme, but not with openapi-generator.
In JSON scheme i've used an jsonschema2pojo feature "customDateTimePattern" : "yyyy-MM-dd'T'HH:mm:ssZ" and it gives me an error.
When i deleted it everything become fine
I'm using Open-API to generate java class using yaml file. when i run
mvn clean install
i'm getting this error :
unexpected error in Open-API generation
org.openapitools.codegen.SpecValidationException: There were issues with the specification. The option can be disabled via validateSpec (Maven/Gradle) or --skip-validate-spec (CLI).
| Error count: 3, Warning count: 6
Errors:
-attribute paths.'/path/{id}'(delete).operationId is repeated
-attribute paths.'/path/name'(get).operationId is repeated
How can i skeep this validation ?
Try this:
Within your POM.xml -> Plugin -> Find openAPI generation plug-in -> configuration -> configOptions ->
<validateSpec>false</validateSpec>
This should work hopefully ! :)
For me, the correct place of validateSpec was configuration instead of configOptions
<plugin>
<groupId>org.openapitools</groupId>
<artifactId>openapi-generator-maven-plugin</artifactId>
<version>6.2.1</version>
<executions>
<execution>
<id>abc-api</id>
<goals>
<goal>generate</goal>
</goals>
<configuration>
...
<validateSpec>false</validateSpec>
<skipValidateSpec>true</skipValidateSpec>
<configOptions>
<developerOrganization>ABC</developerOrganization>
...
</configOptions>
</configuration>
</execution>
</executions>
</plugin>
Note: I have to add both validateSpec and skipValidateSpec for version 6.2.1 as per suggestion in an open bug https://github.com/OpenAPITools/openapi-generator/issues/9815
Try to add syntax "schema": {"type": "string"} to *.Json file.
I aslo got the error with respose like that:
There were issues with the specification. The option can be disabled via validateSpec (Maven/Gradle) or --skip-validate-spec (CLI).
| Error count: 12, Warning count: 20
Errors:
-attribute paths.'/.../items'(get).responses.304.ETag.type is unexpected
-attribute paths.'/.../items'(get).responses.304.Last-Modified.type is unexpected
-attribute paths.'/.../items'(get).responses.200.ETag.type is unexpected
This is my fix:
enter image description here
enter image description here
"headers": {
"ETag": {
"description": "Used as caching key for future requests.",
"schema": {
"type": "string",
"example": "33a64df551425fcc55e4d42a1466666666666666666"
}
},
"Cache-Control": {
"description": "The Cache-Control header.",
"schema": {
"type": "string",
"example": "max-age=1000000"
}
},
"Last-Modified": {
"description": "The last modified the response.",
"schema": {
"type": "string",
"example": "Sun, 11 Nov 2021 15:30:51 ICT"
}
}
}
I hope this one can help you. You cant read more document with response header here: https://swagger.io/docs/specification/describing-responses/
Given the following Avro Schema:
{
"namespace": "ro.dspr.coreentities",
"type": "record",
"name": "Organization",
"fields": [
{
"name": "id",
"type": "string",
"logicalType": "uuid"
},
{
"name": "name",
"type": "string"
},
{
"name": "description",
"type": "string"
}
]
}
Running avro-maven-plugin 1.9.0 goal schema, I get:
#org.apache.avro.specific.AvroGenerated
public class Organization extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"Organization\",\"namespace\":\"ro.dspr.coreentities\",\"fields\":[{\"name\":\"id\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"},\"logicalType\":\"uuid\"},{\"name\":\"name\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}},{\"name\":\"description\",\"type\":{\"type\":\"string\",\"avro.java.string\":\"String\"}}]}");
// truncated
#Deprecated public java.lang.String id;
#Deprecated public java.lang.String name;
#Deprecated public java.lang.String description;
// truncated
}
I want the generated POJO for Organization to have id UUID, not String (what I have now).
Links I looked at:
I do see the logical type def from Avro and there is the Conversion class I am actually trying to trigger, but I cannot connect the dots.
Other
Relevant Maven pom parts
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>${avro.version}</version>
<configuration>
<sourceDirectory>${avro-files-path}</sourceDirectory>
<outputDirectory>${project.build.directory}/generated-sources</outputDirectory>
<stringType>String</stringType>
</configuration>
<executions>
<execution>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${avro-files-path}</sourceDirectory>
<outputDirectory>${project.build.directory}/generated-sources</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
Extra info
I actually trying to use Avro to give my Kafka messages a structure. I am also using Confluent Schema Registry and Confluent Avro Kafka Serializer. Nevertheless, I thought I would only have the id as String, but if I try to send messages to Kafka as something non-UUID, it will fail later. However, I found out that there is actually no constraint, and I managed to send any String to Kafka. So, the "logicalType" in Avro is not enforced at all.
The Question
How can I generate Organization.class#id as UUID?
If there is no Avro-support in doing this, what would be the workaround (preferably reusing the org.apache.avro.Conversions.UUIDConversion) ?
Here is a functional example:
Schema:
{
"name": "pk",
"type": {"type": "string", "logicalType": "uuid"}
},
Maven config:
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.9.1</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/../avro/schemas/commands</sourceDirectory>
<outputDirectory>${project.build.directory}/generated-sources/java</outputDirectory>
<!-- TODO Enable this string type once we have 1.10.0 release, or 1.9.2-->
<!--<stringType>String</stringType>-->
<fieldVisibility>PRIVATE</fieldVisibility>
<customConversions>org.apache.avro.Conversions$UUIDConversion</customConversions>
<imports>
<import>${project.basedir}/../avro/schemas/common</import>
<import>${project.basedir}/../avro/schemas/commands/account</import>
<import>${project.basedir}/../avro/schemas/commands/rec</import>
</imports>
</configuration>
</execution>
</executions>
</plugin>
The important thing to note is that there is a conflict between the UUIDConversion and String in the current release of the plugin, as such you either need to choose which is more important, or use 1.10.0-SNAPSHOT which has a fix merged, but not yet released.
I have the following Avro schema:
{
"namespace":"com.example",
"type":"record",
"name":"BuggyRecord",
"fields":[
{
"name":"my_mandatory_date",
"type":{
"type":"long",
"logicalType":"timestamp-millis"
},
"default":1502250227187
},
{
"name":"my_optional_date",
"type":[
{
"type":"long",
"logicalType":"timestamp-millis"
},
"null"
],
"default":1502250227187
}
]
}
I use maven to generate the Java file.
Avro config:
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.8.2</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
<!--<goal>idl-protocol</goal>-->
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
<outputDirectory>${project.basedir}/src/main/java</outputDirectory>
<enableDecimalLogicalType>true</enableDecimalLogicalType>
<stringType>String</stringType>
</configuration>
</execution>
</executions>
</plugin>
The generated class through mvn compile code fails on some basic code:
#Test
public void Foo(){
BuggyRecord.Builder buggyRecordBuilder = BuggyRecord.newBuilder();
buggyRecordBuilder.build();
}
Error code:
org.apache.avro.AvroRuntimeException: java.lang.ClassCastException: java.lang.Long cannot be cast to org.joda.time.DateTime
at com.example.BuggyRecord$Builder.build(BuggyRecord.java:301)
at BuggyRecordTest.Foo(BuggyRecordTest.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to org.joda.time.DateTime
at com.example.BuggyRecord$Builder.build(BuggyRecord.java:298)
... 23 more
Which I think is due to the Converters not being generated properly:
private static final org.apache.avro.Conversion<?>[] conversions =
new org.apache.avro.Conversion<?>[] {
TIMESTAMP_CONVERSION,
null, // <------ THIS ONE IS NOT SET PROPERLY
null
};
Is it me mis-using the code generator or is it a bug?
{
"name":"my_optional_date",
"type":[ "null", {"type" : "long", "logicalType": "timestamp-millis"}], "default": null
}
More of a workaround than a solution - put "null" type first and "default": null
We ran into a similar issue, with logicalType of date:
{
"name": "date_field",
"type": [
"null",
{
"type": "int",
"logicalType": "date"
}
],
"default": null
}
We needed to update to the latest version (0.17.0) of the gradle plugin gradle-avro-plugin, and the Apache Avro dependency (1.9.1).
Also in our gradle build file, we needed to add
avro {
dateTimeLogicalType = "JODA"
}
as well as a dependency on joda-time
dependencies {
compile "joda-time:joda-time:2.10.3"
}
The default dateTimeLogicalType used by gradle-avro-plugin is JSR310, but that still caused errors (even though we are using Java 8).
Try upgrading the version of avro to 1.9.X in avro dependency and avro-maven-plugin in pom.xml
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.9.1</version>
</dependency>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.9.1</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
<!--<goal>idl-protocol</goal>-->
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
<outputDirectory>${project.basedir}/src/main/java</outputDirectory>
<enableDecimalLogicalType>true</enableDecimalLogicalType>
<stringType>String</stringType>
</configuration>
</execution>
</executions>
</plugin>
Also make sure to delete previously generated class from AVRO schema and do mvn compile.