We are building a pipeline using Apache Beam and DirectRunner as the runner. We are currently attempting a simple pipeline whereby we:
Pull data from Google Cloud Pub/Sub (currently using the emulator to run locally)
Deserialize into a Java object
Window events using fixed windows of 1 minute
Combine these windows using a custom CombineFn that transforms them from events into a list of events.
Pipeline code:
pipeline
.apply(PubsubIO.<String>read().topic(options.getTopic()).withCoder(StringUtf8Coder.of()))
.apply("ParseEvent", ParDo.of(new ParseEventFn()))
.apply("WindowOneMinute",Window.<Event>into(FixedWindows.of(Duration.standardMinutes(1))))
.apply("CombineEvents", Combine.globally(new CombineEventsFn()));
ParseEvent function:
static class ParseEventFn extends DoFn<String, Event> {
#ProcessElement
public void processElement(ProcessContext c) {
String json = c.element();
c.output(gson.fromJson(json, Event.class));
}
}
CombineEvents function:
public static class CombineEventsFn extends CombineFn<Event, CombineEventsFn.Accum, EventListWrapper> {
public static class Accum {
EventListWrapper eventListWrapper = new EventListWrapper();
}
#Override
public Accum createAccumulator() {
return new Accum();
}
#Override
public Accum addInput(Accum accumulator, Event event) {
accumulator.eventListWrapper.events.add(event);
return accumulator;
}
#Override
public Accum mergeAccumulators(Iterable<Accum> accumulators) {
Accum merged = createAccumulator();
for (Accum accum : accumulators) {
merged.eventListWrapper.events.addAll(accum.eventListWrapper.events);
}
return merged;
}
#Override
public EventListWrapper extractOutput(Accum accumulator) {
return accumulator.eventListWrapper;
}
}
When attempting to run this locally using Maven and DirectRunner, we are getting the following error:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Unable to return a default Coder for CombineEvents/Combine.perKey(CombineEvents)/Combine.GroupedValues/ParDo(Anonymous).out [PCollection]. Correct one of the following root causes:
No Coder has been manually specified; you may do so using .setCoder().
Inferring a Coder from the CoderRegistry failed: Unable to provide a default Coder for org.apache.beam.sdk.values.KV<K, OutputT>. Correct one of the following root causes:
Building a Coder using a registered CoderFactory failed: Cannot provide coder for parameterized type org.apache.beam.sdk.values.KV<K, OutputT>: Unable to provide a default Coder for java.lang.Object. Correct one of the following root causes:
Building a Coder using a registered CoderFactory failed: Cannot provide coder based on value with class java.lang.Object: No CoderFactory has been registered for the class.
Building a Coder from the #DefaultCoder annotation failed: Class java.lang.Object does not have a #DefaultCoder annotation.
Building a Coder from the fallback CoderProvider failed: Cannot provide coder for type java.lang.Object: org.apache.beam.sdk.coders.protobuf.ProtoCoder$2#6e610150 could not provide a Coder for type java.lang.Object: Cannot provide ProtoCoder because java.lang.Object is not a subclass of com.google.protobuf.Message; org.apache.beam.sdk.coders.SerializableCoder$1#7adc59c8 could not provide a Coder for type java.lang.Object: Cannot provide SerializableCoder because java.lang.Object does not implement Serializable.
Building a Coder from the #DefaultCoder annotation failed: Class org.apache.beam.sdk.values.KV does not have a #DefaultCoder annotation.
Using the default output Coder from the producing PTransform failed: Unable to provide a default Coder for org.apache.beam.sdk.values.KV<K, OutputT>. Correct one of the following root causes:
Building a Coder using a registered CoderFactory failed: Cannot provide coder for parameterized type org.apache.beam.sdk.values.KV<K, OutputT>: Unable to provide a default Coder for java.lang.Object. Correct one of the following root causes:
Building a Coder using a registered CoderFactory failed: Cannot provide coder based on value with class java.lang.Object: No CoderFactory has been registered for the class.
Building a Coder from the #DefaultCoder annotation failed: Class java.lang.Object does not have a #DefaultCoder annotation.
Building a Coder from the fallback CoderProvider failed: Cannot provide coder for type java.lang.Object: org.apache.beam.sdk.coders.protobuf.ProtoCoder$2#6e610150 could not provide a Coder for type java.lang.Object: Cannot provide ProtoCoder because java.lang.Object is not a subclass of com.google.protobuf.Message; org.apache.beam.sdk.coders.SerializableCoder$1#7adc59c8 could not provide a Coder for type java.lang.Object: Cannot provide SerializableCoder because java.lang.Object does not implement Serializable.
Building a Coder from the #DefaultCoder annotation failed: Class org.apache.beam.sdk.values.KV does not have a #DefaultCoder annotation.
at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at org.apache.beam.sdk.values.TypedPValue.getCoder(TypedPValue.java:51)
at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:130)
at org.apache.beam.sdk.values.TypedPValue.finishSpecifying(TypedPValue.java:90)
at org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:143)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:418)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:334)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
at org.apache.beam.sdk.transforms.Combine$Globally.expand(Combine.java:1459)
at org.apache.beam.sdk.transforms.Combine$Globally.expand(Combine.java:1336)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:420)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:350)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:167)
at ***************************.main(***************.java:231)
... 6 more
Apologies for the huge code dump - wanted to provide all the context.
I'm curious as to why it's complaining about no default coder for both java.lang.Object and org.apache.beam.sdk.values.KV<K, OutputT> - as far as I can tell our pipeline is changing types between String, Event, and EventListWrapper - the latter two classes have their default coders set on the class itself (AvroCoder in both cases).
The error is occurring on the line where we apply the CombineFn - can confirm that without this transform the pipeline works.
I suspect we've set up the combine transform incorrectly somehow, but as of yet have found nothing in the Beam documentation to point us in the right direction.
Any insight would be appreciated - thanks in advance!
The probable reason you are seeing java.lang.Object is because Beam is trying to infer a coder for an unresolved type variable, which will be resolved to Object. This may be a bug in how coder inference is done within Combine.
Separately, I would expect the Accum class to also cause a failure of coder inference. You can override getAccumulatorCoder in your CombineFn to provide one quite directly.
Did you check if adding Serializable to your Accumulator works directly?
So add"implements Serializable" to Accum class ...
public static class Accum implements Serializable {
EventListWrapper eventListWrapper = new EventListWrapper();
}
Related
I'm developing a custom binary Gradle plugin, following the pattern shown here.
My extension, VersionInfoExtension is abstract with abstract methods, just like the BinaryRepositoryExtension in the documentation. My task is also abstract, with abstract methods for the parameters, like the LatestArtifactVersion in the documentation. When I test, I get the following error:
An exception occurred applying plugin request [id: 'com.example.version-info']
> Failed to apply plugin 'com.example.version-info'.
> Could not create an instance of type com.example.gradle.version.VersionInfoExtension.
> Could not generate a decorated class for type VersionInfoExtension.
> Cannot have abstract method VersionInfoExtension.jars().
What am I missing? Are there any good examples of making this work?
I'm using gradle 7.X and Kotlin.
The "Cannot have abstract method myPropertyName" error is reported when the method name is prefixed by "is":
abstract val isRegistered: Property<Boolean>
That was annoying to track down. The type doesn't seem to matter.
The solution was to remove "is" from the name.
The problem here seems to be the name of the abstract method.
All configurable methods must be bean-methods - this holds for both Extensions and Tasks.
So you should have used (assuming a java class):
abstract Property<String> getJars()
instead of
abstract Property<String> jars()
In addition to #Richard Sitze answer, in my case it was a custom setter function that was used in tests only (annotated with #TestOnly) was preventing generating the class.
I had declared serializable object as #Input
#get:Input
abstract val myPropertry : Property<MyObject>
#Test
fun setMyProperty(obj : MyObject){
//...
}
Somehow the name of the function interfered with generating the property setter along generation of the setter.
I'm trying to mock a class derived from an Apache Beam generic class, and call a method of it using Mockito.
This is my real class:
public class MyClass extends DoFn<Entity, TableRow> {
public void processElement(ProcessContext c) {
// some business logic
c.output(new TableRow()) // c.output received a type defined in the derived class
}
}
And this is the test with the required mock:
DoFn<Entity, TableRow>.ProcessContext testContext = (DoFn<Entity, TableRow>.ProcessContext)mock(DoFn.ProcessContext.class);
when(textContext.output(any(TableRow.class))).thenReturn(42);
For some reason, I'm getting an error doing so, on the second line. That's the error:
Required type:
T
Provided:
void
reason: no instance(s) of type variable(s) T exist so that void conforms to T
Any solution for this?
Thanks!
An acceptable answer seems to be found in the comments, but I would also in general advise against writing tests by mocking out Beam DoFns and the like. Instead, the recommendation would be to either factor out the code in the body of the DoFn into something that can be more directly tested (if it's non-trivial) or, preferably, use the DoFn(s) in an actual pipeline and assert that it produces the correct results (see https://beam.apache.org/documentation/pipelines/test-your-pipeline/).
Here i'm reading spring-security-oauth2 source code and found there was a compile error in AuthorizationServerSecurityConfigurer.
below is the capture of such error:
this is errors Idea gives : Ambiguous method call: both AbstractConfiguredSecurityBuilder.getConfigurer(..) and HttpSecurityBuilder matches
But i know abstract class can implements methods both exists in its super class and interface in Java. but Idea do not know, how can i remove such error hint(having obsession about such red color).
At last, this is the hierarchy generated by Idea about class AuthorizationServerSecurityConfigurer:
I'm trying to bind an AAR library to a Xamarin android project.
Using jar2xml as AndroidClassParser in the binding project the main issue that i have is the fact that the AAR library contains a realm DB, realm creates some proxy interfaces that contains some variables with a $ inside the variable name, this result in this error
CS1056 Unexpected character '$'
instead using class-parse as AndroidClassParser result in other error such as this
CS0111 Already defines a member called 'Translate' with the same
parameter types
this is the java code that raise the CS0111 error
public interface ModelTranslator<T extends DomainClass, S> {
S translate(T from);
void translate(T from, S to);
T translate(S from);
void translate(S from, T to);
}
is there a way to fix this?
EDIT: I have fixed most of the issue inside Metadata.xml
the only things that remains to fix is a class that extends Comparator
converting this class result in this error.
CS0534 C# does not implement inherited abstract member
but the class implement everything that is present inside Comparator
This answer might help you
Unexpected Character $ - Java Binding error
If you do need this member, define a rename transformation in metadata like this: " name="managedName">ValidNameHere.
And here some useful blog post that shows the troubleshooting for many Binding errors scenarios:
https://www.codepool.biz/binding-android-library-xamarin-disaster.html
I'm trying to write a MWE2 workflow component using scala language.
Below is the scala code:
package com.ford.modelling.workflow
import org.eclipse.emf.mwe2.runtime.workflow.{IWorkflowComponent, IWorkflowContext}
class SayHello extends IWorkflowComponent {
var message : String = null;
def preInvoke {}
def invoke(ctx : IWorkflowContext) { System.out.println(message) }
def postInvoke {}
}
and below is the workflow code:
module com.ford.modelling.workflow.SomeWorklow
SayHello {}
I can't figure out why does this workflow complain for error:
'com.ford.modelling.workflow.SayHello' does not have a public default constructor.
I'd assume that the scala IDE plugin does not mimic the java protocol completely, e.g. the IType does not expose a no-args constructor. You may want to ask the scale folks about it.
The error message should vanish as soon as you add a default constructor explicitly. Does that make sense?
A quick google search indicated that there probably no syntax for a default constructor so I'd assume it's a scala tooling problem. Does the problem occur at runtime, too?