I want to create a dataflow job template that takes the filename in GCS and publishes it to a PubSub topic. I followed the tutorial at this link, but that doesn't seem to be working for me.
My class definition is the following -
import com.google.cloud.dataflow.sdk.Pipeline;
import com.google.cloud.dataflow.sdk.io.PubsubIO;
import com.google.cloud.dataflow.sdk.io.TextIO;
import com.google.cloud.dataflow.sdk.options.PipelineOptionsFactory;
import com.google.cloud.dataflow.sdk.runners.TemplatingDataflowPipelineRunner;
public class PubSubOutputTest {
public static void main(String[] args) {
// Create pipeline options.
pubSubOutputOptions options = PipelineOptionsFactory.fromArgs(args).as(pubSubOutputOptions.class);
options.setRunner(TemplatingDataflowPipelineRunner.class);
options.setTempLocation("gs://staging-bucket");
Pipeline p = Pipeline.create(options);
// Read the file from the GCS Bucket.
p.apply(TextIO.Read.named("Read file from GCS.").from(options.getInputFile()).withoutValidation())
.apply(PubsubIO.Write.named("Write to Pub Sub topic.")
.topic("projects/my-project/topics/my-topic"));
// Run the pipeline.
p.run();
}
}
The interface that implements the ValueProvider to grab the runtime inputs is the following -
import com.google.cloud.dataflow.sdk.options.Default;
import com.google.cloud.dataflow.sdk.options.PipelineOptions;
import com.google.cloud.dataflow.sdk.options.ValueProvider;
public interface pubSubOutputOptions extends PipelineOptions {
#Default.String("gs://default-file.txt")
ValueProvider getInputFile();
void setInputFile(ValueProvider value);
}
The template creation is giving the following error.
Exception in thread "main" java.lang.IllegalArgumentException: PipelineOptions specified failed to serialize to JSON.
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:408)
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:146)
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.run(DataflowPipelineRunner.java:570)
at com.google.cloud.dataflow.sdk.runners.TemplatingDataflowPipelineRunner.run(TemplatingDataflowPipelineRunner.java:137)
at com.google.cloud.dataflow.sdk.runners.TemplatingDataflowPipelineRunner.run(TemplatingDataflowPipelineRunner.java:44)
at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:181)
at com.my.project.dataflow.PubSubOutputTest.main(PubSubOutputTest.java:32)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected IOException (of type java.io.IOException): Failed to serialize and deserialize property 'inputFile' with value 'RuntimeValueProvider{propertyName=inputFile, default=gs://default-file.txt, value=null}'
at com.fasterxml.jackson.databind.JsonMappingException.fromUnexpectedIOE(JsonMappingException.java:284)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3008)
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:406)
I am new to Google Cloud Dataflow and Java. I implemented everything in the documentation but I could have missed something obvious.
It looks like there's an error with how you're declaring the option. I think you want to provide a template parameter to ValueProvider, like so:
#Default.String("gs://default-file.txt")
ValueProvider<String> getInputFile();
void setInputFile(ValueProvider<String> value);
Related
I'm trying to change some method code so it returns a fixed value, instead of actually executing the original code.
The idea is to create a loader that, using a json that specifies some class methods, specifies the result it should return (another json serializated instance). Something like that:
{
"overrides": [
{
"className": "a.b.C",
"method": "getUser",
"returns": "{ \"name\": \"John\" }"
}
}
}
The result should be that, each time C.getUser() is called, the serializated instance shoud be returned instead of actually executing the method (all this without changing the source code).
I've tried something like this:
ByteBuddyAgent.install();
final ClassReloadingStrategy classReloadingStrategy = ClassReloadingStrategy.fromInstalledAgent();
new ByteBuddy().redefine(className).method(ElementMatchers.named(methodName))
.intercept(FixedValue.nullValue()).make()
.load(clase.getClassLoader(), classReloadingStrategy);
And it returns null instead of executing the method body but, how can I return the deserializated result? When I try to une a FixedValue.value(deserializatedInstance) it throws the following exception:
> java.lang.RuntimeException: java.lang.IllegalStateException: Error invoking java.lang.instrument.Instrumentation#retransformClasses
at es.abanca.heracles.ServiceLauncher.addTiming2(ServiceLauncher.java:129)
at es.abanca.heracles.ServiceLauncher.establecerMocks(ServiceLauncher.java:65)
at es.abanca.heracles.ServiceLauncher.run(ServiceLauncher.java:40)
at es.abanca.heracles.MifidServiceLauncher.main(MifidServiceLauncher.java:6)
Caused by: java.lang.IllegalStateException: Error invoking java.lang.instrument.Instrumentation#retransformClasses
at net.bytebuddy.dynamic.loading.ClassReloadingStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(ClassReloadingStrategy.java:503)
at net.bytebuddy.dynamic.loading.ClassReloadingStrategy$Strategy$2.apply(ClassReloadingStrategy.java:568)
at net.bytebuddy.dynamic.loading.ClassReloadingStrategy.load(ClassReloadingStrategy.java:225)
at net.bytebuddy.dynamic.TypeResolutionStrategy$Passive.initialize(TypeResolutionStrategy.java:100)
at net.bytebuddy.dynamic.DynamicType$Default$Unloaded.load(DynamicType.java:6156)
at es.abanca.heracles.ServiceLauncher.addTiming2(ServiceLauncher.java:127)
... 3 more
Caused by: java.lang.UnsupportedOperationException: class redefinition failed: attempted to change the schema (add/remove fields)
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at net.bytebuddy.dynamic.loading.ClassReloadingStrategy$Dispatcher$ForJava6CapableVm.retransformClasses(ClassReloadingStrategy.java:495)
... 8 more
Thank you all in advance
Your approach only works if the target class was not loaded yet.
When trying to modify (i.e. retransform) a class which was already loaded, as opposed to a class which is just being loaded (i.e. redefine), you need to make sure you
use retransform instead of redefine and
avoid manipulating the target class structure via .disableClassFormatChanges().
I am not a ByteBuddy expert, but I know you can modify methods without changing the class structure via Advice API. I usually do it like this:
import net.bytebuddy.agent.ByteBuddyAgent;
import net.bytebuddy.agent.builder.AgentBuilder;
import net.bytebuddy.asm.Advice;
import net.bytebuddy.dynamic.ClassFileLocator;
import org.acme.Sub;
import java.lang.instrument.Instrumentation;
import static net.bytebuddy.agent.builder.AgentBuilder.RedefinitionStrategy.RETRANSFORMATION;
import static net.bytebuddy.implementation.bytecode.assign.Assigner.Typing.DYNAMIC;
import static net.bytebuddy.matcher.ElementMatchers.is;
import static net.bytebuddy.matcher.ElementMatchers.named;
class Scratch {
public static void main(String[] args) {
System.out.println(new Sub("original name").getName());
Instrumentation instrumentation = ByteBuddyAgent.install();
new AgentBuilder.Default()
.with(RETRANSFORMATION)
.with(AgentBuilder.Listener.StreamWriting.toSystemError().withTransformationsOnly())
.disableClassFormatChanges()
.type(is(Sub.class))
.transform((builder, typeDescription, classLoader, module) ->
builder.visit(
Advice
.to(MyAspect.class, ClassFileLocator.ForClassLoader.ofSystemLoader())
.on(named("getName"))
)
)
.installOn(instrumentation);
System.out.println(new Sub("original name").getName());
}
static class MyAspect {
#Advice.OnMethodEnter(skipOn = Advice.OnDefaultValue.class)
public static boolean before() {
// Default value for boolean is false -> skip original method execution
return false;
}
#Advice.OnMethodExit(onThrowable = Throwable.class, backupArguments = false)
public static void after(
#Advice.Return(readOnly = false, typing = DYNAMIC) Object returnValue
)
{
System.out.println("MyAspect");
// Here you can define your return value of choice, null or whatever else
returnValue = "dummy name";
}
}
}
The console log would be something like:
original name
[Byte Buddy] TRANSFORM org.acme.Sub [sun.misc.Launcher$AppClassLoader#18b4aac2, null, loaded=true]
MyAspect
dummy name
There might be a simpler way. If so, Rafael Winterhalter definitely knows better than I.
Your problem is that you are enhancing a class in way that makes it ineligible for retransformation since you are changing the class's signature. You can force Byte Buddy into attempting to preserve the original shape by adding:
new ByteBuddy().with(Implementation.Context.Disabled.Factory.INSTANCE)
Doing so, Byte Buddy disables a few features but for a simple instrumentation such as FixedValue this has no effect.
I'm trying to use bndtools to create my OSGI program. Here is my previous code, and it can work well with the felix console.
package com.buaa.ate.service.data.command;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;
import org.apache.felix.service.command.CommandProcessor;
import com.buaa.ate.service.api.data.Publisher;
#Component(
service=PublishCommand.class,
property={
CommandProcessor.COMMAND_SCOPE + ":String=example",
CommandProcessor.COMMAND_FUNCTION + ":String=publish",
}
)
public class PublishCommand {
private Publisher publishSvc;
#Reference
public void setPublisher(Publisher publishSvc) {
this.publishSvc = publishSvc;
}
public void publish(String content) {
publishSvc.start();
long result = publishSvc.publish(content);
System.out.println(result);
publishSvc.stop();
}
}
Now, I want to change the annotation like this:
package com.buaa.ate.service.data.command;
import org.osgi.service.component.annotations.Component;
import org.osgi.service.component.annotations.Reference;
import org.apache.felix.service.command.CommandProcessor;
import com.buaa.ate.service.api.data.Publisher;
#Component(
service=PublishCommand.class,
properties="com/buaa/ate/service/data/command/config.properties"
)
public class PublishCommand {
private Publisher publishSvc;
#Reference
public void setPublisher(Publisher publishSvc) {
this.publishSvc = publishSvc;
}
public void publish(String content) {
publishSvc.start();
long result = publishSvc.publish(content);
System.out.println(result);
publishSvc.stop();
}
}
And this is my properties file:
config.properties
It's content like this:
osgi.command.scope\:String:example
osgi.command.function\:String:publish
When I run the program, input the command 'publish something', and then the problem happens:
'gogo: CommandNotFoundException: Command not found: publish'
So, what should I do to fix the problem?
Well, I just realize that it's so easy to fix the problem. This is a part of the osgi javadoc:
property
public abstract java.lang.String[] property
Properties for this Component.
Each property string is specified as "key=value". The type of the property value can be specified in the key as key:type=value. The type must be one of the property types supported by the type attribute of the property element of a Component Description.
To specify a property with multiple values, use multiple key, value pairs. For example, "foo=bar", "foo=baz".
See Also:
"The property element of a Component Description."
Default:{}
So I add the 'type' property to config.properties, and then the code can work well. Here is the current properties file:
current properties file
And it's content like this:
osgi.command.scope=example
osgi.command.scope\:type:String
osgi.command.function=publish
osgi.command.function\:type:String
The program can work well now.
I have a bunch of source files for java classes. I want to find those classes which are annotated by a given annotation class. The names of those classes should be written to a service provider list file.
Is there any machinery I could use to help me with this task? Or do I have to implement this myself from scratch?
If I had to do this myself, there are several approaches I can think of.
Write an Ant Task in Java. Have it create a ClassLoader using a suitable (probably configurable) class path. Use that loader to (attempt to) load the classes matching the input files, in order to inspect their annotations. Requires annotation retention at runtime, and full initialization of all involved classes and their dependencies.
Use javap to inspect the classes. Since I don't know of a programmatic interface to javap (do you?), this probably means iterating over the files and running a new process for each of them, then massaging the created output in a suitable way. Perhaps a <scriptdef>-ed task could be used for this. This would work with class-file annotation retention, and require no initialization.
Use an annotation processor to collect the information at compile-time. This should be able to work with sourcecode-only retention. But I have no experience writing or using annotation compilers, so I'm not sure this will work, and will need a lot of research to figure out some of the details. In particular how to activate the task for use by ant (Java 6 annotation processing configuration with Ant gives some pointers on this, as does What is the default annotation processors discovery process?) and when to create the output file (in each round, or only in the last round).
Which of these do you think has the greatest chances of success? Can you suggest code samples for one of these, which might be close to what I want and which I could adapt appropriately?
Encouraged by Thomas' comment, I gave approach 3 a try and got the following annotation processor working reasonably well:
import java.io.IOException;
import java.io.Writer;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.ProcessingEnvironment;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.QualifiedNameable;
import javax.lang.model.element.TypeElement;
import javax.tools.StandardLocation;
#SupportedSourceVersion(SourceVersion.RELEASE_7)
public class AnnotationServiceProcessor extends AbstractProcessor {
// Map name of the annotation to name of the corresponding service interface
private static Map<String, String> annotationToServiceMap = new HashMap<>();
static {
// Adapt this to your use, or make it configurable somehow
annotationToServiceMap.put("Annotation1", "Service1");
annotationToServiceMap.put("Annotation2", "Service2");
}
#Override public Set<String> getSupportedAnnotationTypes() {
return annotationToServiceMap.keySet();
}
// Map name of the annotation to list of names
// of the classes which carry that annotation
private Map<String, List<String>> classLists;
#Override public void init(ProcessingEnvironment env) {
super.init(env);
classLists = new HashMap<>();
for (String ann: getSupportedAnnotationTypes())
classLists.put(ann, new ArrayList<String>());
}
public boolean process(Set<? extends TypeElement> annotations,
RoundEnvironment env) {
for (TypeElement ann: annotations) {
List<String> classes =
classLists.get(ann.getQualifiedName().toString());
for (Element elt: env.getElementsAnnotatedWith(ann)) {
QualifiedNameable qn = (QualifiedNameable)elt;
classes.add(qn.getQualifiedName().toString());
}
}
if (env.processingOver()) { // Only write results at the end
for (String ann: getSupportedAnnotationTypes()) {
try {
write(ann, classLists.get(ann));
} catch (IOException e) {
throw new RuntimeException(e); // UGLY!
}
}
}
return true;
}
// Write the service file for each annotation we found
private void write(String ann, List<String> classes) throws IOException {
if (classes.isEmpty())
return;
String service = annotationToServiceMap.get(ann);
Writer w = processingEnv.getFiler()
.createResource(StandardLocation.CLASS_OUTPUT,
"", "META-INF/services/" + service)
.openWriter();
classes.sort(null); // Make the processing order irrelevant
for (String cls: classes) {
w.write(cls);
w.write('\n');
}
w.close();
}
}
So far I've hooked this up to ant using <compilerarg>s from https://stackoverflow.com/a/3644624/1468366. I'll try something better and if I succeed will edit this post to include some ant snippet.
I've created a compound document type called SampleCaps in a HippoCMS 7.9 site and set out to build a template for it. In the process, I added hst:sitemap nodes, a pair of nested hst:pages nodes, and an hst:templates node. I also added the appropriate type property to hippo:namespaces/barcom/SampleCaps.
Finally, I created a Component and a Bean to expose the document data to the template, adapting the steps presented in part 2 of the Hippo Video Trails.
To my frustration, while the Component loads properly the Bean is never loaded (or at least, its getter is never called.) My component and bean are as follows:
site/src/main/java/com/footech/barcom/components/SampleCapsComponent.java:
package com.footech.barcom.components;
import com.footech.barcom.beans.SampleCapsDocument;
import org.hippoecm.hst.content.beans.standard.HippoBean;
import org.hippoecm.hst.component.support.bean.BaseHstComponent;
import org.hippoecm.hst.core.component.HstComponentException;
import org.hippoecm.hst.core.component.HstRequest;
import org.hippoecm.hst.core.component.HstResponse;
public class SampleCapsComponent extends BaseHstComponent {
#Override
public void doBeforeRender(final HstRequest request, final HstResponse response) throws HstComponentException {
SampleCapsDocument document = request.getRequestContext().getContentBean();
request.setAttribute("document", document);
System.out.println("Ping"); /* prints "Ping" to console */
}
}
site/src/main/java/com/footech/barcom/beans/SampleCapsDocument.java:
package com.footech.barcom.beans;
import java.util.Calendar;
import org.hippoecm.hst.content.beans.Node;
import org.hippoecm.hst.content.beans.standard.HippoHtml;
import org.onehippo.cms7.essentials.dashboard.annotations.HippoEssentialsGenerated;
#HippoEssentialsGenerated(internalName = "barcom:SampleCapsdocument")
#Node(jcrType = "barcom:SampleCapsdocument")
public class SampleCapsDocument extends BaseDocument {
#HippoEssentialsGenerated(internalName = "barcom:title")
public String getTitle() {
System.out.println("Pong"); /* This never triggers */
return getProperty("barcom:title");
}
}
To my understanding, the annotation #Node(jcrType = "barcom:SampleCapsdocument") in SampleCapsComponent.java should hint to the compiler that the content node should be wrapped with the SampleCapsDocument bean - this does not appear to be the case, as the debug console prints Ping but not Pong. What am I doing wrong?
you'll need to call document.getTitle() because values are lazy loaded.
I am using JPachube.jar and Matlab in order to send data to my datastream. This java code works on my machine:
package smartclassroom;
import Pachube.Data;
import Pachube.Feed;
//import Pachube.FeedFactory;
import Pachube.Pachube;
import Pachube.PachubeException;
public class SendFeed {
public static void main(String arsg[]) throws InterruptedException{
SendFeed s = new SendFeed(0.0);
s.setZainteresovanost(0.3);
double output = s.getZainteresovanost();
System.out.println("zainteresovanost " + output);
try {
Pachube p = new Pachube("MYAPIKEY");
Feed f = p.getFeed(MYFEED);
f.updateDatastream(0, output);
} catch (PachubeException e) {
System.out.println(e.errorMessage);
}
}
private double zainteresovanost;
public SendFeed(double vrednost) {
zainteresovanost = vrednost;
}
public void setZainteresovanost(double vrednost) {
zainteresovanost = vrednost;
}
public double getZainteresovanost() {
return zainteresovanost;
}
}
but I need to do this from Matlab. I have tried rewriting example (example from link is working on my machine): I have compile java class with javac and added JPachube.jar and SendFeed.class into path and then utilize this code in Matlab:
javaaddpath('C:\work')
javaMethod('main','SendFeed','');
pachubeValue = SendFeed(0.42);
I get an error:
??? Error using ==> javaMethod
No class SendFeed can be located on Java class path
Error in ==> post_to_pachube2 at 6
javaMethod('main','SendFeed','');
This is strange because, as I said example from the link is working.
Afterwards, I decided to include JPachube directly in Matlab code and to write equivalent code in Matlab:
javaaddpath('c:\work\JPachube.jar')
import Pachube.Data.*
import Pachube.Feed.*
import Pachube.Pachube.*
import Pachube.PachubeException.*
pachube = Pachube.Pachube('MYAPIKEY');
feed = pachube.getFeed(MYFEED);
feed.updateDatastream(0, 0.54);
And I get this error:
??? No method 'updateDatastream' with matching signature found for class 'Pachube.Feed'.
Error in ==> post_to_pachube2 at 12
feed.updateDatastream(0, 0.54);
So I have tried almost everything and nothing! Any method making this work will be fine for me. Thanks for help in advance!
This done trick for me (answer from here)
javaaddpath('c:\work\httpcore-4.2.2.jar');
javaaddpath('c:\work\httpclient-4.2.3.jar');
import org.apache.http.impl.client.DefaultHttpClient
import org.apache.http.client.methods.HttpPost
import org.apache.http.entity.StringEntity
httpclient = DefaultHttpClient();
httppost = HttpPost('http://api.cosm.com/v2/feeds/FEEDID/datastreams/0.csv?_method=put');
httppost.addHeader('Content-Type','text/plain');
httppost.addHeader('X-ApiKey','APIKEY');
params = StringEntity('0.7');
httppost.setEntity(params);
response = httpclient.execute(httppost);
I would rather use built-in methods. Matlab hasurlread/urlwrite, which could work if all you wish to do is request some CSV data from Cosm API. If you do need to use JSON, it can be handled in Matlab via a plugin.
Passissing the Cosm API key, that can be done via key parameter like so:
cosm_feed_url = "https://api.cosm.com/v2/feeds/61916.csv?key=<API_KEY>"
cosm_feed_csv = urlread(cosm_feed_url)
However, the standard library methods urlread/urlwrite are rather limited. In fact, the urlwrite function is only designed for file input, and I cannot even see any official example of how one could use a formatted string instead. Creating a temporary file would reasonable, unless it's only a few lines of CSV.
You will probably need to use urlread2 for anything more serious.
UPDATE: it appears that urlread2 can be problematic.