I'm running in to a problem with KSQL while trying to set up an ETL pipeline using a UDF. At some point in the ETL process I need to isolate specific info from a description field (VARCHAR) in my data. A made-up example for context:
description = "species=dog.sex=male.color=blonde.age=10." (the real data is formatted in the same way)
I've written a simple UDF to isolate any information on demand. It looks like this:
package com.my.package;
/** IMPORTS **/
import io.confluent.ksql.function.udf.Udf;
import io.confluent.ksql.function.udf.UdfDescription;
/** ClASS DEFINITION **/
#UdfDescription(name = "extract_from_description",
author = "Me",
version = "0.0.1",
description = "Given a description and a request for information, isolates and returns the requested information. Pass requested tag as 'tag='".)
public class Extract_From_Description {
#Udf(description = "Given a description and a request for information, isolates and returns the requested information. Pass requested tag as 'tag='.)
public String extract_from_description(final String description, final String request) {
return description.split(request)[1].split("\\.")[0];
}
}
I can upload and register the function just fine, it's listed and described properly when I run:
ksql> list functions;
ksql> describe function EXTRACT_FROM_DESCRIPTION;
I call the function like this to create a new stream:
CREATE STREAM result AS
SELECT recordId,
OtherVariables,
EXTRACT_FROM_DESCRIPTION(description, 'species=') AS species
FROM parent_stream
EMIT CHANGES;
There I get an error I can't make sense of:
Function 'extract_from_description' does not accept parameters (STRING, STRING).
Valid alternatives are:
Apparently KSQL can't properly interpret what the input for the function is supposed to be (looks like it expects no input?) and I can't figure out why. I've read through documentation to see if I define my function in a weird way but can't find any differences between the examples and my function. I did notice there are supposed to be several ways to define the input a function takes and tried them all, but the result is always the same.
I use Maven to create the jar file for this function (JDK1.8.0_201). Can anyone help me figure out what's going on?
TL;DR: My KSQL UDF doesn't accept input of type (String, String) even though the function specifies the input should be of type (String, String)
Found the problem, answering here for anyone that might run in to the same problem.
You need to specify the parameters using #UdfParameter, like this:
import io.confluent.ksql.function.udf.UdfParameter; // add this to the list of imports
// add #UdfParameter(name) to each input variable
public String extract_from_description(#UdfParameter(value = "description") final String description, #UdfParameter(value = "request") final String request){
function body
}
Related
Following i my code in JCO3.0 to connect to RFC and get the data from function module:
try {
JCoDestination destination = JCoDestinationManager.getDestination(DESTINATION_NAME);
JCoFunction function = destination.getRepository().getFunction("funtion_abap");
***function.getImportParameterList().setValue("IM_ID_NAME", "MTC_ZPR008_TEMPB");***
function.execute(destination);
JCoTable table = function.getTableParameterList().getTable("export_table");
}
catch(Exception e){
}
Following is my ABAP function:
CALL FUNCTION 'funtion_abap' DESTINATION m_vsyid
EXPORTING
IM_ID_NAME = table_vname
IMPORTING
export_table = table_tvarvc
EXCEPTIONS
system_failure = 1
communication_failure = 2
resource_failure = 3
OTHERS = 4.
following is an error m getting while passing String as import parameter while it wants Table field as import parameter:
Exception in thread "main" com.sap.conn.jco.ConversionException: (122) JCO_ERROR_CONVERSION: Cannot convert a value of 'MTC_ZPR008_TEMPB' from type java.lang.String to TABLE at field IM_ID_NAME
at com.sap.conn.jco.rt.AbstractRecord.createConversionException(AbstractRecord.java:468)
at com.sap.conn.jco.rt.AbstractRecord.createConversionException(AbstractRecord.java:462)
at com.sap.conn.jco.rt.AbstractRecord.setValue(AbstractRecord.java:2958)
at com.sap.conn.jco.rt.AbstractRecord.setValue(AbstractRecord.java:4074)
at com.amgen.rfc.RFC_Connection.main(RFC_Connection.java:47)
Please tell me how to solve this problem.
The RFC definition and your code are in direct opposition. According to the ABAP function (as far as I read it) the result of the call is the value in field IM_ID_NAME and the table is the input parameter.
I'm not 100% familiar with the declaration of RFCs in ABAP (I only know the Java side of it), but if I interpret the error message correctly, the table seems to be in the input parameter list rather than the table parameter list (not usual but not never seen before, either). So instead of getTableParameterList you will possible have to call getInputParameterList. Also you should omit the setting of the field IM_ID_NAME because that's the response value and resides in the output parameter list.
I know the question is quite old but someone may find my response useful one day since I had the same problem:
JcoTable tab = function.getImportParameterList().getTable("IM_ID_NAME");
tab.appendRow();
tab.firstRow(); // I'm not sure if this is actually reqiured
tab.setValue("PARAM_NAME", paramValue);
I'm currently working on an application built in Scala with Spray routing.
So for dealing with a JSON document sent over POST, it's pretty easy to access the variables within the body, as follows;
respondWithMediaType(`application/json`) {
entity(as[String]) { body =>
val msg = (parse(body) \ "msg").extract[String]
val url = (parse(body) \ "url").extractOpt[String]
However, I'm now trying to write an additional query with GET, and am having some issues accessing the parameters sent through with the query.
So, I'm opening with;
get {
respondWithMediaType(`application/json`) {
parameterSeq { params =>
var paramsList = params.toList
So, this works well enough in that I can access the GET params in a sequential order (just by accessing the index) - the problem is, unfortunately I don't think we can expect GET params to always be sent in the correct order.
The list itself prints out in the following format;
List((msg,this is a link to google), (url,http://google.com), (userid,13))
Is there any simple way to access these params? For example, something along the lines of;
var message = paramsList['msg']
println(message) //returns "this is a link to google"
Or am I going about this completely wrong?
Apologies if this is a stupid question - I've only switched over to Scala very recently, and am still getting both acquainted with that, and re-acquainted with Java.
What I usually do is use the parameters directive to parse the data out to a case class which contains all the relevant data:
case class MyParams(msg: String, url: String, userId: Int)
parameters(
"msg".as[String],
"url".as[String],
"userId".as[Int]
).as[MyParams] {
myParams =>
// Here you have the case class containing all the data, already parsed.
}
To build your routes you could use the parameters directives. I'm not sure if this is what you're looking for, anyway you could use them as:
get {
parameters('msg) { (msg) =>
complete(s"The message is '$msg'")
}
}
Spray directives can be easily composed so you can use combine them in any way you want.
I hope that helps you.
I just started learning about groovy and trying to transpose my java code to groovy scripts. Usually java allows you have a class with only methods that you can call from other classes. I wanted to translate that to groovy. I have in one file - lets call it File1- a method like this:
def retrieveData(String name){
// do something
}
and in the second file, File2, I call File1 like this:
def file1Class = this.class.classLoader.parseClass(new File("../File1.groovy"))
and then try to call the method in File1 like this:
def data = file1Class.retrieveData("String")
but it keeps giving me this error - MissingMethodException:
groovy.lang.MissingMethodException: No signature of method: static File1.retrieveData() is applicable for argument types: (java.lang.String) values: [String] Possible solutions: retrieveData(java.lang.String)
so it does recognize that I am sending in the correct number of parameters and even the correct object, but it isn't running the method as it should?
Is there something I am missing? I tried to remove the object definition from the method - in other words - like this:
def retrieveData(name){
// do something
}
but that didn't work either. I am clueless about what the next step would be. Can anyone please help push me in the right direction? I would greatly appreciate it.
See the answer provided in this StackOverflow reponse.
Use the GroovyScriptEngine class. What does the GroovyScriptEngine do? From the docs:
Specific script engine able to reload modified scripts as well as
dealing properly with dependent scripts.
See the example below.
def script = new GroovyScriptEngine( '.' ).with {
loadScriptByName( '..\File1.groovy' )
}
this.metaClass.mixin script
retrieveData()
Note how we use the loadScriptByNamemethod to
Get the class of the scriptName in question, so that you can
instantiate Groovy objects with caching and reloading.
This will allow you to access Groovy objects from files however you please.
I'm trying to redirect to a controller method which is taking Long and String as argument by using the reverse controller. I use version 2.4 of the play framework.
I've defined this route in the routes file:
GET /games/play/:id controllers.Games.renderGame(id: Long, feedback: String = "")
To call this route, I'm using in a other method redirect():
return redirect(controllers.routes.Games.renderGame(gameId, "test"));
And here is my renderGame() method:
public Result renderGame(Long id, String feedback) {
//do something
return ok(...);
}
In my opinion this actually should work but play gives me a error:
error: method renderGame in class ReverseGames cannot be applied to
given types;
IntelliJ is trying to do it better:
Error picture
If I define the method just with Long as parameter it's working fine but when a add the String I get the error again.
Any idea what's wrong here?
Related to this Question, it actually should work: Play Framework: Redirect to controller method with arguments
I fixed the problem by myself but it tooks me hours. The problem was that I used a = instead of =?. It has to look like this:
GET /games/play/:id controllers.Games.renderGame(id: Long, feedback: String ?= "")
I think the problem is with the route definition try adding feedback params at the url pattern
GET /games/play/:id/:feedback controllers.Games.renderGame(id: Long, feedback: String = "")
cheers.
The situation seems to be abnormal, but I was asked to build serializer that will parse an object into string by concatenating results of "get" methods. The values should appear in the same order as their "get" equivalent is declared in source code file.
So, for example, we have
Class testBean1{
public String getValue1(){
return "value1";
}
public String getValue2(){
return "value2";
}
}
The result should be:
"value1 - value2"
An not
"value2 - value1"
It can't be done with Class object according to the documentation. But I wonder if I can find this information in "*.class" file or is it lost? If such data exists, maybe, someone knows a ready to use tool for that purpose? If such information can't be found, please, suggest the most professional way of achieving the goal. I thought about adding some kind of custom annotations to the getters of the class that should be serialized.
If you want that you have to parse the source code, not the byte code.
There are a number of libraries that parse a source file into a node tree, my favorite is the javaparser (hosted at code.google.com), which, in a slightly modified version, is also used by spring roo.
On the usage page you can find some samples. Basically you will want to use a Visitor that listens for MethodDefinitions.
Although reflection does not anymore (as of java 7 I think) give you the methods in the order in which they appear in the source code, the class file appears to still (as of Java 8) contain the methods in the order in which they appear in the source code.
So, you can parse the class file looking for method names and then sort the methods based on the file offset in which each method was found.
If you want to do it in a less hacky way you can use Javassist, which will give you the line number of each declared method, so you can sort methods by line number.
I don't think the information is retained.
JAXB, for example, has #XmlType(propOrder="field1, field2") where you define the order of the fields when they are serialized to xml. You can implemenet something similar
Edit: This works only on concrete classes (the class to inspect has its own .class file). I changed the code below to reflect this. Until diving deeper into the ClassFileAnalyzer library to work with classes directly instead of reading them from a temporary file this limitation exists.
Following approach works for me:
Download and import following libarary ClassFileAnalyzer
Add the following two static methods (Attention! getClussDump() needs a little modification for writing out the class file to a temporary file: I removed my code here because it's very special at this point):
public static String getClassDump(Class<?> c) throws Exception {
String classFileName = c.getSimpleName() + ".class";
URL resource = c.getResource(classFileName);
if (resource == null) {
throw new RuntimeException("Works only for concreate classes!");
}
String absolutePath = ...; // write to temp file and get absolute path
ClassFile classFile = new ClassFile(absolutePath);
classFile.parse();
Info infos = new Info(classFile, absolutePath);
StringBuffer infoBuffer = infos.getInfos();
return infoBuffer.toString();
}
public static <S extends List<Method>> S sortMethodsBySourceOrder(Class<?> c, S methods) throws Exception {
String classDump = getClassDump(c);
int index = classDump.indexOf("constant_pool_count:");
final String dump = classDump.substring(index);
Collections.sort(methods, new Comparator<Method>() {
public int compare(Method o1, Method o2) {
Integer i1 = Integer.valueOf(dump.indexOf(" " + o1.getName() + lineSeparator));
Integer i2 = Integer.valueOf(dump.indexOf(" " + o2.getName() + lineSeparator));
return i1.compareTo(i2);
}});
return methods;
}
Now you can call the sortMethodsBySourceOrder with any List of methods (because sorting arrays is not very comfortable) and you will get the list back sorted.
It works by looking at the class dumps constant pool which in turn can be determined by the library.
Greetz,
GHad
Write your custom annotation to store ordering data, then use Method.getAnnotation(Class annotationClass)