I am trying to get an input from the user and pass it to my mapper class that I have created but whenever the value always initialises to zero instead of using the actual value the user input.
How can make sure that whenever I get the variable it always maintain the same value. I have noticed that job1.setMapperClass(Parallel_for.class); creates an instance of the class hence forcing the variable to reinitialize to its original value. Below is the link to the two classes. I am trying to get the value of times from RunnerTool class.
Link to Java TestFor class
Link to RunnerTool class
// setup method in the Mapper
#Override
public void setup(Context context) {
int defaultValue = 1;
times = context.getConfiguration().getInt("parallel_for_iteration", defaultValue );
LOG.info(context.getConfiguration().get("parallel_for_iteration") + " Actually name from the commandline");
LOG.info(times + " Actually number of iteration from the commandline");
}
// RunnerTools class
conf.setInt(ITERATION, times);
You should note that mapper class will be recreated on many cluster nodes so any initalization done to the instance of the mapper class when running the job will not affect other nodes. Technically relevant jar file/s will be distributed among nodes and then mappers will be created there.
So as pointed in the answer above, the only way to pass information to the mappers is using Configuration class.
Mapper get's initialized by reflection, so you can not let the user interact with the mapper class.
Instead you have your Configuration object, which you have to provide if you're setting up your job. There you can set this using conf.set("YOUR KEY", "YOUR VALUE"). In your Mapper class you can override a method called setup(Context context), there you can get the value using context.getConfiguration().get("YOUR KEY"). And maybe save to your mapper local variable.
Related
I am currently trying to migrate an app to firebase from a SQLDatabase. I have done all the work to convert the database to a JSONObject. Is there a way I can just fill a user node in firebase with this object?
I tried to write the JSONObject with
database.getReference("/" + user.getUid() + "/").setValue(jsonObject);
this returns the error
com.google.firebase.database.DatabaseException: No properties to serialize found on class org.json.JSONObject
I'm trying to avoid breaking apart the object and posting individual values seperatly because this could be error prone and I can't risk corrupting data with a messed up loop... Thanks for any help.
You can convert the json into an object using ObjectMapper:
ObjectMapper mapper = new ObjectMapper();
User user = mapper.readValue(jsonObject.toString(), User.class);
https://fasterxml.github.io/jackson-databind/javadoc/2.7/com/fasterxml/jackson/databind/ObjectMapper.html
Then you can use the instance of the class User in setValue():
database.getReference("/" + user.getUid() + "/").setValue(user);
From the docs:
In addition, you can set instances of your own class into this location, provided they satisfy the following constraints:
The class must have a default constructor that takes no arguments
The class must define public getters for the properties to be assigned. Properties without a public getter will be set to their default value when an instance is deserialized
I'm writing an application that has an instance of a class that contains the esper engine. There are a number of instance variables that I would like to read and set using instance method calls from the EPL in the engine. I don't get any compilation errors and the code runs. But the instance method is not called.
epl statements:
module myModule;
create variable com.tp.main.MyClass myClass;
select myProperty from MyEvent unidirectional, method:myClass.getMyProperty() as myProperty;
A hint could be that if I don't use the method: key word in front of the method call I get an error that myClass.getMyProperty class could not be found. The documentation sometimes uses the method: key word and sometimes not in the examples for calling instance methods from Class-type variables.
I have also tried using the addVariable method in the API with the same results.
code for the method.
public Result getMyProperty() {
Result result = new Result();
result.setResult("propertyValue");
logger.info("This method was called");
return result;
}
The class Result is a POJO with getter and setter for a string.
public class Result {
private String result;
public String getResult() {
return result;
}
public void setResult(String str) {
result = str;
}
}
What am I missing?
You could look at a regression test class. The specific one you may want to look at is ExecFromClauseMethodVariable. Maybe your code does not assign a value to the variable?
Github:
https://github.com/espertechinc/esper/blob/3e396d77308532b202ee452100eaaf9e7a044906/esper-regression/src/test/java/com/espertech/esper/regression/epl/fromclausemethod/ExecFromClauseMethodVariable.java
Problem solved and I thought it might be useful to share the solution. Credit to user650839 who pointed me in the right direction. Here is what ended up fixing the problem.
I reverted back to declaring the variable in the runtime configuration API. I found that I must register the variable class, initialize it with the instance object (this) and finally import the class. Here is the snippet of code that does this configuration in the runtime configuration API.
Configuration configuration = new Configuration();
configuration.addVariable("myClass", com.tp.main.MyClass.class, this);
configuration.addImport(com.tp.main.MyClass.class);
epService = EPServiceProviderManager.getProvider(trade.getTradeName(), configuration);
It seems there is a limitation when declaring the Class variable in the EPL. You cannot initialize it with the instance object you want to use. In the runtime configuration API I was able to initialize it with "this" instance of the object which contains all of the instance variables I want to access from the EPL.
The EPL statement did not change. However, it does seem that you must use the key word method: in front of the method call or you get an error "cannot find class..."
In my Spring boot application, a specific value needs to be read from the properties file depending on a string value returned from another function. My properties file is as follows:
A=value_a
B=value_b
A function returns either A or B, and stores it in a String variable called stringValue . I am looking for doing something along the lines of the following:
#Value(stringValue)
String propertyValue
However, I get the following message on my IDE:
Attribute value must be constant
I have tried to convert stringValue to a static final variable, but to no avail.
I do know that I can pass a specific key to be read from the properties file, such as the following:
#Value("${A}")
String valueOfA
My question is whether I can pass a variable to the #Value annotation?
#Value annotations are resolved during startup, when Spring context is built. Since stringValue would not be available at this time, you can't use it for injection purposes.
An exception to this scenario would be if the bean with #Value annotation is prototype-scoped, in which case a new instance of it would be created any time it's requested. Still, stringValue would need to be available to the Spring context in order to be used at injection point.
Without seeing more code, it's not possible to give you any more detailed answer.
You can autowire Environment in your application and use it to read from the properties file as it is supposed to be at runtime.( This is assuming all files where your properties are have been added to environment).
I will be posting my answer with assumptions as you have not updated your question, with all information I requested.
So your code would be-
lets call your class where your making a call to a function called getValue to get value of stringValue- Example. Now lets assume you are making a call to the function getValue in a method in the class Example called doSomething().
class Example{
#Autowire
private Enviornment environment.
private String propertyValue
public void doSomething(){
String value=getValue()// this is where u know whether
its A or B.
propertyValue=environment.getProperty(value);
// do whatever u want know
}
Thanks for the help guys! I read the values from the properties file into a org.springframework.core.io.Resource object, and then used that to retrieve the specific value that I required. The following was how I structured my solution code:
#Value("classpath:config.properties")
private Resource propertiesfile;
I then declared a java.util.Properties object and read the values from the Resource object into it.
Properties properties = new Properties();
try{
properties.load(propertiesfile.getInputStream());
}catch(IOException e){
logger.error("Parsing error while reading properties file",e.toString());
}
Finally, based on the value in my stringValue variable, I read the corresponding value from the properties file
String propertyValue = properties.getProperty(stringValue);
if your variable is another environment variable you can try in this format.
#Value("${${stringvalue}}")
where stringvalue is the environment variable.
i'm writing a cucumber test and i come up with some difficulty:
I have a step which creates dto and saves it using save client which returns dto back again i would need to use that returned dto for other step but don't know how to make it.
Here's how it looks in code :
commonExpenseCreationSteps.java
#Given("^new \"([^\"]*)\" expense with type \"([^"]*)\"$")
public ExpenseDTO newExpense(String description, String expenseType) throws Throwable {
ExpenseDTO expenseDTO = new ExpenseDTO();
expenseDTO.setDefaultPurpose(description);
expenseDTO.setExpenseType(expenseType);
return expenseSaveClient.save(expenseDTO);
}
expenseTransactionsSendSteps.java
#Given("^send expense for Approval$")
public void sendExpenseForApproval() throws InterruptedException {
expenseTransactionSendClient.sendToApproval(expenseDTO);
}
How it would be possible to store value returned by one Step and use it in other one in this case i return ExpenseDTO in newExpense method but i need to use it in sendExpenseForApproval but don't know how to do it !?
Create expenseDTO object outside of your glue code, probably in your stepdef class constructor.
ExpenseDTO expenseDTO = new ExpenseDTO();
The way to share state between steps in the same class is to use instance variables. Set the value in one step and use that value in a later step.
The way to share state between steps with two or more step classes is to use dependency injection.
I wrote a blog post that describes how it can be done using PicoContainer.
I want/need to pass along the rowkey to the Reducer, as the rowkey is calculated in advance, and the information is not available anymore at that stage. (The Reducer executes a Put)
First I tried to just use inner classes, e.g.
public class MRMine {
private byte[] rowkey;
public void start(Configuration c, Date d) {
// calc rowkey based on date
TableMapReduceUtil.initTableMapperJob(...);
TableMapReduceUtil.initTableReducerJob(...);
}
public class MyMapper extends TableMapper<Text, IntWritable> {...}
public class MyReducer extends TableReducer<Text, IntWritable, ImmutableBytesWritable> {...}
}
and both MyMapper and MyReducer have the default constructor defined. But this approach leads to the following exception(s):
java.lang.RuntimeException: java.lang.NoSuchMethodException: com.mycompany.MRMine$MyMapper.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
Caused by: java.lang.NoSuchMethodException: com.company.MRMine$MyMapper.<init>()
at java.lang.Class.getConstructor0(Class.java:2730)
at java.lang.Class.getDeclaredConstructor(Class.java:2004)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
I got rid of the exception by declaring the inner classes static (Runtimeexception: java.lang.NoSuchMethodException: tfidf$Reduce.<init>()) . but then I'd have to make the rowkey static as well, and I'm running multiple jobs in parallel.
I found https://stackoverflow.com/a/6739905/1338732 where the configure method of the Reducer is overwritten, but it doesn't seem to be available anymore. Anyhow, I wouldn't be able to pass along a value.
I was thinking of (mis)using (?) the Configuration, by just adding a new key-value pair, would this be working, and the correct approach?
Is there a way to pass along any custom value to the reducer?
the versions I'm using are: hbase: 0.94.6.1, hadoop: 1.0.4
Your problem statement is a little unclear, however I think something like this is what you are looking for.
The way I currently use to pass information to the reducer is to pass it in the configuration.
in the job setup do the following
conf.set("someName","someValue");
This will create a tag in the configuration that has name someName with value someValue. This can later be retrieved in the Mapper/Reducer by doing the following:
Configuration conf = context.getConfiguration();
String someVariable = conf.get("someName");
The current code will set the value of someVariable to "someValue", allowing the information to be passed to the reducer.
To pass multiple values use setStrings(). I haven't tested this function yet, but according to the documentation is should work with one of the following two options (the documentation is a little unclear, so try both and use whichever works):
conf.setStrings("someName","value1,value2,value3");
conf.setStrings("someName","value1","value2","value3");
retrieve using:
Configuration conf = context.getConfiguration();
String someVariable = conf.getStrings("someName");
Hope this helps
The goal is a little unclear, but I have found that for many types of jobs involving HBase, you do not need a reducer to put data into HBase. The mapper reads a row, modifies it in some way, then writes it back.
Obviously there are jobs for which that is inappropriate (any type of aggregation for example), but the reduce stage can really slow down a job.