I have several Spark big data applications written in Scala. These applications have their other version written in R.
I also have a web server application written in Java. This is provided as API for the web GUI. The purpose is to enable GUI to execute these applications and choose the version: R or Spark. I managed to call the R code from the Java API and get the result to JSON. But now it seems to be quite complicated to execute the Spark programs.
Until now, I was able to merge one of the Scala .jar file with the Java API with Maven. I do this by placing my Spark program as a local repository in pom.xml so that the Scala code is included in the final .jar package. I also mentioned Scala and breeze library as dependencies in the pom.xml. And when I try to send a request with the API, of course it throws an error saying java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession$. By this point, I realize that it was because I haven't mentioned Spark library in Maven dependencies, but then I think I've been doing it wrong, since Spark applications are generally run by executing spark-submit command in terminal.
So now what I'm thinking is putting the Java API .jar and Scala .jar in one folder, and then executing spark-submit from inside of Java API .jar, targeting the Scala .jar. Is this even correct? And how to execute the spark-submit from Java code? Does it have to be using Runtime.exec() as mentioned in here?
SparkLauncher can be used to submit the spark code(written in scala with precomplied jar scala.jar placed at certain location) from the Java Api code.
The saprk documentaion for using SparkLauncher recommends the below way to submit the spark job pro-grammatically from inside Java Applications. Add the below code in your Java Api code.
import org.apache.spark.launcher.SparkAppHandle;
import org.apache.spark.launcher.SparkLauncher;
public class MyLauncher {
public static void main(String[] args) throws Exception {
SparkAppHandle handle = new SparkLauncher()
.setAppResource("/my/scala.jar")
.setMainClass("my.spark.app.Main")
.setMaster("local")
.setConf(SparkLauncher.DRIVER_MEMORY, "2g")
.startApplication();
// Use handle API to monitor / control application.
}
}
Related
I am working on a very simple Snowflake example to call a pre-compiled Java UDF from Intellij. I have made a jar out of my Java UDF and want to call this from Intellij where i am trying to execute the UDF on snowflake.
I want to know how I can upload this jar into snowflake ?
session.udf.registerTemporary("sampleJUdf",
"C:\\project_existing\\snowpark\\src\\main\\Java\\myjarfile.jar")
You can upload the jar file using Session.addDependency() for example
session.addDependency("C:\\project_existing\\snowpark\\src\\main\\Java\\myjarfile.jar")
Then you need to refer the name of the function you have in your myjarfile.jar that is going to be used by the UDF, for example
session.udf.registerTemporary("sampleJUdf", myFunc)
If the jar is part of your running application SNowpark should automatically try to upload it, but you need to import it in your code and refer the name of the function in registerTemporary.
See https://docs.snowflake.com/en/developer-guide/snowpark/creating-udfs.html#specifying-dependencies-for-a-udf for more details.
I want to try out java in aws, specifically lambda.
I don't know much java so hoping to learn it in this endeavor. Please be gentle to a newbie
Trying to get hello world working so I can start iterating but I don't know the syntax.
My java program:
$ cat helloWorld.java
class helloWorld
{
public static void main(String args[])
{
System.out.println("Hello, World");
}
}
I zip it up with zip java above_program.java
The call i make in the lambda is to helloWorld::main
The result I currently get when I test it is
START RequestId: 32ef4680-b741-402d-9af7-7c0b0c9e5f1f Version: $LATEST
Class not found: helloWorld: java.lang.ClassNotFoundException
java.lang.ClassNotFoundException: helloWorld
Lambda will not compile and run your java code by only uploading the file to lambda, for compilation language you have to upload complete package where for interpreter language like python you just need upload the code and also you will able to edit code in the console editor for python or nodejs etc but the same feature is applicable for complication language, you have to upload the deployment package.
AWS Lambda Deployment Package in Java
Your deployment package can be a .zip file or a standalone jar; it is
your choice. You can use any build and packaging tool you are familiar
with to create a deployment package.
We provide examples of using Maven to create standalone jars and using Gradle to create a .zip file. For more information, see the following topics:
Topics
Creating a .jar Deployment Package Using Maven without any IDE (Java)
Creating a .jar Deployment Package Using Maven and Eclipse IDE (Java)
Creating a ZIP Deployment Package for a Java Function
Authoring Lambda Functions Using Eclipse IDE and AWS SDK Plugin (Java)
You can read more details here
I have some code in an Android project that parses HTML using Jsoup. It doesn't use anything Android specific, they're just static methods that take an InputStream, and return my model classes. The app uses Gradle to build itself in Android Studio.
Is there any way I can create a standard Java main method to do something like load HTML from a local file, run it through my parser, and output a JSON file (using Gson on my model class)? I'm thinking maybe I can add a new sourceSet to Gradle like a jvmCompatible set of classes? I would greatly prefer not to copy my code to a separate project.
EDIT:
I guess I didn't make this clear, but I would like the be able to run this locally on my dev machine from the command line, rather than on an Android device or emulator.
You don't necessarily need to do anything in the build file to set this up; the build file generates Java .class files, and you can feed them to Java directly from the command line. You can add a main method to any class:
package com.example.foo;
class MyClass {
...
public static void main(String [] args) {
...
}
}
The main method will be happily ignored util you invoke it via the Java command line. You can do this by setting your classpath to the intermediate build directory and telling the Java command line which class to start:
java -classpath app/build/intermediates/classes/debug/ com.example.foo.MyClass
where you pass in the path to the build/intermediates/classes/debug directory in your app module's build output, and the fully-qualified name of the class.
Note that if you're running a release build that uses ProGuard, this main method could get stripped out if it's not otherwise referenced in the code.
Make sure you don't access any Android classes or you'll get a runtime error.
As an aside, you might find it worthwhile to separate out your Java-only code into a Java-only module in the build. Among other things, it would let you use JUnit to write nice test cases for the classes within; if you're asking this question because you want to do some testing of your parser, you might find it convenient to do so within the auspices of a unit test.
This post is related to this one : How to put properly a libgdx application inside swing application? but the question is different.
In this post I explained that I have two JARs : WorldEditor.jar and GameEngine.jar, and I load GameEngine.jar at runtime from WorldEditor.jar.
My problem is with one of the libraries included in GameEngine.jar, namely JRuby.
When I run java -jar GameEngine.jar everything is fine, but when I launche java -jar worldEditor.jar, the instance of JRuby ScriptManager I use returns null when I call ``getEngineByName`. I just can't point ou what is the problem.
By tracing the list of ScriptManagerFactories, I saw that in the good case I have [JRuby, Rhino], and in the bad one I have only [Rhino].
Would someone have an idea of what's going on ?
I don't have much experience with Java's ScriptEngine, but I ran into this issue while answering another question here. I think your problem boils down to classpath order issues.
Using this code:
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
public class Script {
public static void main(String args[]) throws Exception {
ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("jruby");
System.out.println(engine);
}
}
If I run it two different ways, I get two different results:
$ java -cp .:jruby.jar Script
null
$ java -cp jruby.jar:. Script
org.jruby.embed.jsr223.JRubyEngine#30c01f1c
Looking into it a bit, there is a special file in the jar that registers the various scripting containers:
The ScriptEngineManager uses the service provider mechanism described
in the Jar File Specification to obtain instances of all
ScriptEngineFactories available in the current ClassLoader.
My guess is that the JVM doesn't need to load the JRuby jar in the first case, so it hasn't registered the scripting engine. It would only load that jar when it cannot find some class. This means you might be able to force it to work by using some JRuby object before you ever call into the script.
I'm working on a Java/JRuby project which needs to be able to be able to interact with GAMS. I know we can use the Java API, but I would really like to be able to access it using JRuby if possible, since we're hoping to eventual add a DSL and some other complexity I'm not really excited about having to implement in pure Java.
Following the official Java API documentation for GAMS, I have downloaded and setup everything necessary to run GAMS from the command line, but I can't figure out how to include the GAMS directory in LD_LIBRARY_PATH and still run JRuby irb. When I run
export LD_LIBRARY_PATH=/home/wikk/Downloads/gams24.0_linux_x64_64_sfx
Then try to run irb with JRuby, I get
jruby: /home/wikk/Downloads/gams24.0_linux_x64_64_sfx/libstdc++.so.6: version 'GLIBCXX_3.4.15' not found (required by jruby)
I think this is what the documentation is asking me to do to run a Java program that calls the API, is there maybe some way to set LD_LIBRARY_PATH in irb, but before importing all the Java class files? I can do this successfully if I don't set LD_LIBRARY_PATH, but then GAMS tells me it can't find the main program when I try to create a new GAMSWorkspace object:
irb(main):002:0> ws = GAMSWorkspace.new
Java::ComGamsApi::GAMSException: could not find a GAMS system directory from
your environment variable, please set up properly before running a program!
from com.gams.api.GAMSWorkspace.verifySystemDirectory(GAMSWorkspace.java:335)
Am I doing this wrong? or does the API require some Java feature that isn't implemented in JRuby?
Finally came back to this problem, got it working through some trial and error. I also needed to run jruby with the -J-Djava.library.path=[GAMSDIR]/apifiles/Java/api flag, and add [GAMSDIR]/apifiles/Java/api/GAMSJavaAPI.jar to the classpath.
Once this is all in place, you can run gams models from ruby scripts:
import com.gams.api.GAMSWorkspace
import com.gams.api.GAMSJob
import com.gams.api.GAMSVariable
import com.gams.api.GAMSVariableRecord
import com.gams.api.GAMSWorkspace
ws = GAMSWorkspace.new
j1 = ws.addJobFromGamsLib('trnsport')
j1.run
j1.out_db.get_variable('x').each_entry do |rec|
puts "x(#{rec.get_keys[0]}, #{rec.get_keys[1]}): level = #{rec.get_level}, marginal = #{rec.get_marginal}"
end
I am writing here because it is the only thread related to the GAMS Java API problem.
In Eclipse, you have to go to "Run Configurations" and add two things:
1. (As already said) add a "-Djava.library.path=[GAMSDIR]\apifiles\Java\api\" to VM arguments
2. Go to Environment and SET explicitly a PATH variable to [GAMSDIR]. For some reason seeting path through windows is not working