I have a Spark Application which I initially created using maven. I converted my maven project into an Eclipse project, and I am now working on it via Eclipse.
I also have my own version of the Spark source code to which I have made some modifications and added additional methods. I am able to build my version of Spark and obtain the jar file for spark-core.
In my Eclipse project I now replace the spark-core that is in the build path with the newly built spark-core jar. I am able to utilize the new methods I created and verify this by running the application through Eclipse.
Now, I am trying to submit my application through the command line. I use the spark-submit command from the Spark project I have modified. When I submit my jar, I get an error saying the method I created does not exist. I'm not sure what I'm doing wrong. I would appreciate some insight on what I should be doing.
UPDATE
I now understand that there are certain jars that I need to have from my custom build. Exactly which jars are these? I understand the spark-assembly jar is one. Are there any others?
Related
I am trying to use a number of external libraries in a Java project. The project runs fine from Intelij but I want to package it in a jar, (or something else) so I can distribute it to others. When I package it in a jar it works if I just do hello world, but as soon as I start using my libraries I get the error bellow. I have also tried packaging it as an application but when I run the batch file it just opens and then immediatly closes a command window. I read all the other posts and nothing is fixing my problem. Please help
My error
My build.gradle
The problem is most likely that your classpath does not point to the correct relative location of your 3rd party libraries. You can check the manifest file to verify if the paths are correct.
However, if it is a runnable jar file with a main method (which it looks like it is), you should use the Application plugin and package it with the Distribution plugin. Right now you are using the Java Library Distribution plugin, which is for libraries. If you do this, you can remove most of the stuff under your jar task.
When testing it locally, run gradle run and when ready, use gradle distZip to create a zip of it all. It will create a script used to start the application with the correct classpath.
Alternatively, you could also package it in a fat jar using the Shadow plugin or similar.
I have a Maven-Springboot project setup. After doing a mvn install I can run the jar file in command prompt using java -jar <my-jar-file.jar>
There was a dependency to a jar say as-common-1.0.0.jar that is there in my-jar-file.jar. Now I want to override the version of this dependent jar at run time by giving an external jar. something like:
java -jar my-jar-file.jar using as-common-1.0.1.jar
I went through many SO posts like thisinclude external jar when running java -jar but they didn't help.
is it achievable?
EDIT: The issue we have is many of our applications depend upon one of our internal framework jar which gets updated(version) often.
So every time changing the pom file and re-deploying all the apps doesn't look feasible to us. We want somehow the dependency of this particular jar is given at run time. Any idea regarding best possible way to manage this scenario?
No, this is not sensible. At best, you would load classes twice and this may/may not work.
If you build a complete jar containing everything, you cannot swap content at runtime. You need to rebuild it.
I'm facing an issue. On testing a java code with external jar dependency separately, it is working. But when trying to deploy OSB code with java callout calling the same java code, it is throwing method does not exist.
There are multiple ways to resolve this issue.
you can create a fat jar(it will contain all class files, Tools are available online or can use Eclipse or any ide to export the code.) for your Java code along with Dependencies and libraries.
You can deploy your dependencies as libraries/Application or copy Weblogic lib folder it depends on your dependency type. after deployment, you must restart the domain. Then u can deploy import your java callout Jar/Archive in OSB.
So I've been pigeon-holed into writing some Jython code. I've been using the latest version of Eclipse IDE with the PyDev plugin for development. Up until now, things have been moderately tolerable. I've gotten all my Python scripts working and I'm successfully including a couple of JAR files and the class directory of another Java project as external dependencies. Everything seems to run fine through the Eclipse IDE.
Now I need to package everything up and deploy it. From what I can gather, the best way to do this would be to package everything up in a JAR file. The Jython documentation suggests starting out with the jython.jar file and adding to it. OK. So I modify my main python module and start adding all my python source to the JAR.
It executes but of course can't find all the external dependencies.
How is one supposed to add the external JAR files so that they are correctly seen by the Jython interpreter? How is one supposed to manage more complex dependencies in a setup like this?
Is there a plugin for Eclipse or maybe something like Ant or Maven that can handle all of these steps for me with the push of a button?
I can't be the first person that has needed to deploy Jython code with complex dependencies can I?
I've made some headway on getting this all working so I thought I would put some notes here in case they help anyone else out. I'd still like to hear from others on their experiences trying to put together something like this.
It turns out that Eclipse as of 3.5 has a project export option for Java -> Runnable JAR File. If you use this option, you can point to a Java main class in the export wizard. You also have the option to have it repackage all the JARs that you are dependent on in your new JAR file. Make sure to check the box to save the export as an ANT build so that you can repeat the process quickly. NOTE that the first time you do this through the interface, it may fail, but it will still have created a JAR file.
Now here's where it gets strange. To track all the dependencies, I am still using a mostly incomplete Maven build in my project. I create the Maven .POM file. And I told Maven what my external JAR dependency was. I then told Maven to do a dependency update for me. It pulled everything into my Maven repository as expected.
Now when I do my ANT build, it appears that it is getting its list of JARs to include in the final build from Maven. I'm not really sure if it is supposed to work that way. I'm also not 100% sure that it is working. I guess I'll find out when I have to add another external JAR to the project.
Anyways, if you follow this question you'll see that you can take the latest builds of Jython and pull the org.python.util.JarRunner.java file out and use it in your own project. This is you Java.main class that you will need to point your ANT build at. From there, convert your main Python/Jython script to be the run script that was talked about in that question.
Next, build up another copy of the Jython JAR file in your Jython directory. This one should have the /Lib directory pulled into the JAR. Save that off and then point your Eclipse IDE Jave Build option for your PyDev project at that JAR as an external dependency. Your JarRunner will now work and execute the run.py file under Jython.
If all that works, you should then be able to rerun the ANT exported build from earlier and you will end up with a single JAR file that you can execute on the command line as:
java -jar {yourjar} args
And distribute to your customers without any additional dependencies.
If that all seems a little bit convoluted, it is. If anyone has a better way of setting this all up using Eclipse, please let me know.
Make your life easier and just use Maven and the mavenjython compile plugin.
See my answer of this question: Using Jython with Maven
You get full automation of the build and deploy process, and the result is a jar that includes jython and all other dependencies.
What is the most proper way to accomplish all of the following:
Create a project in Eclipse
With an Apache Ant buildfile
That Hudson (or another more recommended CI system?) uses
And support for JUnit tests that are run by both Ant/Hudson and Eclipse
And check the proper folder structure into SVN so that future developers can replicate the build, and Hudson can automatically grab from SVN and run the Ant file to build/test/deploy the project
And the project has native libraries, if that makes any difference (I've already written an Ant task that can download and unzip the proper library files depending on the OS)
I already have my project with some source files and an Ant file, but I've been having trouble integrating it with Eclipse in an organized manner, so I would really love to start from a fresh Eclipse project, set it up correctly, and then copy my Ant file and my source files piece-by-piece into the project in the most Eclipse-compatible way.
I will be continuing to play around with everything in an attempt to get it working as I like it. But if you have experience with this sort of thing, perhaps at your workplace, please give as much information as you can.
My main goal here is to learn this once and use it in my future projects. For now, I am developing a client-server application consisting of a JOGL applet frontend (using JNLP files) and an unattended server app. Up until now I've been doing it all by hand: writing and building in Eclipse, dragging the applet jar into my FTP client, SSHing the server jar and restarting it by hand, and all with no testing process. I'm hoping that by the end, the build process will be something like this: test locally on my machine with a copy of the native libraries; commit code changes to SVN; Hudson svn updates, uses the Ant buildfile to compile and run all JUnit tests; if all the tests pass, it proceeds to copy the server jar to my dedicated server and restart the running server instance, and then copy the client jar to my web server.
When I start a new project, I generally take the following steps:
Create a new Java project in Eclipse, using the Java project wizard and opting to keep source and class files in separate directories ('src' and 'class')
Set up the project in Eclipse with the package structure you want
Create a new repository (or folder in your repository) for the project, with the normal /trunk/ /branches/ /tags/ set up
Using your SVN client, import the contents of the project folder Eclipse made for the project (at the level where the 'src' and 'class' directories are) into the trunk of the repository
Confirm everything is in Subversion correctly (check the trunk out to a new location)
Using Eclipse, delete the project (and the files on disk)
Create a new project using the 'Checkout projects from SVN' wizard, using the trunk of the repository
Create your Ant script, ensure the project builds correctly, and add the Ant script to the repository
Now, from Hudson:
Create a new job and set the Subversion URL to the root of the repository
Add a build set that will use Ant (I've always had to install my own version of Ant and ensure it's configured correctly to have this work) and will call the Ant script you
Use 'Build Now' to see if the job will build correctly
You can invoke your JUnit tests from Ant in the build script (first compile them with the javac task, then run them with the junit task). Hudson will track the test results if you ask it to.
You can use a shell script build step to copy your server jar to where it's needed, and restart the server instance. Like Mnementh said, it sounds like you've got the native libraries sorted...
If you are not tied to using ant, and are prepared to use Maven, then it is simply a matter of setting up Maven with the Eclipse plugin which generates the Eclipse projects for you.
Hudson already knows how to build Maven projects, so that is taken care of.
So long as you can pull your projects into eclipse, then it will be able to run the unit tests, and hudson can use the previously mentioned maven support to run the unit tests as well.
If you use Maven, then you will want to follow it's guidelines on how to create a project, here is a good starting point.
Hope this helps.
At our company we actually use Eclipse, Java, Ant, SVN, Junit and Hudson. That is all you mentioned except the native libraries. If you said your ant-buildscript already works with the native libraries that problem seems solved too. To integrate it well into eclipse you could do it in two ways: Use Ant also from Eclipse (has downsides) or the developer has to install the native library for his machine properly, so that Eclipse can compile without a problem and for continuous integration it will be downloaded by Ant.