I have a really simple code in Java which reads data from hdfs
try{
InputStream s = new GzipCompressorInputStream(hdfsFileSystem.open(filePath), false);
ByteStreams.copy(s, outputStream);
s.close();
}
catch (Exception ex){
logger.error("Problem with file "+ filePath,ex);
}
Sometimes (not always) it throws me exception
java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.closeQuietly(Ljava/io/Closeable;)V
at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1099)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:533)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.init(GzipCompressorInputStream.java:136)
at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.<init>(GzipCompressorInputStream.java:129)
[...]
On line below line:
InputStream s = new GzipCompressorInputStream(hdfsFileSystem.open(filePath), false);
I am using bellow maven dependency to load hadoop client:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.2.0</version>
</dependency>
Does anybody knows how to fix this problem? Of cource I can change catch(Exception e) to catch(Error e), but it isn't a solution just workaround.
Looks like in yours classpath present several "commons-io.jar" with different versions.
Method "closeQuietly(Ljava/io/Closeable;)" appeared in version 2.0.
Sometimes "commons-io.jar" with older version loaded first, and exception appeared.
Classpath fix required.
Related
I have a java program, that includes org.eclipse.swt libraries like "Display" and "Shell"
One of the first things this program does is:
private void authenticationFlow() {
Display display = new Display();
Shell shell = new Shell(display);
final Browser browser;
//some other code here
}
I exported this program to a runnable jar file and can run it just fine on my PC.
However, when trying to run it on a PC without eclipse installed, the Program does not start. No Exceptions whatsoever. It simply exits and does not run the rest of the code.
I tried to debug by creating a bunch of alert boxes like this:
private void authenticationFlow() {
popAlertBox(AlertType.INFORMATION, "Nice", "Something happened", "Starting auth");
Display display = null;
try {
display = new Display();
} catch (Exception e) {
popAlertBox(AlertType.ERROR, "Oh oh", "Something went wrong", e.getMessage());
}
popAlertBox(AlertType.INFORMATION, "Nice", "Something happened", "created display");
Shell shell = new Shell(display);
popAlertBox(AlertType.INFORMATION, "Nice", "Something happened", "created shell");
final Browser browser;
GridLayout gridLayout = new GridLayout();
gridLayout.numColumns = 3;
try {
shell.setLayout(gridLayout);
} catch (Exception e) {
popAlertBox(AlertType.ERROR, "Shell error", "Could not instantiate Shell: ", e.getMessage());
}
//rest of code
And
private void popAlertBox(AlertType type, String title, String header, String contentText) {
Alert alert = new Alert(type);
alert.setTitle(title);
alert.setHeaderText(header);
alert.setContentText(contentText);
alert.setX(GUIMain.getStageX() + GUIMain.getWidth()*0.4);
alert.setY(GUIMain.getStageY()+ GUIMain.getHeight()*0.4);
alert.showAndWait();
}
I end up seeing the "Starting auth" AlertBox, and that's it, the program exits right after.
I never reach the "created display" AlertBox.
So I figured it has to do with SWT itself.
Now my question is
1. Is this really directly related to SWT or am I misunderstanding something.
2. If it is, how can I have this program run on a PC without eclipse installed?
EDIT:
I use maven for all dependencies
here is an image of my libraries, including swt
I have tried surrounding my method which is called in a try catch like so:
try{
authenticationFlow();
}catch (Exception e) {
popAlertBox(AlertType.ERROR, "oh oh", "Something went wrong", e.getMessage());
}
popAlertBox(AlertType.INFORMATION, "Nice", "Something happened", "If you see this then something is in fact happening. final popup");
And NONE of those two popups are displayed. Not the one inside the catch block and the one afterwards neither.
i added following dependencies inside my pom.xml:
<dependency>
<groupId>org.eclipse</groupId>
<artifactId>swt</artifactId>
<version>3.3.0-v3346</version>
</dependency>
<dependency>
<groupId>org.eclipse.swt</groupId>
<artifactId>org.eclipse.swt.win32.win32.x86_64</artifactId>
<version>4.3</version>
</dependency>
and it is still not running on the pc, which does not have eclipse installed
If you want to run a program which is designed to have a ui written in SWT the best way is to provide the SWT library in the classpath of the application. This does not mean that you have to provide a whole eclipse. Eclipse is also just using SWT as the library for the ui.
Just grab the SWT jars and put them onto the classpath and the program should start presenting you a window which is expressed in the line Shell shell = new Shell(display).
Adding SWT using Maven:
<properties>
<swt.version>4.3</swt.version>
</properties>
<dependencies>
<dependency>
<groupId>org.eclipse.swt</groupId>
<artifactId>org.eclipse.swt.win32.win32.x86_64</artifactId>
<version>${swt.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.swt</groupId>
<artifactId>org.eclipse.swt.win32.win32.x86</artifactId>
<version>${swt.version}</version>
<!-- To use the debug jar, add this -->
<classifier>debug</classifier>
</dependency>
<dependency>
<groupId>org.eclipse.swt</groupId>
<artifactId>org.eclipse.swt.gtk.linux.x86</artifactId>
<version>${swt.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.swt</groupId>
<artifactId>org.eclipse.swt.gtk.linux.x86_64</artifactId>
<version>${swt.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.swt</groupId>
<artifactId>org.eclipse.swt.cocoa.macosx.x86_64</artifactId>
<version>${swt.version}</version>
</dependency>
</dependencies>
This will add SWT for every platform. You can also just add the one you need to build for.
When building the application into a "fat-jar" please make sure all the dependencies are provided.
Is there any way to get OpenCV from repository? Which artifact should I add to pom.xml? Every tutorial I'd found is from '14 and it seems like something changed - they say it is'nt in official Maven repository yet, but I've found entry:
<!-- https://mvnrepository.com/artifact/nu.pattern/opencv -->
<dependency>
<groupId>nu.pattern</groupId>
<artifactId>opencv</artifactId>
<version>2.4.9-7</version>
</dependency>
Sadly, I get error
Caused by: java.lang.UnsatisfiedLinkError: no opencv_java249 in java.library.path
when I'm using System.loadLibrary(Core.NATIVE_LIBRARY_NAME). Can I add this library in a way that would make my project include it and 'forget' about manually adding it to classpath?
Add the following dependency in your POM file:
<dependency>
<groupId>org.openpnp</groupId>
<artifactId>opencv</artifactId>
<version>3.2.0-0</version>
</dependency>
and replace the following lines:
System.loadLibrary(Core.NATIVE_LIBRARY_NAME)
with
nu.pattern.OpenCV.loadShared();
This should solve the problem in WINDOWS also. Happy Coding.
This worked for me.
nu.pattern.OpenCV.loadLibrary();
I'm using following maven dependency
<dependency>
<groupId>nu.pattern</groupId>
<artifactId>opencv</artifactId>
<version>2.4.9-4</version>
</dependency>
Try this, see if it works:
nu.pattern.OpenCV.loadShared();
System.loadLibrary(org.opencv.core.Core.NATIVE_LIBRARY_NAME);
More info here in API section: https://github.com/patternconsulting/opencv
Also have 2.4.9-7 opencv dependency.
There is currently no official way to use the official Java bindings for OpenCV as a Maven dependency (as already mentioned in the comments, the Maven artifact was already requested in #4588, but is still unattended). Nevertheless, there are 3 possible approaches to your problem:
java.lang.UnsatisfiedLinkError was thrown because you need to install the binding's binaries (that is "opencv_java") separately. Most likely, that unofficial artifact does not include them (or not the ones compatible with your system). In order to build the bindings:
git clone the OpenCV repository.
git checkout the intended version (it appears that you are using version 2.4.9, although more recent versions are available)
Follow the instructions here to build OpenCV and its Java bindings, thus yielding a dynamically linked library ("opencv_java249.dll", "libopencv_java249.so", or something else depending on your OS).
Copy the shared library file to your java.library.path (again, this variable is system-dependent, but can be defined when running your application). At this point you should be ready to use that artifact.
An alternative is to use other bindings: the JavaCPP presets for OpenCV seem to work just as nicely as the official ones, and these are registered in maven (binaries for various platforms included!). Just remember that the API may not be exactly the same.
This solution may sound too far out, but it has legitimately worked for me in the past. Basically, you can avoid using the bindings: implement your solution in C++, then either link it with the JVM via JNI or make it a separate application, used by the main application via other mechanisms of your system (process spawning, I/O channels, you name it). For instance, I have once made a service component for feature extraction that other programs would connect to via ZeroMQ sockets.
Just use it
nu.pattern.OpenCV.loadShared();
write a class with this static void method
class Test {
public static void loadOpenCVNativeLibrary() {
nu.pattern.OpenCV.loadShared();
}
}
and after call it in your application class (with static main) for web application (spring boot for example) like this
static {
Test.loadOpenCVNativeLibrary();
}
...
public static void main(String[] args) throws UnknownHostException {
}
All you need:
install jar in local maven repository with:
mvn install:install-file -Dfile=C:\opencv411\build\java\opencv-411.jar -DgroupId=org -DartifactId=opencv -Dversion=4.1.1 -Dpackaging=jar
create a dependency in pom.xml:
<dependency>
<groupId>org</groupId>
<artifactId>opencv</artifactId>
<version>4.1.1</version>
</dependency>
Now that jar is on, we need to somehow add the OpenCV libraries. I did this by adding the lib folder in java.library.path to the maven-surefire plugin:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
<configuration>
<argLine>-Djava.library.path=${project.build.outputDirectory}/lib</argLine>
</configuration>
</plugin>
public static void main(String[] arges) throws MalformedURLException,
IOException, Exception {
loadLibraries();
// create and print on screen a 3x3 identity matrix
System.out.println("Create a 3x3 identity matrix...");
Mat mat = Mat.eye(3, 3, CvType.CV_8UC1);
System.out.println("mat = " + mat.dump());
// prepare to convert a RGB image in gray scale
String location = "resources/Poli.jpg";
System.out.print("Convert the image at " + location + " in gray scale... ");
// get the jpeg image from the internal resource folder
Mat image = Imgcodecs.imread(location);
// convert the image in gray scale
Imgproc.cvtColor(image, image, Imgproc.COLOR_RGB2GRAY);
// write the new image on disk
Imgcodecs.imwrite("resources/Poli-gray.jpg", image);
System.out.println("Done!");
}
private static void loadLibraries() {
try {
InputStream in = null;
File fileOut = null;
String osName = System.getProperty("os.name");
// String opencvpath = System.getProperty("user.dir");
String opencvpath = "C:\\opencv411\\build\\java\\";
if (osName.startsWith("Windows")) {
int bitness = Integer.parseInt(System.getProperty("sun.arch.data.model"));
if (bitness == 32) {
opencvpath = opencvpath + "\\x86\\";
} else if (bitness == 64) {
opencvpath = opencvpath + "\\x64\\";
} else {
opencvpath = opencvpath + "\\x86\\";
}
} else if (osName.equals("Mac OS X")) {
opencvpath = opencvpath + "Your path to .dylib";
}
System.out.println(opencvpath);
// System.out.println("Core.NATIVE_LIBRARY_NAME = " + Core.NATIVE_LIBRARY_NAME);
System.out.println("Core.NATIVE_LIBRARY_NAME = " + "opencv_java411.dll");
// System.load(opencvpath + Core.NATIVE_LIBRARY_NAME + ".dll");
System.load(opencvpath + "opencv_java411.dll");
} catch (Exception e) {
throw new RuntimeException("Failed to load opencv native library", e);
}
}
For those who wants to use OpenCV 3.2 in MacOs environment, you can use following repository definition:
<repositories>
<repository>
<id>kodfarki</id>
<url>https://raw.githubusercontent.com/kodfarki/repository/master/</url>
</repository>
</repositories>
There is also an example project in https://github.com/kodfarki/opencv-example.
To use this example project, you still need to install OpenCV binaries
brew tap homebrew/science
brew install opencv3 --with-java --with-contrib
For windows there was a problem with #Sachin Aryal's answer. The answer by #Anirban Chakraborty is a very good hint. But, there was still issues at runtime as described in this thread.
Finally replacing OpenCV.loadShared(); with OpenCV.loadLocally(); worked for me.
I am trying to use an open source tool built on Batik and I am running into trouble with one of the dependencies when I try to build it. Pretty sure this is something to do with classpaths and library locations, but I can't figure out what is happening.
So the project I am working with ( SVG2EMF ) is using the FreeHep EMF Driver, which in turn uses the FreeHep GraphicsIO project. Because these three have not been playing nicely on my system ( Ubuntu 14.04 ) I've downloaded the source for all three to try and step through the problem.
Everything builds correctly and I can step through the code successfully, but the unit tests on SVG2EMF fail at the point where the EMF Driver makes a call to something from GraphicsIO- the relevant parts of the code in question is here:
import org.freehep.graphicsio.ImageGraphics2D;
import org.freehep.graphicsio.ImageConstants;
// ...snip...
public class AlphaBlend extends EMFTag implements EMFConstants
{
// ...snip...
public void write(int tagID, EMFOutputStream emf) throws IOException
{
emf.writeRECTL(bounds);
emf.writeLONG(x);
emf.writeLONG(y);
emf.writeLONG(width);
emf.writeLONG(height);
dwROP.write(emf);
emf.writeLONG(xSrc);
emf.writeLONG(ySrc);
emf.writeXFORM(transform);
emf.writeCOLORREF(bkg);
emf.writeDWORD(usage);
emf.writeDWORD(size); // bmi follows this record immediately
emf.writeDWORD(BitmapInfoHeader.size);
emf.writeDWORD(size + BitmapInfoHeader.size); // bitmap follows bmi
emf.pushBuffer();
int encode;
// plain
encode = BI_RGB;
ImageGraphics2D.writeImage(
(RenderedImage) image,
ImageConstants.RAW.toLowerCase(),
ImageGraphics2D.getRAWProperties(bkg, "*BGRA"),
new NoCloseOutputStream(emf));
// emf.writeImage(image, bkg, "*BGRA", 1);
// png
// encode = BI_PNG;
// ImageGraphics2D.writeImage(image, "png", new Properties(), new
// NoCloseOutputStream(emf));
// jpg
// encode = BI_JPEG;
// ImageGraphics2D.writeImage(image, "jpg", new Properties(), new
// NoCloseOutputStream(emf));
int length = emf.popBuffer();
emf.writeDWORD(length);
emf.writeLONG(image.getWidth());
emf.writeLONG(image.getHeight());
BitmapInfoHeader header = new BitmapInfoHeader(image.getWidth(), image
.getHeight(), 32, encode, length, 0, 0, 0, 0);
bmi = new BitmapInfo(header);
bmi.write(emf);
emf.append();
}
This throws a NoClassDefFoundError specifically relating to org.freehep.graphicsio.ImageGraphics2D on that writeImage call. When I step through in the debugger, a watch on ImageConstants.RAW has the value of Unknown type "org.freehep.graphicsio.ImageConstants" even though the application built quite happily with those references. Any references to ImageGraphics2D behave in exactly the same way.
The dependency in the SVG2EMF pom.xml looks like this:
<dependencies>
<!-- some other dependencies -->
<dependency>
<groupId>org.freehep</groupId>
<artifactId>freehep-graphicsio-emf</artifactId>
<version>2.1.1</version>
</dependency>
</dependencies>
Dependency from the FreeHEP EMF Driver looks like this:
<dependencies>
<!-- necessary because transitive deps seem to go above inhertied deps -->
<dependency>
<groupId>org.freehep</groupId>
<artifactId>freehep-util</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.freehep</groupId>
<artifactId>freehep-graphicsio</artifactId>
<version>2.1.1</version>
</dependency>
<!-- Other dependencies -->
</dependencies>
Can anybody shed any light on what is actually going on here or what I need to be doing in order to enable this to work?
EDIT: I think I have found where the problem is coming from- way down the StackTrace I see a "Caused by: ExceptionInInitializerError" - which appears to mark the class as inaccessible from then on. So the dependency does exist, but an exception is being thrown by the initializer which causes the JRE to mark it as unusable.
Further Edit: To solve these problems it can be useful ( although it is not mentioned anywhere on the freehep.org website ) to know that the project is now hosted on Github so you can find newer versions from there. In my case going straight to the latest version solved the problem.
I have a project with multiple maven modules
project/pom.xml
/external_services/pom.xml
/ifs/pom.xml
/src/test/java/
/MockIFSClient.java
/IFSClientTest.java
/src/test/java/resources/sample.json
/inventory/pom.xml
/business/pom.xml
/src/main/java/InventorySummary.java
/services/pom.xml
/src/main/java/InventorySummaryResource.java
/src/main/test/InventorySummaryResourceTest.java
MockIFSClient access sample.json as
try {
inventoryPriceDetails = mapper.readValue(new File(getClass().getResource("/getInventoryAndPriceResponse.json").getPath()), new TypeReference<List<InventoryPriceDetail>>() {
});
} catch (final IOException e) {
throw new RuntimeException("could not read resource :" + e.getMessage());
}
so IFSClientTest runs fins since they are in same package.
Problem?
InventorySummaryResourceTest calls MockIFSClient which tries to access the same code, but now it fails as
could not read resource :file:/Users/harith/IdeaProjects/inventory_api/external_services/ifs/target/ifs-1.0-SNAPSHOT-tests.jar!/sample.json (No such file or directory)
services/pom.xml has dependency as
<dependency>
<groupId>com.org.project.external_services</groupId>
<artifactId>ifs</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
Question
What change do I make to
new File(getClass().getResource("/getInventoryAndPriceResponse.json").getPath())
so that it can be accessed from different modules as well
This answer helped me getting it right
How to really read text file from classpath in Java
I changed my code then to
inventoryPriceDetails = mapper.readValue(getClass().getResourceAsStream("/getInventoryAndPriceResponse.json"), new TypeReference<List<InventoryPriceDetail>>() {});
I am using StAX to create a quite large xml document. Until now I was using the IndentingXMLStreamwriter class to get a well formatted document (see also this answer). A few days ago we setup a jenkins server with an older jdk version (6.26), on which i get build errors.
package com.sun.xml.internal.txw2.output does not exist
I assume the package cannot be found because of the installed jdk version. For different reasons this cannot be changed
(by the way, does anyone know the jdk version, at which this package (com.sun.xml.internal.txw2.output) was added?).
Therefore I am looking for an alternative to do the indenting. I would prefer a solution similar to the one I was using, which means without reparsing the document. Any ideas or suggestions?
Thanks
Lars
Instead of com.sun.xml.internal.txw2.output.IndentingXMLStreamWriter use com.sun.xml.txw2.output.IndentingXMLStreamWriter that can be found in:
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>txw2</artifactId>
<version>2.2.11</version>
</dependency>
Just expanding on Michael Kay's answer ( https://stackoverflow.com/a/10108591/2722227 ).
maven dependencies:
<dependency>
<groupId>net.sf.saxon</groupId>
<artifactId>Saxon-HE</artifactId>
<version>9.6.0-5</version>
</dependency>
java code:
import java.io.OutputStream;
import net.sf.saxon.Configuration;
import net.sf.saxon.s9api.Processor;
import net.sf.saxon.s9api.SaxonApiException;
import net.sf.saxon.s9api.Serializer;
import net.sf.saxon.s9api.Serializer.Property;
public class Main {
public static void main(String[] args) throws Exception {
OutputStream outputStream = System.out;
writeXmlDocument(outputStream);
}
private static void writeXmlDocument(OutputStream outputStream){
Configuration config = new Configuration();
Processor processor = new Processor(config);
Serializer serializer = processor.newSerializer();
serializer.setOutputProperty(Property.METHOD, "xml");
serializer.setOutputProperty(Property.INDENT, "yes");
serializer.setOutputStream(outputStream);
try {
XMLStreamWriter writer = serializer.getXMLStreamWriter();
try {
writer.writeStartDocument();
{
writer.writeStartElement("root_element_name");
{
writer.writeStartElement("child_element");
writer.writeEndElement();
}
writer.writeEndElement();
}
writer.writeEndDocument();
writer.flush();
writer.close();
} catch (XMLStreamException e) {
e.printStackTrace();
}
} catch (SaxonApiException e) {
e.printStackTrace();
}
}
}
If other suggestions don't work, you can get an indenting XMLStreamWriter from Saxon like this:
Processor p = new net.sf.saxon.s9api.Processor();
Serializer s = p.newSerializer();
s.setOutputProperty(Property.METHOD, "xml");
s.setOutputProperty(Property.INDENT, "yes");
s.setOutputStream(....);
XMLStreamWriter writer = s.getXMLStreamWriter();
One advantage is that this allows you a lot of control over the serialization using other serialization properties.
There is an alternative implementation of IndentingXmlStreamWriter, which is provided as part of the open source stax-utils project here: http://java.net/projects/stax-utils/pages/Home
stax-utils seems to be a project set up to provide utilities based around the jsr-173 streaming xml api for Java
You'd need to add the stax-utils jar as a dependency for your project. Then you can import javanet.staxutils.IndentingXmlStreamWriter
Since stax-utils is in the maven central repository, if you use maven for your dependencies you can get it with:
<dependency>
<groupId>net.java.dev.stax-utils</groupId>
<artifactId>stax-utils</artifactId>
<version>20070216</version>
<exclusions>
<exclusion>
<groupId>com.bea.xml</groupId>
<artifactId>jsr173-ri</artifactId>
</exclusion>
</exclusions>
</dependency>
Functionality seems very similar / equivalent to the txw2 class
I have excluded jsr173-ri since I am using jdk 1.7. I think 1.6+ has the jsr173 api as a standard feature, but if you are using 1.5 or lower you'd need the extra jsr173 jar.
If you are using Maven and Java 8, you can import the following to use this class:
<!-- https://mvnrepository.com/artifact/com.sun.xml.bind/jaxb-impl -->
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId>
<version>2.1.17</version>
</dependency>
And then, you import it as: import com.sun.xml.txw2.output.IndentingXMLStreamWriter;