I have a project which depends on several 3rd party libraries, the project itself is packaged as a jar and distributed to other developers as a library.
Those developers add the dependencies to their classpath and use my library in their code.
Recently I had an issue with one of the 3rd party dependencies, the apache commons codec libary,
The problem is this:
byte[] arr = "hi".getBytes();
// Codec Version 1.4
Base64.encodeBase64String(arr) == "aGk=\r\n" // this is true
// Codec Version 1.6
Base64.encodeBase64String(arr) == "aGk=" // this is true
As you can see the output of the method has changed with the minor version bump.
My question is, I don't want to force the user of my library to a specific minor version of a 3rd party library. Assuming I know about the change to the dependent library, is there anyway in which I can recognize which library version is being included in the classpath and behave accordingly? or alternatively, what is considered to be the best practice for these kind of scenarios?
P.S - I know that for the above example I can just use new String(Base64.encodeBase64(data, false)) which is backwards compatible, this is a more general question.
You ask what is the "best practice" for this problem. I'm going to assume that by "this problem" you mean the problem of 3rd party library upgrades, and specifically, these two questions:
When should you upgrade?
What should you do to protect yourself against bad upgrades (like the commons-codec bug mentioned in your example)?
To answer the first question, "when should you upgrade?," many strategies exist in industry. In the majority of the commercial Java world I believe the current dominant practice is "you should upgrade when you are ready to." In other words, as the developer, you first need to realize that a new version of a library is available (for each of your libraries!), you then need to integrate it into your project, and you are the one who makes the final go/no-go decision based on your own test bed --- junit, regression, manual testing, etc... whatever it is you do to ensure quality. Maven facilitates this approach (I call it version "pinning") by making multiple versions of most popular libraries available for automatic download into your build system, and by tacitly fostering this "pinning" tradition.
But other practices do exist, for example, within the Debian Linux distribution it is theoretically possible to delegate a lot of this work to the Debian package maintainers. You would simply dial in your comfort level according to the 4 levels Debian makes available, choosing newness over risk, or vice versa. The 4 levels Debian makes available are: OLDSTABLE, STABLE, TESTING, UNSTABLE. Unstable is remarkably stable, despite its name, and OLDSTABLE offers libraries that may as much as 3 years out of date compared to the latest-and-greatest versions available on their original "upstream" project websites.
As for the 2nd question, how to protect yourself, I think the current 'best practice' in industry is twofold: choose your libraries based on reputation (Apache's is generally pretty good), and wait a little while before upgrading, e.g., don't always rush to be on the latest-and-greatest. Maybe choose a public release of the library that has already been available 3 to 6 months, in the hope that any critical bugs have been flushed out and patched since the initial release.
You could go farther, by writing JUnit tests that specifically protect the behaviours you rely on in your dependencies. That way, when you bring down the newer version of a library, your JUnit would fail right away, warning you of the problem. But I don't see a lot of people doing that, in my experience. And it's often difficult to be aware of the precise behaviour you are relying on.
And, by the way, I'm Julius, the guy responsible for this bug! Please accept my apologies for this problem. Here's why I think it happened. I will speak only for myself. To find out what others on the apache commons-codec team think, you'll have to ask them yourself (e.g., ggregory, sebb).
When I was working on Base64 in versions 1.4 and 1.5, I was very much focused on the main problem of Base64, that is, encoding binary data into the lower-127 ASCIi, and the decoding it back to binary.
So in my mind (and here's where I went wrong) the difference between "aGk=\r\n" and "aGk=" is immaterial. They both decode to the same binary result!
But thinking about it in a broader sense after reading your stackoverflow posting here, I realize there is probably a very popular usecase that I never considered. That is, password checking against a table of encrypted passwords in a database. In that usecase you probably do the following:
// a. store user's password in the database
// using encryption and salt, and finally,
// commons-codec-1.4.jar (with "\r\n").
//
// b. every time the user logs in, encrypt their
// password using appropriate encryption alg., plus salt,
// finally base64 encode using latest version of commons-codec.jar,
// and then check against encrypted password in the database
// to see if it matches.
So of course this usecase fails if commons-codec.jar changes its encoding behaviour, even in immaterial ways according to the base64 spec. I'm very sorry!
I think even with all of the "best-practices" I spelled out at the beginning of this post, there's still a high probability of getting screwed on this one. Debian Testing already contains commons-codec-1.5, the version with the bug, and to fix this bug essentially means screwing people who used version 1.5 instead of version 1.4 where you did. But I will try to put some documentation on the apache website to warn people. Thanks for mentioning it here on stack-overflow (am I right about the usecase?).
ps. I thought Paul Grime's solution was pretty neat, but I suspect it relies on projects pushing version info in the the Jar's META-INF/MANIFEST.MF file. I think all Apache Java libraries do this, but other projects might not. The approach is a nice way to pin yourself to versions at build-time though: instead of realizing that you depend on the "\r\n", and writing the JUnit that protects against that, you can instead write a much easier JUnit: assertTrue(desiredLibVersion.equals(actualLibVersion)).
(This assumes run-time libs don't change compared to build-time libs!)
package stackoverflow;
import org.apache.commons.codec.binary.Base64;
public class CodecTest {
public static void main(String[] args) {
byte[] arr = "hi".getBytes();
String s = Base64.encodeBase64String(arr);
System.out.println("'" + s + "'");
Package package_ = Package.getPackage("org.apache.commons.codec.binary");
System.out.println(package_);
System.out.println("specificationVersion: " + package_.getSpecificationVersion());
System.out.println("implementationVersion: " + package_.getImplementationVersion());
}
}
Produces (for v1.6):
'aGk='
package org.apache.commons.codec.binary, Commons Codec, version 1.6
specificationVersion: 1.6
implementationVersion: 1.6
Produces (for v1.4):
'aGk=
'
package org.apache.commons.codec.binary, Commons Codec, version 1.4
specificationVersion: 1.4
implementationVersion: 1.4
So you could use the package object to test.
But I would say that it's a bit naughty for the API to have changed the way it did.
EDIT Here is the reason for the change - https://issues.apache.org/jira/browse/CODEC-99.
You could calculate a md5 sum of the actual class file and compare it to the expected. Could work like this:
String classname = "java.util.Random"; //fill in the your class
MessageDigest digest = MessageDigest.getInstance("MD5");
Class test = Class.forName(classname);
InputStream in = test.getResourceAsStream("/" + classname.replace(".", "/") + ".class");
byte[] buffer = new byte[8192];
int read = 0;
while ((read = in.read(buffer)) > 0) {
digest.update(buffer, 0, read);
}
byte[] md5sum = digest.digest();
BigInteger bigInt = new BigInteger(1, md5sum);
String output = bigInt.toString(16);
System.out.println(output);
in.close();
Or maybe you could iterate over the filenames in the classpath. Of course this only works, if the devs use the original filenames.
String classpath = System.getProperty("java.class.path");
for(String path:classpath.split(";")){
File o = new File(path);
if(o.isDirectory()){
....
}
}
Asaf, I solve this problem by using Maven . Maven has nice versioning support for all artifacts you use in your project. On top of that, I use the excellent Maven Shade Plugin which gives you ability to package all 3rd party libraries (maven artifacts) in a single JAR file, ready for deployment. All other solutions are just inferior - I am talking from my personal experience - I've been there, done that... Even wrote my own plugin-manager, etc. Use Maven, that is my friendly advice.
replacing the newline with empty string could be a solution?
Base64.encodeBase64String(arr).replace("\r\n","");
I would create 2+ different versions of a library to complement appropriate third party library version and provide manual which one to use. Probably write correct pom for it.
To resolve your problem I think the best way is to use a OSGi container, so you can choose your version of the 3rd party dependency and other libraries can safely use the other version without any conflict.
If you cannot rely on a OSGi container then you can use the implementation version in the MANIFEST.MF
Maven is a great tool, but cannot alone resolve your problem.
Related
I am aware that it's quite a weird use case to depend on having JVM installed for some OS source sets, allow me to go through my use case.
I'm writing a simple utility to wrap calls for the steamCMD (https://developer.valvesoftware.com/wiki/SteamCMD), which has platform dependent installation procedures. So, naturally I should have
// commonMain / steamCmdGetter.kt
expect interface SteamCmdGetter {
fun installClient()
}
// [OS] / steamCmdGetter.kt
actual interface SteamCmdGetter { /* ... */ }
On the other hand, my utility also needs to do work with the file storage (for example, downloading and checking client existence in storage), so I could also use a file class.
// commonMain / File.kt
expect interface File
I am aware that the JB team has an explicit recommendation on its tutorials.
We recommend that you use expected and actual declarations only for Kotlin declarations that have platform-specific dependencies. It is better to implement as much functionality as possible in the shared module even if doing so takes more time.
Yet, against the warnings I wish not to write a MyFile implementation to save efforts from reinventing the wheel for such a common task, but java.io.File has been so dominant in the scene that I could not find any Kotlin alternatives on Gradle / Maven.
Does this means I am forced to write MyFile in the end? Or is there a workaround for importing Java libraries to Kotlin MPP platform sourceSets?
First of all, one can use Java libraries only for jvm and android targets, not the others provided by the Kotlin/Multiplatform. In fact, this is exactly a targets subset that is using Kotlin/JVM. Neither Kotlin/JS nor Kotlin/Native provide interoperability with Java, they has their own interop capabilities. See this page to get some details on the difference. About working with files in particular. Most probably the answer is yes and you'll have to implement it per-target. This kind of work is usually platform-specific, as it hardly rely on the OS implementation. However, part of the functionality you search for should be definitely found in the platform.posix.* platform library, even if it would appear more C-stylish.
P.S. Quick search across the Web led me to this community libraries list, maybe it would help. Also, kotlinlang Slack community(find link here) may have some interesting solutions to share.
In Java, I often see JAR files named with the version number of the software (jsoup-1.11.2.jar), while others are not (freemarker.jar).
Is this just a best practice/convention, or is there some functional reason for it?
Simple answer: no, this is purely a convention.
Obviously, tooling that checks versions can do that easily when version numbers are hard-coded like this. But there is no generic (like jvm based) tool relying on it.
And beyond that - sometimes this scheme is even counter productive. In our self grown build setup we have to always remember to update the build scripts after replacing JAR files - because a new version changes the file name (because version part of the file name).
Having the version in the name of the file allows you to quickly determine which of the n files you have is the latest. Also if you have no way of determining what the version is from within the program it can be helpful.
I want to transliterate (not translate!) text from arbitrary (as far as possible) languages to English in an Android app. Is there a built-in way?
I've found https://android.googlesource.com/platform/libcore/+/master/luni/src/main/java/libcore/icu/Transliterator.java but it doesn't seem to be available by default (at least the IDE doesn't find it). Do I simply need to add this code, as suggested by the comment in Where can I get a JAR to import libcore.io??
Alternately, I could add ICU4J to dependencies and follow icu4j cyrillic to latin. But this is a very large dependency (though Proguard should help).
Finally, I could easily add transliteration from Cyrillic myself and wait until/if someone actually needs other languages (with obvious drawbacks).
Under the hood, Android has all of ICU4J available under android.icu, but only a subset is exposed as public API. If you want to use a class that isn't exposed, you can write code that uses the class and it should work fine. However, doing so is not technically supported, so there could be some version of Android somewhere that doesn't have the class for whatever reason and causes your code to break. (This is unlikely to happen in practice, but possible nonetheless.)
NOTE: The namespace android.icu was added in Android 7 Nougat, so it may or may not be usable yet depending on the version of Android you are targeting.
try compile "org.robovm:robovm-rt:+".
robovm-rt has libcore inside. works for me.
I normally use known CLI parsers (external libraries):
Apache Commons CLI http://commons.apache.org/cli/ (version 1.2)
Java Gems http://code.google.com/p/javagems/
JArgs http://jargs.sourceforge.net/
...
I haven't found one in the standard Java library, and I wonder if new versions of Java are providing an implementation so I can save a dependency. Does anyone know if there is something like that or a plan to include it in the future?
If when you say "native" you mean "java implementation included into JDK", the answer is "no". Obviously you can always create your own (more or less simple) parser based on arrays and string operations provided by java and JDK.
Concerning to choice among java CLI parsers I'd suggest you to use arg4j and can refer you to the following discussion: Java library for parsing command-line parameters?
One of the answers contains a very long list of libraries.
And the last note. I do not know why do you want to "save the dependencies". Use one of build tools that manage your dependencies (e.g. ivy, maven, gradle) and forget about such problems. if you want to distribute your program as a single jar, you can pack all your dependencies together with your application. Both maven and gradle can do this. If you want to achieve minimal jar size ... make your choice: what is more important for you - size or modularity. In most cases size is not an issue these days.
I don't think there is anything included in the JDK. Actually OpenJDK itself uses JOpt Simple (see comment at the bottom of the page).
I'm using com.sun.net.httpserver.HttpServer in my project. However, it seems that the server leaks connections when it gets invalid data from the HTTP connection. The bug is this one:
http://bugs.sun.com/view_bug.do;jsessionid=dfe841c3152d878571573bafceb8?bug_id=6946825
Now, this is reported to be fixed in version "7(b94)" - however, we are still using Java 1.6 and it is unlikely that we would want switch Java versions at this point.
So, I am looking for ways to fix this situation. I don't have a lot of time, so I'd prefer quick solutions that work for now, over reimplementing a lot of things for later.
I have a few ideas on how to go about this:
Update to a more recent Java - this is something I don't want to do.
Find a jar which only contains a more recent version of com.sun.net.httpserver and make sure that jar loads before the system jars.
Find a drop-in replacement for com.sun.net.httpserver - I'm open to pointers here.
Modify code to work with another embedded HTTP server, hopefully one that isn't too different from the current one. I can rewrite the server setup code, somewhat, but most of the interfaces should stay the same.
Decompile the com.sun.net.httpserver.ServerImpl class, fix the offending places, and recompile that single class to a jar of it's own
But, I'm open to good suggestions!
Thank you in advance.
Fix is now implemented and works. I will paste here the relevant bits if anyone else needs these:
final Field httpserverimpl_server = Class.forName("sun.net.httpserver.HttpServerImpl").getDeclaredField("server");
final Field httpsserverimpl_server = Class.forName("sun.net.httpserver.HttpsServerImpl").getDeclaredField("server");
final Field serverimpl_allconnections = Class.forName("sun.net.httpserver.ServerImpl").getDeclaredField("allConnections");
final Field httpconnection_closed = Class.forName("sun.net.httpserver.HttpConnection").getDeclaredField("closed");
httpserverimpl_server.setAccessible(true);
httpsserverimpl_server.setAccessible(true);
serverimpl_allconnections.setAccessible(true);
httpconnection_closed.setAccessible(true);
Object serverimpl = httpserverimpl_server.get(server);
Set allconnections = (Set)serverimpl_allconnections.get(serverimpl);
LinkedList<Object> toRemove = new LinkedList<Object>();
for (Object conn : allconnections) {
if (httpconnection_closed.getBoolean(conn)) {
toRemove.add(conn);
}
}
for (Object conn : toRemove) {
allconnections.remove(conn);
}
Could you put a reverse proxy infront of the HTTP server, to make sure you only allow known good requests to come through? Varnish or Squid or Apache?
Or knock something up in Jetty so that it acts as a reverse proxy?
Another approach would be to grab the source code of the fixed version, rename the class and package so that it fits into your project, make the class public, and then use that implementation instead.
I can understand your reluctance to upgrade to a pre-release build of Java 7.
Here are my suggestions:
Get a Java support contract from Oracle and get them to provide you with a patch for Java 6 that fixes the bug.
Download the Java 6 sources for the release you are currently using, backport the bug fix from the Java 7 sources and build. Maybe you only need to do a build of certain JAR files.
Look at the code and see if you could develop a workaround. For example, you might be able to use reflection to dig out the "list of HttpConnection instances" that the bug report talks about, and periodically remove entries that look like they are dead. (I'd treat this as a last resort.)
(Updated: 2012-05-15)
And, now that Java 7 is well and truly released (we are now at 1.7u4):
upgrade to Java 7, and
get rid of the nasty reflective hacks that you used as a TEMPORARY workaround.
Do you have access to 7(b94)? Then you can compare the sources and see whether you can fix it by overriding or providing different accessors.