I have a javaagent attached to a remote process using agentmain which sends serialized java objects via socket to a client application.
There appears to be a bug in the formatting of the stream which I am not sure if it is an error in my writeExternal implementation or if it is something to do with the java versions. I am experimenting with different combinations of java 1.6 32 / 64 bits.
Basically the object stream is throwing a class not found exception, but the package name printed is something like org.mycode.b.e.b., the b.e.b part is nothing to do with any packages I own and I have searched all over and there is nothing in my code or libraries with that string... b.e.b.
So I am suspicious it is an encoding issue caused by either a encoding format discrepancy or a 32/64 bit word length issue, but I am not that experienced in these areas.
I am quite used to opening up the debugger to track down code issues, but less experienced diagnosing problems of this nature, with serialization and byte formats so I am asking to see what are the recommended approaches to debugging?
I have control/compilation/source of the client which uses code like:
objectInput = new ObjectInput(inputStream);
Object object = objectInput.readObject();
which currently throws class not found for the org.mycode.b.e.b.
I was thinking about re-writing it to store the serialized stream to a file and taking a look ( not sure what format the file will actually be in in regard to reading it), is that the right angle?
The format for ObjectOutputStream is unchanged whether you use 32-bit or 64-bit or a big endian or little endian machine. Where there have been problems in the past has been between Java 1.3 and Java 1.4 using slightly different formats. Also Eclipse and Sun used different algos to generate serialVersionUID so you can run into problems if you didn't hard code it.
However if you have packages like b.e.b you most likely have obfuscated your code which is designed to make it difficult to work with. I suggest you only debug/monitor your un-obfuscated code.
I addition to commend of #Peter Lawrey.
I think that to simplify your debugging you should work with non-obfuscated code. This will give you at least the tip where the problem is.
Moreover probably the fact of partial obfuscation is the root cause of your problem. Think: you object is serialized on client side that is not obfuscated. Let's say your class named MyFirstClass depends on MySecondClass and is serialized accordingly. But on server side the MySecondClass is named q. Why q? This is up to obfuscator. Moreover this name may be different for each build. So, the server side cannot find the class MySecondClass.
Other possible problem is serialiVersionUID. Probably you have to define this variable for all relevant classes and handle its value.
But first just avoid obfuscation and try again. I believe everything will work. Good luck.
Related
This question already has answers here:
What is a Java ClassLoader?
(8 answers)
Closed 5 months ago.
I'm still fairly new to java and currently working on a text-based adventure game as a practice project.
The engine loads scenes to play that are all child-classes from a "Scene" superclass and appear as for eg. "dungeon.java". Now I want the game to be expandable.
The ideas is that a user can drop new scenes as .java-files into a "Scenes" folder and everytime the game is launched it reads all files in that folder and safes them into a "Scene"-class array.
My problem is that I don't know how to code this. I googled a lot and tried various phrasings but all I can find are tutorials for reading lines from txt-files or similar.
Is it even possible to read a complete file into a variable without serialzation?
I already tried the following code, but didn't get around to test it yet.
private static void buildScenePool() {
File scenesFolder = new File("/scenes");
\\ setup filter for .java files
FilenameFilter javafilter = (dir, name) -> name.endsWith(".java");
File[] sceneList = scenesFolder.listFiles(javafilter);
\\ create new arry large enough for all scenes
allScenes = new Scene[sceneList.length];
try{
FileInputStream fileIn = new FileInputStream(scenesFolder);
ObjectInputStream objectIn = new ObjectInputStream(fileIn);
\\ iterate trough the list array and safe files to array
for (int x = 0; x < allScenes.length; x++) {
allScenes[x] = objectIn.readObject( (Scene)sceneList[x] );
}
objectIn.close;
} catch (IOException e) {
System.err.println(e.toString());
}
}
This is rather very complicated. java code is normally compiled, and thus, that means you'd have to scan for a new java file, compile it (which is its own complicated ordeal, as that also means setting up the classpath and the like properly so that the compiler knows what to do), then load in the class file the compiler produced, and then call the appropriate methods in there.
You can do that. It's just, quite complicated. Your standard JVM doesn't necessarily even ship with a compiler; this is solvable too (either demand that this runs only on one that does, and the modern deployment rules for java involve you getting a JVM on your user's machines, so you thus pick one that does include a compiler – or you can just ship the compiler as dependency with your app, javac is itself a java app and runs on any JVM).
However, the more usual approach is to not actually use java for this. Instead, use something java-like, but not java: Scripting languages, like groovy, or javascript (okay, that is not particularly java-like perhaps).
That is its own sort of complication. There are no easy answers to any of this.
I think it's time to first think broad strokes and determine how you want the user's experience (that is, a user that wants to add a scene) should be, and then ask a new SO question about your specific choice.
You write em, they install em
In this model, users simply download or pick a 'scene' impl that someone else wrote (a 'real' programmer with a full fork of the entire source code, an IDE, a build tool, the works). Don't knock it - programming is hard, and saying: "Oh, look, anybody can customize a scene, it's easy, just open notepad.exe, write something like this (example java file here), and dump it in the Scene folder and off you go!", but this is not at all easy. To us programmers that seems familiar at least, but to your average user you're asking them to just rattle off a poem in ancient sumerian - normal people don't program. And if they do program, they're programmers. They would much rather get instructions about how to fork a project and set it up in an IDE than some bizarreness about tossing raw java files someplace.
Scripting
This is what Processing (for programming arduinos) does, more or less what webbrowsers did (which explains why javascript is so weird), and most 'plugin systems' for pseudo-smart editors like TextMate and Emacs do: You script them. As in, you fully buy into the idea that the custom stuff written by the user is extremely simple and highly targeted (really only makes sense to run within the confines of your app, not as standalone stuff - and dependencies don't generally come up), and pick a language that fits that model, and java certainly is not that.
Obvious options are javascript and groovy. This is certainly possible but not at all easy. Still, if you're interested, search the web for tutorials on how to run javascript or groovy inside a JVM, and you'll get plenty of hits.
Java code, and you compile it
That's what your question is positing as only option. I don't recommend it, but you can do this if you must. Be aware that it seems to me, based on the way you worded your question and your example code which makes various newbie mistakes (such as print-and-continue exception handling, which is always wrong, using obsolete APIs, and messing with built-in serialization) that this is a few ballparks beyond your current skillset. A challenge is always cool, so, if you want to go for it, you should! Just be aware it'll be the most difficult thing you've ever written and it'll take a few fully dedicated weeks, with a lot of reading and experimenting.
Just definitions, really
The central tenet so far has been that you can actually program. Instructions that make the machine act in certain ways. Possibly you don't need any of that. If a Scene is really just a background colour and a few widgets displayed here and there, should it even be code at all? Maybe you just want a declarative setup: The plugin/scene writer just declares certain properties and that's all they get to do, you just 'run' such a declarative definition. In which case the 'language' of the declaration can be in JSON, XML, YAML, TOML, or any other format designed for configuration files and such, and you can forego the hairy business of attempting to compile/run user-provided code in the first place.
In order to load the Java classes into your application, you need to compile them. You could do this from Java by invoking the javac executable. See Starting a process in Java? for instructions on how you could do that. Once compiled, you'd then need to load the classes into the JVM using a class loader, e.g. by invoking ClassLoader.defineClass. You probably want to configure a protection domain as well, to prevent user provided classes from misbehaving.
However, Java might not be the best approach for extending your application. You could consider using a scripting language instead, like JavaScript. See Nashorn (an open source script engine that was included in previous versions of Java, and that can now be downloaded separately) and the Java Scripting Programmer's Guide for more information.
I am developing a Minecraft plugin which uses a class that I made called customPlayer. When I save the plugin data from a running instance, I put all of these objects into a HashMap<String,customPlayer> and save them with ObjectOutputStream. Loading these classes back into the same version of the plugin works great, but my problem arises when I modify the class and try to read the object using that modified class (usually associated with a new version of my plugin).
I thought about it for a bit, and thought I came up with a clever solution. My idea was to just include the old class files as an External Library inside the new version of the plugin, cross my fingers and hope it worked. It didn't.
Is there a better way to do this? I'm new to serialization and this kind of stuff, so any suggestions would be greatly appreciated. Below I will include a few Screenshots of the customPlayer class and the crash log of the server. Ideally any solution that is presented should be able to be used easily with future modifications to the class (Updates to the Jar downloaded Via a Github repo).
Instance Variables and Constructor of customPlayer.java
Is there a better way to do this?
There certainly is. Stop using Serialization and ObjectOutputStream. These classes are a disaster (even OpenJDK core team effectively agrees with this assessment). The output they generate is not particularly efficient (it's more bytes than is needed), it is not human readable, nor (easily) read by anything except java code, and it results in such hairy situations as you ran into.
Instead use e.g. Jackson to turn your objects into JSON, or use google's protobuf to turn it into efficient binary blobs.
You can read this JSON or these binary blobs in any language you want and you'll have your pick of the litter as far as libraries go. You will need to write some explicit code to 'save' an object (turn it into JSON / protobuf), and to 'read' one, but now you are free to change your code.
If you insist on continuing with serialization, you need to add a field named serialVersionUID, and set up readObject and writeObject. it's convoluted rocket science that's hard to get right. The details are in the javadoc of java.io.Serializable.
Do yourself a favour though. Don't do it.
I am designing a simple library that deals with properties files.
I noticed that since JRE 1.5 the class Properties defines methods like:
public synchronized void loadFromXML(InputStream in)
public void storeToXML(OutputStream os, String comment)
I am questioning the fact that this is a real enhancement in the API of this class. Properties files have been, since JRE 1.5 text based files, and the newly introduced XML format is not adding anything to the functionalities, other than the possibility to use a different forma which is
more verbose
more complex (to understand, to change, to parse)
more inefficient (it uses dom internally to parse into an hastable: it consumes more memory, it requires helper classes in the implementation, and most likely is also slower)
more fragile (xml requires escaping of characters <>&"' while properties only need to escape backslashes, since it also supports Java backslash escaping)
it breaks backward compatibility of the programs using it, since users running JDK 1.4 won't be able to read xml properties. (ok, who cares...)
So I fail to understand the reason behind why engineers in Sun added this feature.
The question is:
Does anybody finds some advantage of using an XML-based properties files over a traditional text based one?
I need to evaluate this problem, since I don't want to add a useless feature to my simple library that I cited before.
Did you ever used an XML-based properties file over a Java Properties file? And why?
Note: same question can be made for Log4J xml file format, but at least Log4J xml format adds nesting ability and some sort of syntax which has some meaning, and I do understand that. But with this xml format for properties, I don't.
If staying within the Java environment, using a Java properties file works great. Even if you expect other programming languages to interact with your library, you'll probably be ok with a 'regular' properties file. However, for hierarchical data, XML is the standard. The reason you may want to support this change, and possibly the reason why Sun included it, is that other programming languages have extensive libraries for parsing XML files for hierarchical data.
The reason I'm answering is because I have actually used this feature before! But not for a great reason. In one program I'm working on now, I've found it easiest to keep a set of data in a properties object and I output the object to XML so that it can later be read by Python. At the moment, the data is further manipulated in a Python script and more children are added to the XML file. Without being able to output easily to XML, this would be a little more painful.
If I had the time, I wouldn't bother outputting to XML though. The main reason I'm using the Python code that takes in the XML is because somebody else wrote it and I'm temporarily using it until I have the time to reevaluate that section of my program and re-code it.
So there's a reason for using the XML! It isn't a good one, but it's a reason.
I imagine there are other cases like this where having the properties outputted as an XML aids in compatibility with other languages, since most languages have a robust XML parsing library and it makes it easier to manipulate hierarchical data. And in scientific programming, it seems you rarely get the luxury of sticking to one language.
Some points:
You can use standard, cross-platform tools to create it
You don't need to worry about peculiarities of escaping and character encoding, as you can use standard tools, which actually makes it more robust. The old properties file format is poorly specified.
Standard, cross-platform tools can use the data.
For most applications Java is used in, a bit of start up time isn't going to make much difference (particularly given the start up time of the rest of the system).
Java SE 1.6 is a bout to complete its end-of-life. Pre-1.5 isn't particularly relevant for Java SE (or EE).
But no, I've never seen it actually used.
Afaik the XML format is encouraged because of the encoding: (by specs) strictly ASCII for plain files (may I suggest you http://mojo.codehaus.org/native2ascii-maven-plugin/), UTF-8 (default) for XML property files as stated in http://docs.oracle.com/javase/1.5.0/docs/api/java/util/Properties.html
edit: I beg your pardon: ISO-8859-1 for property plain files
I'm trying to create a process to patch our current java application so users only need to download the diffs rather than the entire application. I don't think I need to go as low level as a binary diff since most of the jar files are small, so replacing an entire jar file wouldn't be that big of a deal (maybe 5MB at most).
Are there standard tools for determining which files changed and generating a patch for them? I've seen tools like xdelta and vpatch, but I think they work at a binary level.
I basically want to figure out - which files need to be added, replaced or removed. When I run the patch, it will check the current version of the software (from a registry setting) and ensure the patch is for the correct version. If it is, it will then make the necessary changes. It doesn't sound like this would be too difficult to implement on my own, but I was wondering if other people had already done this. I'm using NSIS as my installer if that makes any difference.
Thanks,
Jeff
Be careful when doing this--I recommend not doing it at all.
The biggest problem is public static variables. They are actually compiled into the target, not referenced. This means that even if a java file doesn't change, the class must be recompiled or you will still refer to the old value.
You also want to be very careful of changing method signatures--you will get some very subtle bugs if you change a method signature and do not recompile all files that call that method--even if the calling java files don't actually need to change (for instance, change a parameter from an int to a long).
If you decide to go down this path, be ready for some really hard to debug errors (generally no traces or significant indications, just strange behavior like the number received not matching the one sent) on customer site that you cannot duplicate and a lot of pissed off customers.
Edit (too long for comment):
A binary diff of the class files might work but I'd assume that some kind of version number or date gets compiled in and that they'd change a little every compile for no reason but that could be easily tested.
You could take on some strict development practices of not using public final statics (make them private) and not every changing method signatures (deprecate instead) but I'm not convinced that I know all the possible problems, I just know the ones we encountered.
Also binary diffs of the Jar files would be useless, you'd have to diff the classes and re-integrate them into the jars (doesn't sound easy to track)
Can you package your resources separately then minimize your code a bit? Pull out strings (Good for i18n)--I guess I'm just wondering if you could trim the class files enough to always do a full build/ship.
On the other hand, Sun seems to do an okay job of making class files that are completely compatible with the previous JRE release, so they must have guidelines somewhere.
You may want to see if Java WebStart can help you as it is designed to do exactly those things you want to do.
I know that the documentation describes how to create and do incremental updates, but we deploy the whole application as it changes very rarely. It is then an issue of updating the JNLP when ready.
How is it deployed?
On a local network I just leave everything as .class files in a folder. The startup script uses robocopy or rsync to copy from network share to local. If any .class file is different it is synced down. If not, it doesn't sync.
For non-local network I created my own updater. It downloads a text file of md5sums and compares to local files. If different it pulls file down from http.
A long time ago the way we solved this was to used Classpath and jar files. Our application was built in a Jar file, and it had a launcher Jar file. The launcher classpath had a patch.jar that was read into the classpath before the main application.jar. This meant that we could update the patch.jar to supersede any classes in the main application.
However, this was a long time ago. You may be better using something like the Java Web Start type of approach, which offers more seamless application updating.
Has anyone used the noweb literate programming tool on a large Java project, where several source code files must be generated in different subdirectories? How did you manage this with noweb? Are there any resources and/or best practices out there?
Noweb will dump out files relative to the current working directory, or at the absolute path you specify. Just don't use * at the end of your filename (to avoid inserting the # preprocessor directives). I would recommend using %def with # to show where you define and use names.
<</path/to/file.java>>=
reallyImportantVariable += 1;
# %def reallyImportantVariable
noweb lets you reorder and (the real win) reuse snippets of code, which I don't think javac would understand.
I'd agree that since most people expect that you'll use javadoc, you're probably swimming against the stream to use noweb.
Literate Programming works its best if the generated intermediate code can point back to the original source file to allow debugging, and analyzing compiler errors. This usually means pre processor support, which Java doesn't support.
Additionally Literate Programming is really not necessary for Java, as the original need for a strict sequential order - which was what prompted Knuth to write a tool to put snippets together in the appropriate sequence - is not present. The final benefit of literate programming, namely being able to write prose about the code, is also available as Javadoc which allow you to put everything in as comments.
To me, there is no benefit in literate programming for Java, and only troubles (just imagine getting IDE support).
Any particular reason you are considering it?