We Keep Coding

Java deobfuscator supporting variable renaming?

I'm trying to reverse engineering a .jar file (basically a group of .class files). However, there are two difficulties:
Some of the variables, methods and classes are named with unicode characters, which cannot be properly displayed. This makes source editing very inconvenient.
Some of the classes are named with Java keywords (such as if). Therefore Java compiler will complain when recompiling the reverse-engineered source files.
Are there any Java deobfuscators that can overcome these difficulties?

There are several tools that can rename Java identifers. For example, you can just run Proguard on it to rename everything.
That being said, I would suggest reconsidering your approach. Java compilation and decompilation are both lossy processes. Decompilation is useful for reverse engineering, but you generally cannot expect to be able to recompile the decompiled code. If it has previously been obfuscated, you may as well forget it.
The best way to modify obfuscated code is to use the Krakatau assembler and disassembler. Krakatau assembly can be reversibly transformed to and from bytecode, meaning you can edit arbitrary classes without messing things up, no matter how they have been obfuscated. The only downside is that you have to be familiar with Java bytecode.

Cuchaz's Enigma supports renaming local variables. It's no longer being updated and runs on BCEL which is also no longer being updated. As long as you're not decompiling anything BCEL can't handle it should work just fine though.

How eclipse deduces the methods of a class in a jar file

like:
import com.xxx.utility.*;
class MyClass{
public static void main(String[] args){
MyUtiliy ut = new MyUtiliy();
MyUtility.doAdd(5, 6);
.......
}
}
When put the "." after MyUtiliy, eclipse will tell you all the methods you can use, how does eclipse achieve this?
Does eclipse use the reflection on the fly? (like the answer of this thread? )

The architecture of the eclipse software is describe here, in the section 6.1.2. Java Development Tools (JDT) it briefly describes the incremental build system used. That system would have all the relavent information to populate the autocomplete mechanism.
For the exact mechanism, you would have to look at the eclipse source code.

Yes Eclipse (and any other Java IDEs) uses reflection.
If fact Eclipse uses a ClassLoader for each project's libraries, so it load the classes in jar files, and after that everything is easy, it can get information using reflection.
By the way java IDEs not only use reflection, but also read class debug info, to extract parameter names, and so on.

There is an explanation in this article. Basically the Eclipse Java compiler builds an Abstract Syntax Tree (AST) of your code which lets it find all the information it needs for autocompletion very quickly.
So it is not using reflection for this, rather it is compiling the code in to an internal form for quick access.
When no source code is available (you just have a .class file) it is still possible to construct the part of the AST containing the class methods and types which are needed for completion. This appears to be done by reading the .class files directly rather than using a class loader (org.eclipse.jdt.internal.compiler.classfmt.ClassFileReader)

Is there a Java library to generate class files from an AST?

This page describes how I can use the code generator in javac to generate code given that I can build an AST (using a separate parser which I wrote). The technique involves editing javac's source code to basically bypass the Java parser, so that one could supply his/her own AST to the code generator. This could work, but I was hoping to do it in a slightly cleaner way. I want to include the code generating part of javac as a library in my project so I can use it to generate code, without bringing with it the rest of javac's source.
Is there a way to do this with javac, or is there perhaps a better library?
Also, feel free to change the question's title. I couldn't think of a better one, but it's a little ambiguous. If you suggest an edit for a better title, I'll accept it.

I think what you might be interested in is a java library like BCEL(ByteCode Engineering Library)
I played around with it back when I took a class on compiler construction, basically, it has a nice wrapper for generating the constant pool, inserting named bytecode instructions into a method and whatnot, then when you are done, you can either load the class at runtime with a custom classloader, or write it out to a file in the normal way.
With BCEL, it should be relatively easy to go from the syntax tree to the java bytecodes, albeit a bit tedious, but you may want to just use BCEL to generate the raw bytecode without building the tree as well in some cases.

Another cool framework is ASM, a bytecode analysis and manipulation framework.
In case you do not want to use a framework, as of now (2014), it is not possible to generate bytecode from a tree using the arbitrary representations of com.sun.source.tree.* as said here.

Automatically generating Java source code

I'm looking for a way to automatically generate source code for new methods within an existing Java source code file, based on the fields defined within the class.
In essence, I'm looking to execute the following steps:
Read and parse SomeClass.java
Iterate through all fields defined in the source code
Add source code method someMethod()
Save SomeClass.java (Ideally, preserving the formatting of the existing code)
What tools and techniques are best suited to accomplish this?
EDIT
I don't want to generate code at runtime; I want to augment existing Java source code

What you want is a Program Transformation system.
Good ones have parsers for the language you care about, build ASTs representing the program for the parsed code, provide you with access to the AST for analaysis and modification, and can regenerate source text from the AST. Your remark about "scanning the fields" is just a kind of traversal of the AST representing the program. For each interesting analysis result you produce, you want to make a change to the AST, perhaps somewhere else, but nonetheless in the AST.
And after all the chagnes are made, you want to regenerate text with comments (as originally entered, or as you have constructed in your new code).
There are several tools that do this specifically for Java.
Jackpot provides a parser, builds ASTs, and lets you code Java procedures to do what you want with the trees. Upside: easy conceptually. Downside: you write a lot more Java code to climb around/hack at trees than you'd expect. Jackpot only works with Java.
Stratego and TXL parse your code, build ASTs, and let you write "surce-to-source" transformations (using the syntax of the target language, e.g., Java in this case) to express patterns and fixes. Additional good news: you can define any programming language you like, as the target language to be processed, and both of these have Java definitions.
But they are weak on analysis: often you need symbol tables, and data flow analysis, to really make analyses and changes you need. And they insist that everything is a rewrite rule, whether that helps you or not; this is a little like insisting you only need a hammer in toolbox; after all, everything can be treated like a nail, right?
Our DMS Software Reengineering Toolkit allows the definition of an abitrary target language (and has many predefined langauges including Java), includes all the source-to-source transformation capabilities of Stratego, TXL, the procedural capability of Jackpot,
and additionally provides symbol tables, control and data flow analysis information. The compiler guys taught us these things were necessary to build strong compilers (= "analysis + optimizations + refinement") and it is true of code generation systems too, for exactly the same reasons. Using this approach you can generate code and optimize it to the extent you have the knowledge to do so. One example, similar to your serialization ideas, is to generate fast XML readers and writers for specified XML DTDs; we've done that with DMS for Java and COBOL.
DMS has been used to read/modify/write many kinds of source files. A nice example that will make the ideas clear can be found in this technical paper, which shows how to modify code to insert instrumentation probes: Branch Coverage Made Easy.
A simpler, but more complete example of defining an arbitrary lanauges and transformations to apply to it can be found at How to transform Algebra using the same ideas.

Have a look at Java Emitter Templates. They allow you to create java source files by using a mark up language. It is similar to how you can use a scripting language to spit out HTML except you spit out compilable source code. The syntax for JET is very similar to JSP and so isn't too tricky to pick up. However this may be an overkill for what you're trying to accomplish. Here are some resources if you decide to go down that path:
http://www.eclipse.org/articles/Article-JET/jet_tutorial1.html
http://www.ibm.com/developerworks/library/os-ecemf2
http://www.vogella.de/articles/EclipseJET/article.html

Modifying the same java source file with auto-generated code is maintenance nightmare. Consider generating a new class that extends you current class and adds the desired method. Use reflection to read from user-defined class and create velocity templates for the auto-generating classes. Then for each user-defined class generate its extending class. Integrate the code generation phase in your build lifecycle.
Or you may use 'bytecode enhancement' techniques to enhance the classes without having to modify the source code.
Updates:
mixing auto-generated code always pose a risk of someone modifying it in future to just to tweak a small behavior. It's just the matter of next build, when this changes will be lost.
you will have to solely rely on the comments on top of auto-generated source to prevent developers from doing so.
version-controlling - Lets say you update the template of someMethod(), now all of your source file's version will be updated, even if the source updates is auto-generated. you will see redundant history.

You can use cglib to generate code at runtime.

Iterating through the fields and defining someMethod is a pretty vague problem statement, so it's hard to give you a very useful answer, but Eclipse's refactoring support provides some excellent tools. It'll give you constructors which initialize a selected set of the defined members, and it'll also define a toString method for you.
I don't know what other someMethod()'s you'd want to consider, but there's a start for you.

I'd be very wary of injecting generated code into files containing hand-written code. Hand-written code should be checked into revision control, but generated code should not be; the code generation should be done as part of the build process. You'd have to structure your build process so that for each file you make a temporary copy, inject the generated source code into it, and compile the result, without touching the original source file that the developers work on.

Antlr is really a great tool that can be used very easily for transforming Java source code to Java source code.

Programmatically inspect .class files

I'm working on a project where we're doing a lot of remote object transfer between a Java service and clients written in other various languages. Given our current constraints I've decided to see what it would take to generate code based on an existing Java class. Basically I need to take a .class file (or a collection of them) parse the bytecode to determine all of the data members and perhaps getters/setters and then write something that can output code in a different language to create a class with the same structure.
I'm not looking for standard decompilers such as JAD. I need to be able to take a .class file and create an object model of its data members and methods. Is this possible at all?

I've used BCEL and find it really quite awkward. ASM is much better. It very extensively uses visitors (which can be a little confusing) and does not create an object model. Not creating an object model turns out to be a bonus, as any model you do want to create is unlikely to look like a literal interpretation of all the data.

I have used BCEL in the past and it was pretty easy to use. It was a few years ago so there may be something better now.
Apache Jakarta BCEL

From your description, it sounds like simple reflection would suffice. You can discover all of the static structure of the class, as well as accessing the fields of a particular instance.
I would only move on to BCEL if you are trying to translate method instructions. (And if that's what you're trying to automate, good luck!)

I'm shocked that no one has mentioned ASM yet. It's the best bytecode library your money can buy. Well, ok it's free.

JAD is a java decompiler that doesn't allow programmatic access. It isn't readily available anymore, and probably won't work for newer projects with Java7 bytecodes.

I think javassist might help you too.
http://www.jboss.org/javassist/
I have never had the need of using it, but if you give it a try, would you let us know your comments about it?
Although I think it is more for bytecode manipulation than .class inspection.