We Keep Coding

Developing Networking Security Applications & System utilities in Java

I come from a C/Linux background and don't have much background in Java. I generally develop system administrator utilities like :
disk cleanup
retrieve lost data / files
repairing file systems
disk de-fragmentation
I also develop Network monitoring security applications which help admins monitor :
- their networks,
- scan incoming & outgoing data packets,
- remotely block ports / USBs
- monitor emails with attachments etc
Right now we write code in C for Linux which has to be ported to windows but such a problem will not exist in Java.
My questions are :
Is Java the right language for writing these applications & utilities (as mentioned above)?
I understand Java will provide Libraries and classes to access system resources / network / sockets but will Java abstraction be a hindrance at some point (which would restrict the flexibility which C/C++ provide )?
If for example I want to write a utility to repair a file system / or retrieve data for Windows & Unix ...will I be using same API for both OS or there are different API for different OS?
I am not concerned about the speed / execution trade off since none of my applications have to make real time decisions as in the gaming industry.

Java is the right language if you want portability. You can do almost everything you can do with C/C++ and you can utilize patterns and libraries that help you create great maintainable designs. In case there is something very low level you cannot do with Java, you always can create your own native code that is loaded with Java Native Interface. Thus the only non-portable code you will have will be these native-code libraries.

Right now we write code in C for Linux which has to be ported to
windows but such a problem will not exist in Java.
Java can abstract away only so much since in the end, low level stuff always boils down to making system calls, which are different between OSes.
As long as you're working with pure java logic, or simple operating system utilities, you'll be golden. You want to open a TCP socket and connect to google.com? No problem. You want to open a file in a known location, read some lines, process them, and write the results to a different file? No problem, Java has you covered.
But, if you want to do more low-level stuff with Java, you'll run into trouble pretty soon. You want to open a raw socket and send a TCP packet? You can't, windows doesn't allow that. You want to get a file's creation time on Linux? You can't, Linux doesn't keep that information. Java's BasicFileAttributes.creationTime() will return a file's modification time on Linux. You want to get a nanosecond resolution timestamp? Well, you can, but only on some OSes. Or say you want to get the computer's hostname without resorting to a network lookup (which depends on a network being actually available), well, get ready for some hacking (this is my own answer by the way).
Now, to your more specific questions:
Is Java the right language for writing these applications & utilities (as mentioned above)?
I frankly don't know. I never tried defragmenting or restoring a file programmatically from Java. But since it involves some very low level filesystem operations, I suggest you do some serious reading before moving to Java. Check whether the APIs you need exist in the language itself or in some good libraries.
I understand Java will provide Libraries and classes to access system
resources / network / sockets but will Java abstraction be a hindrance
at some point (which would restrict the flexibility which C/C++
provide )?
Yes. For instance, it's impossible to open a raw socket using pure Java. And if I recall correctly, it's also impossible to set some socket options.
If for example I want to write a utility to repair a file system / or
retrieve data for Windows & Unix ...will I be using same API for both
OS or there are different API for different OS?
I never tried repairing a file system in Java, so I can't tell you about the APIs involved. But I find it hard to believe you'll find a pure Java api for doing low level stuff with the file system. You'll probably have to write your own native code (and run it through JNI) or use someone else's library (which probably uses JNI, like the raw socket library I mentioned earlier).

Is it possible for node.js to import a java library

I have a nodejs application that has some expensive computations. I'm thinking of doing this part in java so I can more easily take advantage of threading and math libraries. Is there an easy way to have nodejs talk to external java libraries?
The java library will contain a loop that frequently calls javascript functions. Will I see a big performance hit due to having these two libraries constantly cross talk (rather than packaging the entire task, sending it to the jvm, and then getting a result back)

It may be better to just create a java server to do the computations and communicate with your node.js application over a messaging queue. Here is an example which shows how to do that - http://blog.james-carr.org/2010/09/09/rabbitmq-nodejs-and-java-goodness/

You might want to take a look a Vert.X, which will let you mix and match JavaScript and Java as you see fit and communicate via a local message bus.

Get access of active socket

I'd like to somehow get control over active sockets on my computer with java/scala. For example, if a program has stablished a tcp connection, i want to be able to retrieve and listen/write onto this socket.
So, the first question is: is that possible?
And the second: how?

You can do this by writing a custom Socket factory. This is fairly complicated and I don't suggest you do this unless you are trying a to hack a program you have no control over (again a very bad idea)
Instead I suggest you monitor your own usage. When you read/write from a Socket you also keep any information you need. i.e Its your program so change it to do what you need.

I found a java wrapper of lipcap http://jnetpcap.com/ that let me do the sort of things i was after. From the web:
jNetPcap is an open-source java library. It contains:
A Java wrapper for nearly all libpcap library native calls
Decodes captured packets in real-time
Provides a large library of network protocols (core protocols)
Users can easily add their own protocol definitions using java SDK
jNetPcap uses a mixture of native and java implementation for optimum packet decoding performance

Any better options than porting from C# to Java?

I have an existing library written in C# which wraps a much lower-level TCP/IP API and exposes messages coming down the wire from a server (proprietary binary protocol) as .NET events. I also provide method calls on an object which handles the complexities of marshalling convenient .NET types (like System.DateTime) down to the binary encodings and fixed-length structures that the API requires (for outgoing messages to the server). There are a fair number of existing applications (both internally and used by third parties) built on top of this .NET library.
Recently, we've been approached by someone who doesn't want to do all the legwork of abstracting the TCP/IP themselves, but their environment is strictly non-Windows (I assume *nix, but I'm not 100% sure), and they've intimated that their ideal would be something callable from Java.
What's the best way to support their requirements, without me having to:
Port the code to Java now (including an unmanaged DLL that we currently P/Invoke into for decompression)
Have to maintain two separate code-bases going forwards (i.e. making the same bug-fixes and feature enhancements twice)
One thing I've considered is to re-write most of the core TCP/IP functionality once into something more cross-platform (C / C++) and then change my .NET library to be a thin layer on top of this (P/Invoke?), and then write a similarly thin Java layer on top of it too (JNI?).
Pros:
I mostly spend my time writing things only once.
Cons:
Most of the code would now be unmanaged - not the end of the world, but not ideal from a productivity point of view (for me).
Longer development time (can't port C# sockets code to C / C++ as quickly as just porting to Java) [How true is this?]
At this point, the underlying API is mostly wrapped and the library is very stable, so there's probably not a lot of new development - it might not be that bad to just port the current code to Java and then have to make occasional bug-fixes or expose new fields twice in the future.
Potential instability for my existing client applications while the version they're running on changes drastically underneath them. (Off the top of my head I can think of 32/64 bit issues, endianness issues, and general bugs that may crop up during the port, etc.)
Another option I've briefly considered is somehow rigging Mono up to Java, so that I can leverage all of the existing C# code I already have. I'm not too clued up though on how smooth the developer experience will be for the Java developers who have to consume it though. I'm pretty sure that most of the code should run without trouble under Mono (bar the decompression P/Invoke which should probably just be ported to C# anyway).
I'd ideally not like to add another layer of TCP/IP, pipes, etc. between my code and the client Java app if I can help it (so WCF to Java-side WS-DeathStar is probably out). I've never done any serious development with Java, but I take some pride in the fact that the library is currently a piece of cake for a third-party developer to integrate into his application (as long as he's running .NET of course :)), and I'd like to be able to keep that same ease-of-use for any Java developers who want the same experience.
So if anyone has opinions on the 3 options I've proposed (port to Java & maintain twice, port to C and write thin language bindings for .NET and Java or, try and integrate Java and Mono), or any other suggestions I'd love to hear them.
Thanks
Edit: After speaking directly with the developer at the client (i.e. removal of broken telephone AKA Sales Department) the requirements have changed enough that this question no longer applies very well to my immediate situation. However, I'll leave the question open in the hopes that we can generate some more good suggestions.
In my particular case, the client actually runs Windows machines in addition to Solaris (who doesn't these days?) and is happy for us to write an application (Windows Service) on top of the library and provide a much more simplified and smaller TCP/IP API for them to code against. We will translate their simple messages into the format that the downstream system understands, and translate incoming responses back for them to consume, so that they can continue to interface with this downstream system via their Java application.
Getting back to the original scenario after thinking about this for a couple of weeks, I do have a few more comments:
A portable C-based library with different language bindings on top would probably be the way to go if you knew up front that you'd need to support multiple languages / platforms.
On *nix, can a single process host both a Java runtime and a Mono runtime simultaneously? I know in earlier versions of .NET you couldn't have two different .NET runtimes in the same process, but I believe they've fixed this with .NET 4? If this is possible, how would one communicate between the two? Ideally you'd want something as simple as a static method call and a delegate to raise responses with.
If there's no easy direct interface support between Java & Mono (methods & delegates, etc.), one might consider using something like ZeroMQ with Protocol Buffers or Apache Thrift as the message format. This would work in-process, inter-process and over the network because of ZeroMQ's support for different transports.

Spend more time getting the requirements nailed down before deciding on an implementation. Until you know what is required, you don't have any criteria for choosing between designs.
If it's a non-windows environment, it doesn't make sense to have .NET anywhere in there, for example.

If you need something that runs on the Java Virtual Machine but looks a lot like C#, you should check out Stab. This will not help you with P/Invoke and the like but you may find it less work to port your C# code to Java and maintain it.
You should look into Mono though. I expect that all your C# code would run unmodified (except the parts that touch the unmanaged DLL).
I have not used it but jni4net is supposed to allow calling .NET code from Java. If your clients want a Java interface, this may be a solution.
I use Mono on Linux and the Mac all the time even when .NET compatibility is not a priority. I like C# and the .NET libraries and prefer the CLR to the JVM. Mono is MIT/X11 licensed which means that you can use it commercially if you like. Unlike some others, I see no reason to avoid technology championed by Microsoft while favouring technology championed by Oracle and IBM.
Using Mono will not help you with the unmanaged bits, although you can still P/Invoke into a native DLL. You will just have to port that DLL yourself or find some equivalent.
You may also want to look into Mono Ahead of Time compilation.

Have you considered mono? It would most likely support your existing code in the non-windows environment. The trick would be calling it from java, but the mono folks might have something to help you out there, too.

This probably isn't the right solution in your case, but for completeness:
There are a few languages that can target both the JVM and .NET, in particular Ruby (JRuby and IronRuby) and Python (Jython and IronPython). Scala might eventually get there too, although right now the .NET version is a long way behind the JVM version.
Anyway, you could potentially rewrite your library in Ruby or Python and target both runtimes.

If what you really, really want is to be able to code in .NET and have it run on the JVM, you could check out Grasshopper (2015-09: link possibly dead). That is what it is designed to do.
I know the Mainsoft guys have been contributors to Mono over the years. If I remember correctly, they wrote the Visual Basic compiler for Mono.
There is also the C# to Java converter from Tangible. I have heard good things but I have never used it myself.
Also, it does not help your situation much but I should point out Mono for Android.
Mono for Android runs the CLR and the Dalvik VM in parallel. In other words, the C# code you wrote for Android can be calling into Java libraries (like the Android UI for example) and executing as a single app. You had asked about the ability to run .NET and Java code in the same process. Clearly, it can be done.

One thing I've considered is to re-write most of the core TCP/IP functionality once into something more cross-platform (C / C++) and then change my .NET library to be a thin layer on top of this (P/Invoke?), and then write a similarly thin Java layer on top of it too (JNI?).
That's a possibility. On the Java side, you should consider using JNA rather than JNI. (If you use JNI, the C / C++ code needs to be written to use JNI-specific signatures.)
Another possibility is to replace the proprietary binary protocol with something that "just works" with multiple programming languages. This is the kind of problem space where CORBA and similar technologies provide a good solution.

Efficient file transfer from Java server to multiple C++ clients?

I need to transfer files fast over the Internet from a Java server to C++ clients, where often many clients would need the same files. I was looking at say transferTo() in Java which sounds like it would be a decently optimized function to send files. However, I'm not sure when I use transferTo() how to best receive that in C++ (i.e. is it just a raw data transfer, how do I determine when the file is over on the client side, etc.). I need this to work on both Windows and Linux. Also, other than transferTo(), would there be some way to be more efficient, especially by taking advantage of the fact that many clients will usually need the same files? I'm not sure how to do say multicast etc. Also, I'm using application-level security rather than a VPN, and on the Java server, encrypting with AES and using MAC digital signing, so I'm also looking for a cross-platform library recommendation to deal with the crypto on the C++ side with minimal pain.
I'm very proficient in C++ but have no previous experience with network programming, so please consider than in any suggestions.
Thanks.

An embedded webserver? http-transfers are efficient enough for you?
The simplest embeddable Java webserver I remember seeing is http://acme.com/java/software/Acme.Serve.Serve.html. We use embedded Jetty 6 in production at work, but that takes more elbow grease.
If your clients doesn't know where to find your webserver in the first place, consider announcing using Zeroconf. http://jmdns.sourceforge.net/

For scalability reasons, Thorbjørns suggestion of using http seems like a very good idea as it would allow you to easily set up http proxies for caching, use standard load balancing tools and so forth.
If you are looking to transfer more than just a blob of data, you might want to have a look at googles protocol buffers. They allow for very easy and fast encoding/decoding on the java and c++ end.

Consider chunking the file and sending via UDP datagram. C++ can re-compile as it receives it. Have you considered implementing/embedding an existing P2P protocol implementation?

If you need effecient transfer to many clients then your bottleneck is the server.
For this please look at the bit-torrent protocol as it distributes the transfer between the clients.