I need to transfer files fast over the Internet from a Java server to C++ clients, where often many clients would need the same files. I was looking at say transferTo() in Java which sounds like it would be a decently optimized function to send files. However, I'm not sure when I use transferTo() how to best receive that in C++ (i.e. is it just a raw data transfer, how do I determine when the file is over on the client side, etc.). I need this to work on both Windows and Linux. Also, other than transferTo(), would there be some way to be more efficient, especially by taking advantage of the fact that many clients will usually need the same files? I'm not sure how to do say multicast etc. Also, I'm using application-level security rather than a VPN, and on the Java server, encrypting with AES and using MAC digital signing, so I'm also looking for a cross-platform library recommendation to deal with the crypto on the C++ side with minimal pain.
I'm very proficient in C++ but have no previous experience with network programming, so please consider than in any suggestions.
Thanks.
An embedded webserver? http-transfers are efficient enough for you?
The simplest embeddable Java webserver I remember seeing is http://acme.com/java/software/Acme.Serve.Serve.html. We use embedded Jetty 6 in production at work, but that takes more elbow grease.
If your clients doesn't know where to find your webserver in the first place, consider announcing using Zeroconf. http://jmdns.sourceforge.net/
For scalability reasons, Thorbjørns suggestion of using http seems like a very good idea as it would allow you to easily set up http proxies for caching, use standard load balancing tools and so forth.
If you are looking to transfer more than just a blob of data, you might want to have a look at googles protocol buffers. They allow for very easy and fast encoding/decoding on the java and c++ end.
Consider chunking the file and sending via UDP datagram. C++ can re-compile as it receives it. Have you considered implementing/embedding an existing P2P protocol implementation?
If you need effecient transfer to many clients then your bottleneck is the server.
For this please look at the bit-torrent protocol as it distributes the transfer between the clients.
Related
I'd like to somehow get control over active sockets on my computer with java/scala. For example, if a program has stablished a tcp connection, i want to be able to retrieve and listen/write onto this socket.
So, the first question is: is that possible?
And the second: how?
You can do this by writing a custom Socket factory. This is fairly complicated and I don't suggest you do this unless you are trying a to hack a program you have no control over (again a very bad idea)
Instead I suggest you monitor your own usage. When you read/write from a Socket you also keep any information you need. i.e Its your program so change it to do what you need.
I found a java wrapper of lipcap http://jnetpcap.com/ that let me do the sort of things i was after. From the web:
jNetPcap is an open-source java library. It contains:
A Java wrapper for nearly all libpcap library native calls
Decodes captured packets in real-time
Provides a large library of network protocols (core protocols)
Users can easily add their own protocol definitions using java SDK
jNetPcap uses a mixture of native and java implementation for optimum packet decoding performance
What is the best way to send "messages" from PHP script to Java program in real time. PHP script and Java programs are both working at the same work station with OS Windows. Maybe some kind of client/server? The main feature is real time; that's why I don't want to use files.
PS: I'm going to send logger messages (php) and display (java) them at OS system tray tooltip.
PPS: I'm real noob in Java; it will be my first Java program. :)
Thank you.
You could use sockets (probably UDP, but depends on your needs). This way, if in the future you will need to put scripts and Java programs on different machines, you'll be able to do that without modifing the code.
In addition, once you established a communication protocol between client and server, this solution is language independent. So it's easy switch from PHP to another scripting language (the same for Java).
This depend on how heavy weight your application is.
If it is your first program and it is just a little project, a possibility is to open a socket on the server, connect to it with a client and send the data as a string, make your php program the client and java program the server.
Their are things that you can borrow to avoid doing everything on the low level. But they will add weight to your program, for example using a JSON/XML parser to serialize(make the messages into bytes readable on both side) the message instead of using your own format.
Or, use a framework like JAX-RS to quickly and easily (for people familar with it, you may need some time to understand it because it is quite different from writing plain java program) to build a little web service like professionals would do.
Possibilities are:
Send your data as a POST to jsp page.
Make your java code read your php logs.
use queuing systems like RabbitMQ, AciveMQ, Redis etc.
For simplicity use a database table as exchange medium.
It is also easier to debug.
(It is asynchrone, one side, PHP or Java, may be down. Performance is fast, as DB-Server will keep as much in memory.)
I have done some searching but haven't come up with anything on this topic. I was wondering if anyone has ever compared (to some degree) the performance difference between an RPC over a socket and a REST web service. If both do the same thing, which would have a tendency to be the better performer? I've already started building some socket code and would like to know if REST would give better performance before I progress much further. Any input would be really appreciated. Thanks indeed
RMI
Feels like a local API, much like
XMLRPC
Can provide some fairly nice remote
exception data
Java specific means this causes lock
in and limits your options
Has horrible versioning problems
between different versions of clients
Skeleton files must be compiled in
like CORBA, which is not very flexible
REST:
easy to route around firewalls
useful for uploading files as it can
be rather lightweight
very simple if you just want to shove
simple things at something and get
back an integer (like for uploaders)
easy to proxy security behind Apache
and let it take the heat
does not define any standard format
for the way the data is being
exchanged (could be JSON, YAML 1.0,
YAML 2.0, arbitrary XML format, etc)
does not define any convention about
having remote faults sent back to the
caller, integer codes are frequently
used, but method of sending back data
is not defined. Ideally this would be
standardized.
may require a lot of work on the
client side caller of the library to
make use of data (custom serialization
and so forth)
In short from here
web services do allow a loosely
coupled architecture. With RMI, you
have to make sure that the objects
stay in sync in all applications
RMI works best for smaller
applications, that are not
internet-related and thus not scalable
Its hard to imagine that REST is faster than a simple socket connection given it also goes over a Socket.
However REST may be performant enough, standard and easier to use. I would test whether REST is fast enough and meets your requirements first (or one of the many other existing solutions) before attempting your own Socket solution.
I need to communicate between server/client. I saw that CORBA is used for different languages to work like RMI, is it?
In my application I will have to transfer objects between client/server, transfer binary files (which I saw that I can do with RMI) and also play live streaming from one client to another.
I was thinking about CORBA because it also can be used with C++ if I need, isnt it?
So can I play streaming with CORBA?
RMI and CORBA are technologies for distributed objects. You then invoke methods on a remote object the same way as on a local object.
Sure, you can send and receive bytes if you implement methods that do so (e.g. void sendChunk(byte[] data)). But I wouldn't consider them appropriate for streaming. Also for streaming, you should pick something to address the quality of service of the stream -- which RMI or CORBA definitively don't do. For that I would maybe have a look at UDP sockets, or something like that, which just drops packets if the channel is saturated.
CORBA provides you a lot of services and it possibly is not the best of the options for streaming media. Two reasons I can think of (though one can find more reasons against too)
The object payload is more than just the data (marshalling and unmarshalling)
CORBA (specifically the implementations) generally strive for a good QoS aka there will be retries for the same call
That said, it has been demonstrated that ORBs can work with real-time communication too. So, CORBA as a framework is not completely off the table.
I am not sure of the multicast communication capabilities of CORBA though.
If you're hell bent on using CORBA to address this, have a look at RT-CORBA (Real-time corba). I believe TAO has an implementation of it, however I've never used RT-CORBA so I can't speak first-hand if it'll give you the performance you'll require for streaming.
We're come across a problem here at my company and I'm trying to find the best solution.
Software was recently purchased that utilizes a Java program to get the tax for a certain shipment. The site that needs this was written in PHP4. How can I communicate between the two?
It was suggested to use files to communicate but that was horribly slow since the Java program needed to be recompiled every time. So, what is the best solutions to this:
Create a mutli-threaded Java server and use PHP to send/receive the info.
Some other type of file-writing method
Something cool that I dont even know about.
Thanks in advance!
Edit:
I understand the importance of web services but why would this be more efficient that using a mutli-threaded socket-based java server? The only thing connecting to this web services will be my PHP program, no one else. It seems like it might be overkill for my simple task. Am I mistaken? If so, why? Thanks.
Wrap the Java program in a Web Service, and invoke it from PHP. You can even use caching in the Web Service, to optimize performance.
Why not dump the info into a database and have some sort of schedualed job read from it once and a while?
You can always use Quercus which allows you to run PHP in a Tomcat Servlet container.
Web Services is the elegant solution. But in many cases I found much practical to go for a quick-and-dirty solution: start a Java server that communicates using a lightweight communication protocol (none of the heavyweight stuff like XML from Web Services) - example: Apache Thrift. The write a very light client, that takes parameters from command line and writes the output to the console. The client can be in Java or even in other languages, like C++ (Apache Thrift supports that). Then you call the client with system() or with exec() from PHP.
This is not a solution I would ever recommend for production, but it's great for prototyping. Quick and dirty and flexible and extremely modest learning curve (if you already use light-weight communication between your Java processes).
Since you are using PHP4, you may want to just set up a tomcat server that is on a closed network, or just local on the machine of interest, and have it communicate with a servlet, that way you don't have to write a multi-threaded server and deal with creating a communication interface.
If you can upgrade, this page has two other options that may of interest:
http://us3.php.net/manual/en/intro.java.php
Give a look at Quercus
Quercus is Caucho Technology's fast, open-source, 100% Java implementation of the PHP language
I never used it though,
Web Services is the answer. Here's a nice intro link. Your problem is the very reason web services came to the forefront - communication between systems that couldn't ordinarily communicate.
What a web service is essentially going to do is send XML between the PHP and the Java systems. You're going to have to establish an interface for the two, which might be more difficult at the upstart, but you'll reap the benefits later on. In either case, it will be much faster than reading and writing files on the server. Disk I/O are the major bottlenecks on any server.
I may miss something, but if your java program output the needed values, can't you just start the java program from php using exec (http://dk.php.net/manual/en/function.exec.php)
Use the PHP/Java Bridge from sourceforge.net. It is mature, fast and easy to install.