I'm on a project that exposes ABAP functionality as a web service via Netweaver / Java (7.01 SP3 I think). We consume it on an .NET 4 UI tier. We are dealing with some large message structures (12MB serialized XML) that are taking too many seconds to shuttle between the various tiers.
We're tackling this performance on a number of fronts:
Disk, network, CPU and memory are fine and nowhere near saturated.
We're working to trial WCF Streaming mode
We may try gzip compression on the web service's server
And lastly, the point of this question: is there a way to enable binary serialization that's interoperable?
Assuming that you already tried everything to get the payload size down and to split it into smaller pieces (12 MB XML, seriously!), I'd say that depends on the kind of XML processing you need on the ABAP side. You could try to implement your own ICF HTTP handlers and go for some REST-style interface. This is especially interesting if you really want to transfer binary data (for example, some document you retrieve from an archive system) because you can then transfer the document via HTTP without the XML-binary-ugliness. Even if you have to use fill WSDL-y web services, you could try to refactor the binary parts out of that interface, just send some (GU)ID through the web service and have the client fetch the binary part from your custom ICF handler.
Related
I'm have just start learning about micro-services and I have a question that I cannot answer myself. (and I'm also a Java based developer)
I have a situation like this:
I have service A (an API service) that call Thrift services (Named T1) for get data.
Then I have a service B that can use data response from A, parse these data and then generate some new data, finally, return it to client.
The question is: Which I should use?
B call API from A and parse (for example JSON data) with HttpClient/ AsyncHttpClient with connection pool or B direct call T1 and repeat what A do?
IMHO, I think Thrift (with connection pooling too) is faster than HTTP call? Am I right?
I see a lot of services that use HTTP for internal like Elastic search, Neo4j, Eureka Netflix, etc ...
So, which one should I use? And why HTTP is so popular for internal use instead of RPC like Thrift, ProtoBuf, ...?
Sorry for my bad english.
Thank you in advance.
HTTP and JSON or XML are generally used because they're platform and language independent. The HTTP API allows for a ReSTful architecture, which has proven to be a scalable model for developing distributed systems.
Historically, RPC-based approaches to distributed systems have shown a number of weak points:
often they're language dependent. Thrift and Protobuf are more interoperable but they're still dependent on fairly specific 3rd party libraries. In comparison, there are many implementations of HTTP clients and XML or JSON data bindings / processors.
by tying together the client and server upgrades can become difficult - the client often must be upgraded at the same time as the server. In a truly distributed network this can be impossible.
RPC is often not a great metaphor in a distributed system. By abstracting the network to an implementation concern they often encourage low-level 'chatty' interfaces which either involve too much network traffic or are not resilient to unreliable networks.
binary transfer formats are more difficult to analyse / debug when something goes wrong.
For these kinds of reasons people tend to choose Rest-with-HTTP-based APIs over proprietary RPC APIs.
I have a JRuby/Rails app that needs to get data from a system written in Java. I have already worked out how to call Java code from Ruby.
However, lets say that the Client object I need to create, starts threads, communicates with the internal system etc, and that the data is delivered asynchronously in callbacks.
Do I have to write a separate web service (that creates this Client) with a permanent connection to the Java system? In such a way that my Ruby/Rails code can call it synchronously. Or is it possible to write this asynch handler directly in Rails?
When multiple HTTP clients issue their GET's, I would of course have to connect to the Java system when the first client arrives. For the following clients, the data would already be there.
I realize what the proper solution would be, but I'm curious whether I can do it all in Rails.
I can (for now) live without the realtime updating of the webpage, as long as the data in the Java callbacks are stored "somewhere" so that the next HTTP refresh/GET can return it. Nest step would be SSE, Javascript etc.
I think I know how to create the Java web service, however I would rather keep the solution a bit simpler with fewer services.
Thanks
Since you also have access to the java code, I have two approaches to extend the java backend, in order to provide the data you want to use in your ruby frontend application.
Use REST or a HTTP Service
Your ruby webservice could interact with the backend utilizing REST (or any other HTTP approach). This would result in cleaner and more reusable code. Since you're able to access the backend data with any client that is capable of HTML.
Use TCP together with a little protocol
In this approach the clients have to connect to a TCP socket on the backend to send data back and forth. You have to write a little byte or string based protocol which you also have to parse. It's more complex than the first approach, but also more performat and you don't have to rely on external libraries (e.g. jersey, for REST). It also has all the advantages of the former approach, you can serve any client capable of network communication and socket handling.
I am planning to implement RESTful web services to return large XML response (upto 50MB), is it ideal for such requets or SOAP JX-WS better? Do I need to use some other protocol to make it more robust when it comes to marshalling/unmarshalling?
REST uses a regular HTTP get. HTTP get is stable for very large files. Downloading a 50MB file (or other content) is done quite regularly over HTTP.
You only need to make sure that there are not any other delays due to processing in the middle that would cause the connection to time out (usually ~2 minutes). This is unlikely to be a problem.
If you use Restlet you can stream data of any size back to the client using ReadableRepresentation (I'm doing gigabytes). It takes a bit of effort but it works fine.
I'm looking for a full duplex streaming solution with Java EE.
The situation: client applications (JavaFX) read data from a peripheral device. This data needs to be transferred in near real-time to a server for processing and also get the response back asynchronously, all while it keeps sending new data for processing.
Communication with the server needs to have an overhead as low as possible. Data coming in is basically some sensor data and after processing it is turned in what can be described as a set of commands.
What I've looked into:
A TCP/IP server (this is a non-Java EE approach).This would be the obvious solution. Two connections opened in parallel from each client app: one for upstream data and one for downstream data.
Remote & stateless EJBs. This would mean that there's no streaming involved and that I pack sensor data in smaller windows (1-2 seconds worth of sensor data) which I then send to the server for processing and get the processing result as a response. For this approach, while it is scalable, I am not sure how fast it will be considering I have to make a request each 1-2 seconds. I still need to test this but I have my doubts.
RMI. Is this any different than EJBs, technically?
Two servlets (up/down) with long polling. I've not done this before, so it's something to be tested.
For now I would like to test the performance for my approach #2. The first solution will work for sure, but I'm not too fond of having a separate server (next to Tomcat, where I already have something running).
However, meanwhile, it would be worth knowing if there are any other Java specific (EE or not) technologies that could easily solve this. If anyone has an idea, then please share it.
This looks like a good place for using JMS. Instead of stateless EJBs, you will probably be using Message-Driven Beans.
This gives you an approach similar to your first solution, using two message queues instead of TCP/IP connections. JMS makes your communications fully asynchronous and is low-overhead in the sense that your clients can send messages as fast as they can regardless of how fast your server can consume them. You also get delivery guarantees and other JMS goodness.
Tomcat does not come with JMS, however. You might try TomEE or integrate your existing Tomcat with a JMS implementation like ActiveMQ.
There are numerous options you could try. Appropriate solutions depend on the nature of your application, communication protocol, data transfer type, control you have over the client and server and firewall restrictions on client server routes.
There's not much info on this in your question, but given what you have provided, you may like to look at netty as it is quite general purpose and flexible and seems to fit your requirements. Netty also includes a duplex websocket implementation. Note that a netty based solution may be more complex to implement and require more background study than some other solutions (such as jms).
Yet another possible solution in GraniteDS, which advertises a JavaFX client integration and multiple server integrations for full duplex client/server communication, though I have not used it. GraniteDS uses comet (your two asynchronous servlets with long polling model) with the Active Message Format for data which you may be familiar with from Flex/Flash.
Have you looked at websockets as a solution? They are known to keep persistent connections and hence the asynchronous response will be quick.
I'm trying to combine these articles: http://java.sun.com/developer/technicalArticles/RMI/rmi_corba/ and http://netbeans.org/kb/docs/javaee/entappclient.html to make simple client-server app using Glassfish, in which I could send a file from (local) client to a directory on the (local) server. This is something new for me and I feel a little overwhelmed at the moment. Any advice, please?
You're kind of in the wrong area. The things you're looking at are for support of RPC sessions. In theory you could send over an enormous byte array, but it is likely unwise to do so.
What would be preferable is to create a simple web app and push the file over HTTP.
Or you could try using a WS Web Service that's configured for MTOM -- it will handle large payloads as well. You can look here for an article of streaming MTOM messages. It's for WebLogic, but it's basically Sun JAX-WS so it should work on Glassfish out of the box.
An advantage of the Web Service is you can host it in an EJB, rather than having to deploy a separate WAR for this facility. What you want to watch out for is having the payload being all stored in RAM. For example, if you want to send a 10Gb file, the actual traffic is going to be the same, but done naively, you will end up holding all 10Gb in the heap on the client and/or the server, which obviously is not desirable.
In the end either way will work. The Web Service had the downside of having to dig in to the shadowy corners of the Web Service stack, where as with a generic Servlet and web app, it's more out in the open, however you will likely be diving in to the inner depths of HTTP to pull that off. For example, if you wanted to use Apache HTTP Client, you would need to create a custom RequestEntity to handle the streaming for you.
All possible, it's just less used and not the default, out of the box, 2 lines of code tutorial example.