I'm trying to implement receive exit in java as proposed here:
http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.0.1/com.ibm.mq.csqzaw.doc/jm11171_.htm
// This method implements the receive exit interface
public ByteBuffer channelReceiveExit(MQCXP channelExitParms,
MQCD channelDefinition,
ByteBuffer agentBuffer)
{
// Complete the body of the receive exit here
}
I see that exit method is called, and following documentation I expect agentBuffer to contain data of incoming message. But agentBuffer appears to be null everytime, so that I see no way to affect the message data.
Any clue would be appreciated.
UPD
I was wrong about agentBuffer is always null, it is not. Following T.Rob answer I managed to catch several calls with MQCXP.ExitReason = 14 (MQXR_XMIT).
With one of these calls I receive message data in buffer, and it looks possible to modify such data.
But still, there is a problem with other calls, having the same ExitReason, but not corresponding to any real message transmission. I need to skip such calls, but yet have no idea how to distinguish them.
Any idea how can I do it?
There are several kinds of channel exit in MQ. The MCA channels (QMgr-to-QMgr) make an exit point available when the entire message is in the buffer. This exit point invokes the Message Exit. All channels, MCA and MQI (client), make Send/Receive exit points available. The Send/Receive exit points are exercised any time that a transmission from one channel agent to another is invoked.
In the case of client channels, the receive exit may be invoked for flows that do not include message transport, or they may be invoked many times for a single message. It is up to the program to check the MQCXP structure to determine the API call that is flowing over the client channel and whether the agent buffer should or should not contain anything at that point in time.
Note that Send and Receive exits normally work in pairs. Since the queue manager does not invoke Java programs from the MCA, any corresponding Send or Receive exit on the queue manager's side would be written in C.
See the manual page on Channel send and receive exit programs for more info.
Coding an exit in MQ is extremely difficult and requires an indepth knowledge of MQ. A SVRCONN/CLNTCONN channel pair has bi-directional traffic flow on it and you do NOT see the whole message (unless it is less than 32758 bytes). What you see in a channel send/receive exit are Transmission Segment Header (TSH) messages. Note: TSH messages are NOT documented by IBM!!!
The MCA (Message Channel Agent) invokes a channel send/receive exit for various 'ExitReason' defined by MQXR_***.
But agentBuffer appears to be null everytime, so that I see no way to
affect the message data.
That is because you do not understand MQ nor the flow of the bidirectional channel. You do realize that the TSH messages are in binary not plain text!?! Right? Are you dumping/printing the buffer in HEX or doing System.out.println?? If the latter then see my first sentence in this post.
Use a debugger and view the TSH data. Please don't ask why you are looking at weird stuff as I said, TSH messages are NOT documented by IBM! If you want information then go look at the code for Wire Shark as they have reverse engineered the TSH messages.
Finally, why are you writing a channel send/receive exit as there are many other ways to accomplish looking at message flow than using an exit.
Related
I am using grpc-streaming in java. I have a long-lasting open stream where the client and server communicate simultaneously. When I call onNext to send a message, grpc buffers the message internally and will send it on the wire async'ly. Now, if the stream is lost in the middle of sending data, onError is called. I wonder what are the right practices:
to find out which messages were sent successfully
how to retry unsent messages
Currently, I am thinking of implementing an "ack" mechanism in the application layer where for every x items received, the receiver sends back an ack message. Then in order to implement retries, I need to buffer items on the sender side and only remove them from the buffer when the ack is received. Also, on the receiver side, I need to implement a mechanism to ignore duplicate items received.
Example:
Suppose we send an ack for every 100 items sent. We receive ack on batch 3 (200-300) and then we receive an error while sending items 300-400. we try again to send items 300-400 but the client has successfully received 300-330 and it is going to receive them again. so, the client needs to ignore the first 30 items.
It is possible to implement this in the application layer. However, I am wondering if there are better practices/frameworks out there that solve this problem.
The term often used is guaranteed delivery to describe delivery data from one place to another without loss.
Your use case is similar to trying to provide guaranteed delivery over best effort delivery transport layers like UDP. The usual approach is to acknowledge every packet, although you could devise a scheme to check at a higher level as you suggest.
You also usually want to use some form of sliding window which means you don't have to wait for the previous ack before sending the next packet - this helps avoid delays.
There is a very good overview of this approach on UDP in this answer: https://stackoverflow.com/a/15630015/334402
For your case, you will receive a response for your RPC calls which will effectively be the ack - using a sliding window would allow you make th next call before you have received the ack from the previous one.
Your duplicate delivery example is also quite common - one common way to avoid double counting or getting confused is to have packet numbers and simply discard any duplicated packets.
How can I be sure that data is successfully delivered to the other end in socket programming?
outStream.write() doesn't guarantee that bytes are received on the other end. I can force server to send back some confirmation data, but how long should client wait for it? If I wait too short, maybe data is delivered to the server just when I throw timeout exception in client (which then shows error dialog, but server actually received data). On the other hand, I don't want to wait too much.
Should client wait some time and if confirmation is received, a third "commit" message is sent to server which then supplies data for further processing (so first client writes, then server replies and then client confirms). But then again, if the commit message is not received on server, client thinks that data is successfully sent but server will ignore it after some time, because it didn't receive commit message. And so on, bouncing never ends...
How is this situation generally handled?
Every tutorial that I read is just about creating/closing sockets, and sending data on client side and receiving it on server side.
If you have links to blogs which explain this problem (or even books), that would be good too.
[EDIT]
I should clarify some things. I'm using Java for client and server, and later I will create C# client. Everything is working perfectly for now. Both client and server are on the same LAN and I have never had any real problems. Scenario explained above is just theoretical, because I would like to cover as much as possible, including error handling.
I know TCP guarantees delivery, but in Java, out.write() doesn't block until underlying TCP delivers or fails and then continues execution or throws an exception. It just continues execution and I don't know if sending failed or not. There is no callback function. I'm starting with socket programming so maybe there is very simple solution which I don't know about. All I need to do is to make sure client knows that server received the message (if that is even possible).
If you have this kind of extreme need for reliability, you need to build that into your application and protocol. One way I have done that in the past is as follows.
Say you have a stream of "objects" (objects here defined in whatever way makes sense to your application) that need to be communicated from client C to server S. Associate a unique identifier with each object on the client side. Then have C send each object along with its identifier to S. But have C keep its copy of the object for now (in memory, or on disk, or whatever makes sense).
For each object S receives, it stores the object together with its unique identifier in its own local data store, and sends back an acknowledgment to C that it received the object (using the identifier to communicate that). C can now delete that object from its data store (strictly speaking it can delete all the ones it sent prior to that object as well -- since TCP guarantees sequenced delivery -- but that slightly complicates things).
This process can continue indefinitely and C never needs to explicitly wait for a confirmation for any one object. It simply maintains a local copy of each object. As long as the connection stays up, S will continually acknowledge every object it has received.
If the connection is broken for any reason, C assumes that S has not received any object it sent since the most recently received acknowledgment. When the connection is re-established, C may therefore resend a few objects that S previously received but since S stored the unique identifier along with each object, it simply acknowledges again that it received the object.
If S hangs for some reason, then eventually buffers between client and server will fill up and C's send will block. The client may need to be prepared for this eventuality.
At the end of the stream of objects -- if there is an end -- C will need to wait for the last object to be acknowledged. There's simply no way around that, and so you will need to decide how long it's appropriate to wait before C gives up and declares an error.
(Of course, this is all essentially duplicating at the application layer what TCP is doing at the transport layer: acknowledging what was actually received with the ability for the sender to re-transmit anything that was lost.)
TCP:
TCP guarantees packet delivery at layer 4 of the OSI Model. TCP is based on a handshake in which the receiving party must confirm the packet's delivery. In that case there is either something wrong in your code or your network is malfunctioning. If you are talking about the packet not making it to its destination, make sure you have properly bound the TCP server to the port, and that the destination is correct. While waiting for a packets arrival, make sure you have a receive timeout in place in order to prevent you application from getting hung on the receive.
i'm working on an Java client-server application.
The client send a message sequence (the messages can be different types, i,ve got header), and listens for the replies. I've got 2 thread, one for the transmission and one for the receipts.
So i need to handle the replies, in case of errors or in case of the replies doesn't arrive, for example i can try to send the message another time.
My question is.. is there any java patterns that can helps me?
i would like to handle the send and the relative repliy like a single transaction, but note that i don't need to have a synchronous communication. I send all the message in the sequence in the TX thread and wait for the replies on the RX thread.
I've thought to the mediator Pattern, but i don't know if it is the right way.
Thanks
If the question is purely about transmission protocol I would take a look at NAK. http://en.wikipedia.org/wiki/NAK_(protocol_message)
I have implemented a protocol I made up called JCast that sends files over multicast. The files are broken down into small fragments that are numbered. The receiving clients then respond back with any missing fragments that it did not get (these are the NAK's). The server then would resend only the NAK'd fragments.
EDIT: The benefit of NAK over ACK is that the server can send all the packets it needs to without having to wait for ACK's. Since networks are very much improved nowadays, most of the packets would arrive. The few that do not arrive would then just be resent.
I am making an application that will work much like a real time chat. A user will be constantly writing on lets say a text area and messages will be send to other users. On the communications class I have set up a receiver. When a message from someone reaches the client, the receive method will be invoked and will get the message. What I can't understand is how the code will be executed. What happens if, while the user is typing/sending a message the receive message is invoked ? What do I need to do in order for this to work properly ?
Hope the question is clear enough.
ps : Im still in the design phase thats why I haven't tested it to see what happens.
Also atm I only use a second thread to receive messages which calls the receive method.
There should not be a problem at all.
When a message from someone reaches the client, the receive method
will be invoked and will get the message. What I can't understand is
how the code will be executed?
You should have a Receiver class that will encapsulate a socket (from which your receive data) and keep a set of listeners (see Observer pattern). A GUI can be one of the listeners. When a message is received via the socket, you need to notify all listeners by forwarding the data received. This way, you have a clean and nice way to notify the GUI about new messages arrivals.
What happens if, while the user is typing/sending a message the
receive message is invoked ?
This depends on the type of IP protocol you are using but in general your don't have to worry about this although I suggest you protect your sockets using lock mechanisms.
What do I need to do in order for this to work properly ?
Here is a nice example that can give you some inspiration :)
EDIT: As for your question regarding execution flow, sending and receiving are two different and uncorrelated operations that can happen at the same time. This can be achieved by implementing send and receive operations in two different threads. Here is an article on socket communications and multithreading.
You should either do what traditional Java EE app servers have done, which is assign a separate thread for processing each incoming message, or try a Java NIO solution along the lines of Netty.
I'm making my own custom server software for a game in Java (the game and original server software were written with Java). There isn't any protocol documentation available, so I am having to read the packets with Wireshark.
While a client is connecting the server sends it the level file in Gzip format. At about 94 packets into sending the level, my server crashes the client with an ArrayIndexOutOfBoundsException. According to the capture file from the original server, it sends a TCP Window Update at about that point. What is a TCP Window Update, and how would I send one using a SocketChannel?
TCP windows are used for flow control between the peers on a connection. With each ACK packet, a host will send a "window size" field. This field says how many bytes of data that host can receive before it's full. The sender is not supposed to send more than that amount of data.
The window might get full if the client isn't receiving data fast enough. In other words, the TCP buffers can fill up while the application is off doing something other than reading from its socket. When that happens, the client would send an ACK packet with the "window full" bit set. At that point, the server is supposed to stop sending data. Any packets sent to a machine with a full window will not be acknowledged. (This will cause a badly behaved sender to retransmit. A well-behaved sender will just buffer the outgoing data. If the buffer on the sending side fills up too, then the sending app will block when it tries to write more data to the socket!)
This is a TCP stall. It can happen for a lot of reasons, but ultimately it just means the sender is transmitting faster than the receiver is reading.
Once the app on the receiving end gets back around to reading from the socket, it will drain some of the buffered data, which frees up some space. The receiver will then send a "window update" packet to tell the sender how much data it can transmit. The sender starts transmitting its buffered data and traffic should flow normally.
Of course, you can get repeated stalls if the receiver is consistently slow.
I've worded this as if the sender and receiver are different, but in reality, both peers are exchanging window updates with every ACK packet, and either side can have its window fill up.
The overall message is that you don't need to send window update packets directly. It would actually be a bad idea to spoof one up.
Regarding the exception you're seeing... it's not likely to be either caused or prevented by the window update packet. However, if the client is not reading fast enough, you might be losing data. In your server, you should check the return value from your Socket.write() calls. It could be less than the number of bytes you're trying to write. This happens if the sender's transmit buffer gets full, which can happen during a TCP stall. You might be losing bytes.
For example, if you're trying to write 8192 bytes with each call to write, but one of the calls returns 5691, then you need to send the remaining 2501 bytes on the next call. Otherwise, the client won't see the remainder of that 8K block and your file will be shorter on the client side than on the server side.
This happens really deep in the TCP/IP stack; in your application (server and client) you don't have to worry about TCP windows. The error must be something else.
TCP WindowUpdate - This indicates that the segment was a pure WindowUpdate segment. A WindowUpdate occurs when the application on the receiving side has consumed already received data from the RX buffer causing the TCP layer to send a WindowUpdate to the other side to indicate that there is now more space available in the buffer. Typically seen after a TCP ZeroWindow condition has occurred. Once the application on the receiver retrieves data from the TCP buffer, thereby freeing up space, the receiver should notify the sender that the TCP ZeroWindow condition no longer exists by sending a TCP WindowUpdate that advertises the current window size.
https://wiki.wireshark.org/TCP_Analyze_Sequence_Numbers
A TCP Window Update has to do with communicating the available buffer size between the sender and the receiver. An ArrayIndexOutOfBoundsException is not the likely cause of this. Most likely is that the code is expecting some kind of data that it is not getting (quite possibly well before this point that it is only now referencing). Without seeing the code and the stack trace, it is really hard to say anything more.
You can dive into this web site http://www.tcpipguide.com/free/index.htm for lots of information on TCP/IP.
Do you get any details with the exception?
It is not likely related to the TCP Window Update packet
(have you seen it repeat exactly for multiple instances?)
More likely related to your processing code that works on the received data.
This is normally just a trigger, not the cause of your problem.
For example, if you use NIO selector, a window update may trigger the wake up of a writing channel. That in turn triggers the faulty logic in your code.
Get a stacktrace and it will show you the root cause.