I currently have a game, for which I have implemented a client and a server.
I then have the server sending data to the client about it's position, the client sending movement inputs into the server, etc.
The problem is that the CPU skyrockets to 100%. I have directly connected the high usage to the following code, which is in an update() method that is called ten times per second:
try{
sendToClientUDP(("ID:" + String.valueOf(uid)));
sendToClientUDP(("Scale:" + GameServer.scale));
for (Clients cl : GameServer.players){
//sendToClient(("newShip;ID:" + cl.uid).getBytes(), packet.getAddress(), packet.getPort());
sendToClientUDP((("UID:" + cl.uid +";x:" + cl.x)));
sendToClientUDP((("UID:" + cl.uid +";y:" + cl.y)));
sendToClientUDP((("UID:" + cl.uid +";z:" + cl.z)));
sendToClientUDP((("UID:" + cl.uid +";Rotation:" + (cl.rotation))));
cl.sendToClientUDP(new String("newShip;ID:" + uid));
sendToClientUDP(new String("newShip;ID:" + cl.uid));
}
}catch (Exception e){
e.printStackTrace();
}
Removing the code, and the high CPU usage disappears.
Here is my sendToClientUDP() method.
public void sendToClientUDP(String str){
if (!NPC){ //NPC is checking if it is a computer-controlled player.
UDP.sendData(str.getBytes(), ip, port);
}
}
And here is my UDP.sendData() method:
public static void sendData(String data, InetAddress ip, int port) {
sendData(data.getBytes(), ip, port);
}
public static void sendData(byte[] data, InetAddress ip, int port) {
DatagramPacket packet = new DatagramPacket(data, data.length, ip, port);
try {
socket.send(packet);
} catch (IOException e) {
e.printStackTrace();
}
}
Why is so much CPU being used simply by sending UDP packets? And what, if anything, can I do to reduce it?
I suggest you take out or optimise the code which is producing so much CPU, A CPU profiler is the best place to start but these are likely to be causes of CPU consumption.
creating Strings and byte[] are expensive, I would avoid doing those.
creating multiple packets instead of batching them is also expensive.
Creating a new DatagramPacket can be avoided.
I would remove duplication between messages as this adds redundant work you can avoid.
you might consider using a binary format to avoid the translation overhead of convert to/from text.
There is almost never a good time to use new String() it is almost certainly redundant.
EDIT: This is what I had in mind. Instead of sending 5 packets per client, you send just one packet, total. For ten clients you send 1/50 of the packets, reducing the overhead.
import java.io.IOException;
import java.net.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.ArrayList;
import java.util.List;
/**
* Created by peter on 31/07/15.
*/
public class PacketSender {
public static void main(String[] args) throws IOException {
PacketSender ps = new PacketSender(InetAddress.getByName("localhost"), 12345);
List<Client> clients = new ArrayList<>();
for(int i=0;i<10;i++)
clients.add(new Client());
for(int t = 0; t< 3;t++) {
long start = System.nanoTime();
int tests = 100000;
for (int i = 0; i < tests; i++) {
ps.sendData(1234, 1, clients);
}
long time = System.nanoTime() - start;
System.out.printf("Sent %,d messages per second%n", (long) (tests * 1e9 / time));
}
}
final ThreadLocal<ByteBuffer> bufferTL = ThreadLocal.withInitial(() -> ByteBuffer.allocate(8192).order(ByteOrder.nativeOrder()));
final ThreadLocal<DatagramSocket> socketTL;
final ThreadLocal<DatagramPacket> packetTL;
public PacketSender(InetAddress address, int port) {
socketTL = ThreadLocal.withInitial(() -> {
try {
return new DatagramSocket(port, address);
} catch (SocketException e) {
throw new AssertionError(e);
}
});
packetTL = ThreadLocal.withInitial(() -> new DatagramPacket(bufferTL.get().array(), 0, address, port));
}
public void sendData(int uid, int scale, List<Client> clients) throws IOException {
ByteBuffer b = bufferTL.get();
b.clear();
b.putInt(uid);
b.putInt(scale);
b.putInt(clients.size());
for (Client cl : clients) {
b.putInt(cl.x);
b.putInt(cl.y);
b.putInt(cl.z);
b.putInt(cl.rotation);
b.putInt(cl.uid);
}
DatagramPacket dp = packetTL.get();
dp.setData(b.array(), 0, b.position());
socketTL.get().send(dp);
}
static class Client {
int x,y,z,rotation,uid;
}
}
When this performance test runs it prints
Sent 410,118 messages per second
Sent 458,126 messages per second
Sent 459,499 messages per second
Edit: to write/read text you can do the following.
import java.nio.ByteBuffer;
/**
* Created by peter on 09/08/2015.
*/
public enum ByteBuffers {
;
/**
* Writes in ISO-8859-1 encoding. This assumes string up to 127 bytes long.
*
* #param bb to write to
* #param cs to write from
*/
public static void writeText(ByteBuffer bb, CharSequence cs) {
// change to stop bit encoding to have lengths > 127
assert cs.length() < 128;
bb.put((byte) cs.length());
for (int i = 0, len = cs.length(); i < len; i++)
bb.put((byte) cs.charAt(i));
}
public static StringBuilder readText(ByteBuffer bb, StringBuilder sb) {
int len = bb.get();
assert len >= 0;
sb.setLength(0);
for (int i = 0; i < len; i++)
sb.append((char) (bb.get() & 0xFF));
return sb;
}
private static final ThreadLocal<StringBuilder> SB = new ThreadLocal<>() {
#Override
protected Object initialValue() {
return new StringBuilder();
}
};
public static String readText(ByteBuffer bb) {
// TODO use a string pool to reduce String garbage.
return readText(bb, SB.get()).toString();
}
}
If you need something more complicated you should consider using Chronicle-Bytes which I wrote. It has
support for 64-bit memory sizes, including memory mapping 64-bit.
thread safe operation off heap.
UTF-8 encoding of strings.
compressed types such as stop bit encoding.
automatic string pooling to reduce garbage.
deterministic clean up of off heap resources via reference counting.
Related
My code throws an OutOfMemoryError when running the following line:
int numBytes = socketChannel.write(_send_buffer);
where socketChannel is an instance of java.nio.channels.SocketChannel
and _send_buffer is an instance of java.nio.ByteBuffer
The code arrives at this point via a non-blocking selector write operation, and throws this on the first attempt to write when the capacity of _send_buffer is large. I have no issues with the code when _send_buffer is less than 20Mb, but when attempting to test this with larger buffers (e.g. > 100Mb) it fails.
According to the docs for java.nio.channels.SocketChannel.write():
An attempt is made to write up to r bytes to the channel, where r is the number of bytes remaining in the buffer, that is, src.remaining(), at the moment this method is invoked.
Suppose that a byte sequence of length n is written, where 0 <= n <= r. This byte sequence will be transferred from the buffer starting at index p, where p is the buffer's position at the moment this method is invoked; the index of the last byte written will be p + n - 1. Upon return the buffer's position will be equal to p + n; its limit will not have changed.
Unless otherwise specified, a write operation will return only after writing all of the r requested bytes. Some types of channels, depending upon their state, may write only some of the bytes or possibly none at all. A socket channel in non-blocking mode, for example, cannot write any more bytes than are free in the socket's output buffer.
My channels should be setup to be non-blocking, so I would think the write operation should only attempt to write up to the capacity of the socket's output buffer. As I did not previously specify this I tried setting it to 1024 bytes via the setOption method with the SO_SNDBUF option. i.e:
socketChannel.setOption(SO_SNDBUF, 1024);
Though I am still getting the OutOfMemoryError. Here is the full error message:
2021-04-22 11:52:44.260 11591-11733/jp.oist.abcvlib.serverLearning I/.serverLearnin: Clamp target GC heap from 195MB to 192MB
2021-04-22 11:52:44.260 11591-11733/jp.oist.abcvlib.serverLearning I/.serverLearnin: Alloc concurrent copying GC freed 2508(64KB) AllocSpace objects, 0(0B) LOS objects, 10% free, 171MB/192MB, paused 27us total 12.714ms
2021-04-22 11:52:44.261 11591-11733/jp.oist.abcvlib.serverLearning W/.serverLearnin: Throwing OutOfMemoryError "Failed to allocate a 49915610 byte allocation with 21279560 free bytes and 20MB until OOM, target footprint 201326592, growth limit 201326592" (VmSize 5585608 kB)
2021-04-22 11:52:44.261 11591-11733/jp.oist.abcvlib.serverLearning I/.serverLearnin: Starting a blocking GC Alloc
2021-04-22 11:52:44.261 11591-11733/jp.oist.abcvlib.serverLearning I/.serverLearnin: Starting a blocking GC Alloc
Now I can inline debug and stop at the write line and nothing crashes, so I believe there is no problem handling the memory requirement for the _send_buffer itself, but when attempting to write, something in the background is creating another allocation that's too much to handle.
Maybe I'm thinking about this wrong, and need to limit my _send_buffer size to something smaller, but I'd think there should be a way to limit the allocation made by the write command no? Or at least some way to allocate more of the Android memory to my app. I'm using a Pixel 3a, which according to the specs it should have 4GB of RAM. Now I realize that has to be shared with the rest of the system, but this is a bare bones test device (no games, personal apps, etc. are installed) so I'd assume I should have access to a fairly large chunk of that 4GB. As I'm crashing with a growth limit of 201,326,592 (according to the logcat above), it seems strange to me that I'm crashing at 0.2 / 4.0 = 5% of the spec'd memory.
Any tips in the right direction about a fundamental flaw in my approach, or recommendations for avoiding the OutOfMemoryError would be much appreciated!
Edit 1:
Adding some code context as requested by comments. Note this is not a runnable example as the code base is quite large and I am not allowed to share it all due to company policies. Just note that the _send_buffer is has nothing to do with the sendbuffer of the socketChannel itself (i.e. what is referenced by getSendBufferSize, it is just a ByteBuffer that I use to bundle together everything before sending it via the channel. As I can't share all the code related to generating the contents of _send_buffer just note it is a ByteBuffer than can be very large (> 100Mb). If this is fundamentally a problem, then please point this out and why.
So with the above in mind, the NIO related code is pasted below. Note this is very prototype alpha code, so I apologize for the overload of comments and log statements.
SocketConnectionManager.java
(Essentially a Runnable in charge of the Selector)
Note the sendMsgToServer method is overridden (without modification) and called from the main Android activity (not shown). The byte[] episode arg is what gets wrapped into a ByteBuffer within SocketMessage.java (next section) which later gets put into the _send_buffer instance within the write method of SocketMessage.java.
package jp.oist.abcvlib.util;
import android.util.Log;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.SocketOption;
import java.nio.channels.CancelledKeyException;
import java.nio.channels.ClosedSelectorException;
import java.nio.channels.IllegalBlockingModeException;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.util.Set;
import static java.net.StandardSocketOptions.SO_SNDBUF;
public class SocketConnectionManager implements Runnable{
private SocketChannel sc;
private Selector selector;
private SocketListener socketListener;
private final String TAG = "SocketConnectionManager";
private SocketMessage socketMessage;
private final String serverIp;
private final int serverPort;
public SocketConnectionManager(SocketListener socketListener, String serverIp, int serverPort){
this.socketListener = socketListener;
this.serverIp = serverIp;
this.serverPort = serverPort;
}
#Override
public void run() {
try {
selector = Selector.open();
start_connection(serverIp, serverPort);
do {
int eventCount = selector.select(0);
Set<SelectionKey> events = selector.selectedKeys(); // events is int representing how many keys have changed state
if (eventCount != 0){
Set<SelectionKey> selectedKeys = selector.selectedKeys();
for (SelectionKey selectedKey : selectedKeys){
try{
SocketMessage socketMessage = (SocketMessage) selectedKey.attachment();
socketMessage.process_events(selectedKey);
}catch (ClassCastException e){
Log.e(TAG,"Error", e);
Log.e(TAG, "selectedKey attachment not a SocketMessage type");
}
}
}
} while (selector.isOpen()); //todo remember to close the selector somewhere
} catch (IOException e) {
Log.e(TAG,"Error", e);
}
}
private void start_connection(String serverIp, int serverPort){
try {
InetSocketAddress inetSocketAddress = new InetSocketAddress(serverIp, serverPort);
sc = SocketChannel.open();
sc.configureBlocking(false);
sc.setOption(SO_SNDBUF, 1024);
socketMessage = new SocketMessage(socketListener, sc, selector);
Log.v(TAG, "registering with selector to connect");
int ops = SelectionKey.OP_CONNECT;
sc.register(selector, ops, socketMessage);
Log.d(TAG, "Initializing connection with " + inetSocketAddress);
boolean connected = sc.connect(inetSocketAddress);
Log.v(TAG, "socketChannel.isConnected ? : " + sc.isConnected());
} catch (IOException | ClosedSelectorException | IllegalBlockingModeException
| CancelledKeyException | IllegalArgumentException e) {
Log.e(TAG, "Initial socket connect and registration:", e);
}
}
public void sendMsgToServer(byte[] episode){
boolean writeSuccess = socketMessage.addEpisodeToWriteBuffer(episode);
}
/**
* Should be called prior to exiting app to ensure zombie threads don't remain in memory.
*/
public void close(){
try {
Log.v(TAG, "Closing connection: " + sc.getRemoteAddress());
selector.close();
sc.close();
} catch (IOException e) {
Log.e(TAG,"Error", e);
}
}
}
SocketMessage.java
This is greatly inspired from the example Python code given here, in particular the libclient.py and app-client.py. This is because the server is running python code and clients are running Java. So if you want the reasoning behind why things are the way they are, reference the RealPython socket tutorial. I essentially used the app-server.py as a template for my code, and translated (with modifications) to Java for the clients.
package jp.oist.abcvlib.util;
import android.util.Log;
import org.json.JSONException;
import org.json.JSONObject;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.channels.ClosedChannelException;
import java.nio.channels.SelectionKey;
import java.nio.channels.Selector;
import java.nio.channels.SocketChannel;
import java.nio.charset.StandardCharsets;
import java.text.DecimalFormat;
import java.util.Vector;
public class SocketMessage {
private final SocketChannel sc;
private final Selector selector;
private final ByteBuffer _recv_buffer;
private ByteBuffer _send_buffer;
private int _jsonheader_len = 0;
private JSONObject jsonHeaderRead; // Will tell Java at which points in msgContent each model lies (e.g. model1 is from 0 to 1018, model2 is from 1019 to 2034, etc.)
private byte[] jsonHeaderBytes;
private ByteBuffer msgContent; // Should contain ALL model files. Parse to individual files after reading
private final Vector<ByteBuffer> writeBufferVector = new Vector<>(); // List of episodes
private final String TAG = "SocketConnectionManager";
private JSONObject jsonHeaderWrite;
private boolean msgReadComplete = false;
private SocketListener socketListener;
private long socketWriteTimeStart;
private long socketReadTimeStart;
public SocketMessage(SocketListener socketListener, SocketChannel sc, Selector selector){
this.socketListener = socketListener;
this.sc = sc;
this.selector = selector;
this._recv_buffer = ByteBuffer.allocate(1024);
this._send_buffer = ByteBuffer.allocate(1024);
}
public void process_events(SelectionKey selectionKey){
SocketChannel sc = (SocketChannel) selectionKey.channel();
// Log.i(TAG, "process_events");
try{
if (selectionKey.isConnectable()){
sc.finishConnect();
Log.d(TAG, "Finished connecting to " + ((SocketChannel) selectionKey.channel()).getRemoteAddress());
Log.v(TAG, "socketChannel.isConnected ? : " + sc.isConnected());
}
if (selectionKey.isWritable()){
// Log.i(TAG, "write event");
write(selectionKey);
}
if (selectionKey.isReadable()){
// Log.i(TAG, "read event");
read(selectionKey);
// int ops = SelectionKey.OP_WRITE;
// sc.register(selectionKey.selector(), ops, selectionKey.attachment());
}
} catch (ClassCastException | IOException | JSONException e){
Log.e(TAG,"Error", e);
}
}
private void read(SelectionKey selectionKey) throws IOException, JSONException {
SocketChannel socketChannel = (SocketChannel) selectionKey.channel();
while(!msgReadComplete){
// At this point the _recv_buffer should have been cleared (pointer 0 limit=cap, no mark)
int bitsRead = socketChannel.read(_recv_buffer);
if (bitsRead > 0 || _recv_buffer.position() > 0){
if (bitsRead > 0){
// Log.v(TAG, "Read " + bitsRead + " bytes from " + socketChannel.getRemoteAddress());
}
// If you have not determined the length of the header via the 2 byte short protoheader,
// try to determine it, though there is no gaurantee it will have enough bytes. So it may
// pass through this if statement multiple times. Only after it has been read will
// _jsonheader_len have a non-zero length;
if (this._jsonheader_len == 0){
socketReadTimeStart = System.nanoTime();
process_protoheader();
}
// _jsonheader_len will only be larger than 0 if set properly (finished being set).
// jsonHeaderRead will be null until the buffer gathering it has filled and converted it to
// a JSONobject.
else if (this.jsonHeaderRead == null){
process_jsonheader();
}
else if (!msgReadComplete){
process_msgContent(selectionKey);
} else {
Log.e(TAG, "bitsRead but don't know what to do with them");
}
}
}
}
private void write(SelectionKey selectionKey) throws IOException, JSONException {
if (!writeBufferVector.isEmpty()){
SocketChannel socketChannel = (SocketChannel) selectionKey.channel();
Log.v(TAG, "writeBufferVector contains data");
if (jsonHeaderWrite == null){
int numBytesToWrite = writeBufferVector.get(0).limit();
// Create JSONHeader containing length of episode in Bytes
Log.v(TAG, "generating jsonheader");
jsonHeaderWrite = generate_jsonheader(numBytesToWrite);
byte[] jsonBytes = jsonHeaderWrite.toString().getBytes(StandardCharsets.UTF_8);
// Encode length of JSONHeader to first two bytes and write to socketChannel
int jsonLength = jsonBytes.length;
// Add up length of protoHeader, JSONheader and episode bytes
int totalNumBytesToWrite = Integer.BYTES + jsonLength + numBytesToWrite;
// Create new buffer that compiles protoHeader, JsonHeader, and Episode
_send_buffer = ByteBuffer.allocate(totalNumBytesToWrite);
Log.v(TAG, "Assembling _send_buffer");
// Assemble all bytes and flip to prepare to read
_send_buffer.putInt(jsonLength);
_send_buffer.put(jsonBytes);
_send_buffer.put(writeBufferVector.get(0));
_send_buffer.flip();
Log.d(TAG, "Writing to server ...");
// Write Bytes to socketChannel //todo shouldn't be while as should be non-blocking
if (_send_buffer.remaining() > 0){
int numBytes = socketChannel.write(_send_buffer); // todo memory dump error here!
int percentDone = (int) Math.ceil((((double) _send_buffer.limit() - (double) _send_buffer.remaining())
/ (double) _send_buffer.limit()) * 100);
int total = _send_buffer.limit() / 1000000;
// Log.d(TAG, "Sent " + percentDone + "% of " + total + "Mb to " + socketChannel.getRemoteAddress());
}
} else{
// Write Bytes to socketChannel
if (_send_buffer.remaining() > 0){
socketChannel.write(_send_buffer);
}
}
if (_send_buffer.remaining() == 0){
int total = _send_buffer.limit() / 1000000;
double timeTaken = (System.nanoTime() - socketWriteTimeStart) * 10e-10;
DecimalFormat df = new DecimalFormat();
df.setMaximumFractionDigits(2);
Log.i(TAG, "Sent " + total + "Mb in " + df.format(timeTaken) + "s");
// Remove episode from buffer so as to not write it again.
writeBufferVector.remove(0);
// Clear sending buffer
_send_buffer.clear();
// make null so as to catch the initial if statement to write a new one.
jsonHeaderWrite = null;
// Set socket to read now that writing has finished.
Log.d(TAG, "Reading from server ...");
int ops = SelectionKey.OP_READ;
sc.register(selectionKey.selector(), ops, selectionKey.attachment());
}
}
}
private JSONObject generate_jsonheader(int numBytesToWrite) throws JSONException {
JSONObject jsonHeader = new JSONObject();
jsonHeader.put("byteorder", ByteOrder.nativeOrder().toString());
jsonHeader.put("content-length", numBytesToWrite);
jsonHeader.put("content-type", "flatbuffer"); // todo Change to flatbuffer later
jsonHeader.put("content-encoding", "flatbuffer"); //Change to flatbuffer later
return jsonHeader;
}
/**
* recv_buffer may contain 0, 1, or several bytes. If it has more than hdrlen, then process
* the first two bytes to obtain the length of the jsonheader. Else exit this function and
* read from the buffer again until it fills past length hdrlen.
*/
private void process_protoheader() {
Log.v(TAG, "processing protoheader");
int hdrlen = 2;
if (_recv_buffer.position() >= hdrlen){
_recv_buffer.flip(); //pos at 0 and limit set to bitsRead
_jsonheader_len = _recv_buffer.getShort(); // Read 2 bytes converts to short and move pos to 2
// allocate new ByteBuffer to store full jsonheader
jsonHeaderBytes = new byte[_jsonheader_len];
_recv_buffer.compact();
Log.v(TAG, "finished processing protoheader");
}
}
/**
* As with the process_protoheader we will check if _recv_buffer contains enough bytes to
* generate the jsonHeader objects, and if not, leave it alone and read more from socket.
*/
private void process_jsonheader() throws JSONException {
Log.v(TAG, "processing jsonheader");
// If you have enough bytes in the _recv_buffer to write out the jsonHeader
if (_jsonheader_len - _recv_buffer.position() < 0){
_recv_buffer.flip();
_recv_buffer.get(jsonHeaderBytes);
// jsonheaderBuffer should now be full and ready to convert to a JSONobject
jsonHeaderRead = new JSONObject(new String(jsonHeaderBytes));
Log.d(TAG, "JSONheader from server: " + jsonHeaderRead.toString());
try{
int msgLength = (int) jsonHeaderRead.get("content-length");
msgContent = ByteBuffer.allocate(msgLength);
}catch (JSONException e) {
Log.e(TAG, "Couldn't get content-length from jsonHeader sent from server", e);
}
}
// Else return to selector and read more bytes into the _recv_buffer
// If there are any bytes left over (part of the msg) then move them to the front of the buffer
// to prepare for another read from the socket
_recv_buffer.compact();
}
/**
* Here a bit different as it may take multiple full _recv_buffers to fill the msgContent.
* So check if msgContent.remaining is larger than 0 and if so, dump everything from _recv_buffer to it
* #param selectionKey : Used to reference the instance and selector
* #throws ClosedChannelException :
*/
private void process_msgContent(SelectionKey selectionKey) throws IOException {
if (msgContent.remaining() > 0){
_recv_buffer.flip(); //pos at 0 and limit set to bitsRead set ready to read
msgContent.put(_recv_buffer);
_recv_buffer.clear();
}
if (msgContent.remaining() == 0){
// msgContent should now be full and ready to convert to a various model files.
socketListener.onServerReadSuccess(jsonHeaderRead, msgContent);
// Clear for next round of communication
_recv_buffer.clear();
_jsonheader_len = 0;
jsonHeaderRead = null;
msgContent.clear();
int totalBytes = msgContent.capacity() / 1000000;
double timeTaken = (System.nanoTime() - socketReadTimeStart) * 10e-10;
DecimalFormat df = new DecimalFormat();
df.setMaximumFractionDigits(2);
Log.i(TAG, "Entire message containing " + totalBytes + "Mb recv'd in " + df.format(timeTaken) + "s");
msgReadComplete = true;
// Set socket to write now that reading has finished.
int ops = SelectionKey.OP_WRITE;
sc.register(selectionKey.selector(), ops, selectionKey.attachment());
}
}
//todo should send this to the mainactivity listener so it can be customized/overridden
private void onNewMessageFromServer(){
// Take info from JSONheader to parse msgContent into individual model files
// After parsing all models notify MainActivity that models have been updated
}
// todo should be able deal with ByteBuffer from FlatBuffer rather than byte[]
public boolean addEpisodeToWriteBuffer(byte[] episode){
boolean success = false;
try{
ByteBuffer bb = ByteBuffer.wrap(episode);
success = writeBufferVector.add(bb);
Log.v(TAG, "Added data to writeBuffer");
int ops = SelectionKey.OP_WRITE;
socketWriteTimeStart = System.nanoTime();
sc.register(selector, ops, this);
// I want this to trigger the selector that this channel is writeReady.
} catch (NullPointerException | ClosedChannelException e){
Log.e(TAG,"Error", e);
Log.e(TAG, "SocketConnectionManager.data not initialized yet");
}
return success;
}
}
Stumbled upon this in the Android Docs, which answers the question of why I get the OutOfMemoryError.
To maintain a functional multi-tasking environment, Android sets a hard limit on the heap size for each app. The exact heap size limit varies between devices based on how much RAM the device has available overall. If your app has reached the heap capacity and tries to allocate more memory, it can receive an OutOfMemoryError.
In some cases, you might want to query the system to determine exactly how much heap space you have available on the current device—for example, to determine how much data is safe to keep in a cache. You can query the system for this figure by calling getMemoryClass(). This method returns an integer indicating the number of megabytes available for your app's heap.
After running the ActivityManager.getMemoryClass method, I see for my Pixel 3a I have a hard limit of 192 MB. As I was trying to allocate just over 200 MB, I hit this limit.
I also checked the ActivityManager.getLargeMemoryClass and see I have a hard limit of 512 MB. So I can set my app to have a "largeHeap", but despite having 4GB of RAM, I have a hard limit of 512 MB I need to work around.
Unless someone else knows any way around this, I'll have to write some logic to piecewise write the episode to file if it goes above a certain point, and piecewise send it over the channel later. This will slow things down a fair bit I guess, so if anyone has an answer that can avoid this, or tell me why this won't slow things down if done properly, then I'm happy to give you the answer. Just posting this as an answer as it does answer my original question, but rather unsatisfactorily.
I am using the Java library at https://jitsi.org/Projects/LibJitsi .
I want to stream H264 video (in this case the video is a desktop/screen stream) over RTP and then render it. I can figure out how to stream it, but not how to render the stream. Given the following code (fully compilable and runnable with the Libjitsi Jars and native libraries), what do I do next to render the video stream into a Swing JFrame or JPanel? Apparently there is some sort of JMF JAWTRenderer or maybe I can use Java Media Framework (JMF), Freedom for Media in Java (FMJ), JavaFX in Swing embedding, or VLC media player Swing embedding with the VLCj library. What is the best (easiest, good performance, bug free, non-deprecated) way to render this RTP video stream into a Java Swing application?
Also, at the very bottom, I have a few more related questions.
import java.io.IOException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.SocketException;
import java.util.List;
import java.util.Map;
import org.jitsi.service.libjitsi.LibJitsi;
import org.jitsi.service.neomedia.DefaultStreamConnector;
import org.jitsi.service.neomedia.MediaDirection;
import org.jitsi.service.neomedia.MediaService;
import org.jitsi.service.neomedia.MediaStream;
import org.jitsi.service.neomedia.MediaStreamTarget;
import org.jitsi.service.neomedia.MediaType;
import org.jitsi.service.neomedia.MediaUseCase;
import org.jitsi.service.neomedia.StreamConnector;
import org.jitsi.service.neomedia.device.MediaDevice;
import org.jitsi.service.neomedia.format.MediaFormat;
/**
* This class streams screen recorded video. It can either send an H264 encoded
* RTP stream or receive one depending on the value of the variable
* isReceivingVideo_.
*/
public class VideoStreamer {
// Set to false if sending video, set to true if receiving video.
private static final boolean isReceivingVideo_ = true;
public final MediaService mediaService_;
private final Map<MediaFormat, Byte> RTP_payload_number_map_;
public static final int LOCAL_BASE_PORT_NUMBER = 15000;
public static final String REMOTE_HOST_IP_ADDRESS = "127.0.0.1";
public static final int REMOTE_BASE_PORT_NUMBER = 10000;
private MediaStream videoMediaStream_;
private final int localBasePort_;
private final InetAddress remoteAddress_;
private final int remoteBasePort_;
/**
* Initializes a new VideoStreamer instance which is to send or receive
* video from a specific host and a specific port.
*
* #param isReceiver - true if this instance of VideoStreamer is receiving a
* video stream, false if it is sending a video stream.
*/
public VideoStreamer(boolean isReceiver) throws IOException {
this.remoteAddress_ = InetAddress.getByName(REMOTE_HOST_IP_ADDRESS);
mediaService_ = LibJitsi.getMediaService();
RTP_payload_number_map_ = mediaService_.getDynamicPayloadTypePreferences();
if (isReceiver) {
this.localBasePort_ = LOCAL_BASE_PORT_NUMBER;
this.remoteBasePort_ = REMOTE_BASE_PORT_NUMBER;
startVideoStream(MediaDirection.RECVONLY);
} else {
// switch the local and remote ports for the transmitter so they hook up with the receiver.
this.localBasePort_ = REMOTE_BASE_PORT_NUMBER;
this.remoteBasePort_ = LOCAL_BASE_PORT_NUMBER;
startVideoStream(MediaDirection.SENDONLY);
}
}
/**
* Initializes the receipt of video, starts it, and tries to record any
* incoming packets.
*
* #param intended_direction either sending or receiving an RTP video
* stream.
*/
public final void startVideoStream(final MediaDirection intended_direction) throws SocketException {
final MediaType video_media_type = MediaType.VIDEO;
final int local_video_port = localBasePort_;
final int remote_video_port = remoteBasePort_;
MediaDevice video_media_device = mediaService_.getDefaultDevice(video_media_type, MediaUseCase.DESKTOP);
final MediaStream video_media_stream = mediaService_.createMediaStream(video_media_device);
video_media_stream.setDirection(intended_direction);
// Obtain the list of formats that are available for a specific video_media_device and pick H264 if availible.
MediaFormat video_format = null;
final List<MediaFormat> supported_video_formats = video_media_device.getSupportedFormats();
for (final MediaFormat availible_video_format : supported_video_formats) {
final String encoding = availible_video_format.getEncoding();
final double clock_rate = availible_video_format.getClockRate();
if (encoding.equals("H264") && clock_rate == 90000) {
video_format = availible_video_format;
}
}
if (video_format == null) {
System.out.println("You do not have the H264 video codec");
System.exit(-1);
}
final byte dynamic_RTP_payload_type_for_H264 = getRTPDynamicPayloadType(video_format);
if (dynamic_RTP_payload_type_for_H264 < 96 || dynamic_RTP_payload_type_for_H264 > 127) {
System.out.println("Invalid RTP payload type number");
System.exit(-1);
}
video_media_stream.addDynamicRTPPayloadType(dynamic_RTP_payload_type_for_H264, video_format);
video_media_stream.setFormat(video_format);
final int local_RTP_video_port = local_video_port + 0;
final int local_RTCP_video_port = local_video_port + 1;
final StreamConnector video_connector = new DefaultStreamConnector(
new DatagramSocket(local_RTP_video_port),
new DatagramSocket(local_RTCP_video_port)
);
video_media_stream.setConnector(video_connector);
final int remote_RTP_video_port = remote_video_port + 0;
final int remote_RTCP_video_port = remote_video_port + 1;
video_media_stream.setTarget(new MediaStreamTarget(
new InetSocketAddress(remoteAddress_, remote_RTP_video_port),
new InetSocketAddress(remoteAddress_, remote_RTCP_video_port))
);
video_media_stream.setName(video_media_type.toString());
this.videoMediaStream_ = video_media_stream;
videoMediaStream_.start();
listenForVideoPackets(video_connector.getDataSocket());
}
public void listenForVideoPackets(final DatagramSocket videoDataSocket) {
new Thread(new Runnable() {
#Override
public void run() {
boolean socket_is_closed = false;
while (!socket_is_closed) {
final byte[] buffer = new byte[5000];
final DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
try {
videoDataSocket.receive(packet);
final byte[] packet_data = new byte[packet.getLength()];
System.arraycopy(packet.getData(), packet.getOffset(), packet_data, 0, packet.getLength());
final StringBuilder string_builder = new StringBuilder();
for (int i = 0; i < ((packet_data.length > 30) ? 30 : packet_data.length); ++i) {
byte b = packet_data[i];
string_builder.append(String.format("%02X ", b));
}
System.out.println("First thirty (or fewer) bytes of packet in hex: " + string_builder.toString());
} catch (SocketException socket_closed) {
System.out.println("Socket is closed");
socket_is_closed = true;
} catch (IOException exception) {
exception.printStackTrace();
}
}
}
}).start();
}
/**
* Checks if the given format exists in the list of formats with listed
* dynamic RTP payload numbers and returns that number.
*
* #param format - format to look up an RTP payload number for
* #return - RTP payload on success or -1 either if payload number cannot be
* found or if payload number is static.
*/
public byte getRTPDynamicPayloadType(final MediaFormat format) {
for (Map.Entry<MediaFormat, Byte> entry : RTP_payload_number_map_.entrySet()) {
final MediaFormat map_format = (MediaFormat) entry.getKey();
final Byte rtp_payload_type = (Byte) entry.getValue();
if (map_format.getClockRate() == format.getClockRate() && map_format.getEncoding().equals(format.getEncoding())) {
return rtp_payload_type;
}
}
return -1;
}
/**
* Close the MediaStream.
*/
public void close() {
try {
this.videoMediaStream_.stop();
} finally {
this.videoMediaStream_.close();
this.videoMediaStream_ = null;
}
}
public static void main(String[] args) throws Exception {
LibJitsi.start();
try {
VideoStreamer rtp_streamer
= new VideoStreamer(isReceivingVideo_);
try {
/*
* Wait for the media to be received and (hopefully) played back.
* Transmits for 1 minute and receives for 30 seconds to allow the
* tranmission to have a delay (if necessary).
*/
final long then = System.currentTimeMillis();
final long waiting_period;
if (isReceivingVideo_) {
waiting_period = 30000;
} else {
waiting_period = 60000;
}
try {
while (System.currentTimeMillis() - then < waiting_period) {
Thread.sleep(1000);
}
} catch (InterruptedException ie) {
}
} finally {
rtp_streamer.close();
}
System.err.println("Exiting VideoStreamer");
} finally {
LibJitsi.stop();
}
}
}
When I run the above code by first linking the Libjitsi jar files (by listing them under "Libraries") and specifying the location of native (.so, .dll) libraries via "-Djava.library.path=/path/to/native/libraries", I first run it with final boolean isReceivingVideo = true, and then I run another instance with final boolean isReceivingVideo = false and then the two instances of this application stream to each other. In addition, I have a function, public void listenForVideoPackets, that prints out the first 30 bytes of each packet in hexadecimal format. When I run it, I get the following hexadecimal byte values:
I am just an undergraduate student, so my networking knowledge is limited. Can someone explain what do all these hex patterns mean? Why is the fourth byte of the RTP packet always increasing (33, 35, 37, 39, etc.)? Why is the first packet only 16 bytes while all the other packets are much longer? What does the first packet mean? Why are the first 12 or so bytes identical across all the packets, except for the fourth byte, which is always increasing? What do these numbers mean and how do I handle this RTP stream?
I found a folder called "PacketPlayer" in one person's Libjitsi examples folder (not the one that comes with the library). Their git might contain some useful hints... https://github.com/Metaswitch/libjitsi/tree/master/src/org/jitsi/examples/PacketPlayer
Note that there is a "VideoContainer" class that may be useful. See https://github.com/jitsi/libjitsi/blob/master/src/org/jitsi/util/swing/VideoContainer.java
Also, the first 12 bytes are the RTP header. Using the header diagram at http://www.siptutorial.net/RTP/header.html and the fact that RTP payload type in the above code is 99, the RTP header above breaks down into something like:
RTP version: 2, padding: 0, extension: 0, CSRC count: 0, [first byte]
marker: 0, payload type: 99, [second byte]
sequence number: -11221 [3rd, 4th byte]
timestamp: 1082411848
SSRC source: -504863636
Oddly enough, the sequence number is definitely NOT increasing by 1 as it should. It is increasing by 2. This might mean that your datagram socket is getting every other packet rather than every packet.
I implemented a TCP client-server model to test my bandwidth with the server through sending number of packets with different sizes and see the RTT then calculate the bandwidth through linear regression,
Here is the server code:
import java.io.*;
import java.net.*;
public class Server implements Runnable {
ServerSocket welcomeSocket;
String clientSentence;
Thread thread;
Socket connectionSocket;
BufferedReader inFromClient;
DataOutputStream outToClient;
public Server() throws IOException {
welcomeSocket = new ServerSocket(6588);
connectionSocket = welcomeSocket.accept();
inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));
outToClient = new DataOutputStream(connectionSocket.getOutputStream());
thread = new Thread(this);
thread.start();
}
#Override
public void run() {
// TODO Auto-generated method stub
while(true)
{
try {
clientSentence = inFromClient.readLine();
if (clientSentence != null) {
System.out.println("Received: " + clientSentence);
outToClient.writeBytes(clientSentence + '\n');
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
public static void main(String[] args) throws IOException {
new Server();
}
}
And this is the method in the Client class that return an array of the RTT by each packet
public int [] getResponseTime() throws UnknownHostException, IOException {
timeArray = new int[sizes.length];
for (int i = 0; i < sizes.length; i++) {
sentence = StringUtils.leftPad("", sizes[i], '*');
long start = System.nanoTime();
outToServer.writeBytes(sentence + '\n');
modifiedSentence = inFromServer.readLine();
long end = System.nanoTime();
System.out.println("FROM SERVER: " + modifiedSentence);
timeArray[i] = (int) (end - start);
simpleReg.addData(timeArray[i]* Math.pow(10, -9), sizes[i] * 2); // each char is 2 bytes
}
return timeArray;
}
when i get the slope it returns me a BW with kilo bytes however they are in the same network and the bandwidth should be much more . What i am doing wrong ?
Are you obliged to use linear regression or could it be a different estimator? I am actually not sure if linear regression is the best approach here. I am curious, do you happen to know any sources that suggest to use it in this kind of situation?
Note, that especially the initial BW measurements are much smaller than the real maximal goodput (due to TCP slow-start), so it is important to use a metric estimation that takes large wrong outliers into account.
In previous work I have used the harmonic mean to monitor the bandwidth over a longer period of time and it worked pretty good (also on links with a large bandwidth). The advantage of the harmonic mean over other means, is that while it is still very easy to compute, it mitigates the impact of large outliers, meaning the estimate is not as easily falsified.
Given a series of bandwidth measurements R_i, where i=0,1,2,..., n-1, the harmonic mean is calculated as:
R_total = (n+1)/((n/R_total) + (1/R_n))
It is also good practice to skip the first few measurement values (depending on how often you measure...), e.g., R_(0..5), since you might have initial bursts due to initial preparations in the different layers and are in the slow-start phase anyways.
Here an example implementation in Java. Even though in this case the measurement is done through a file download, it can be easily applied to your environment too - simply use your echo server instead of the file download:
public class Estimator
{
private static double R; // harmonic mean of all bandwidth measurements
private static int n = 0; // number of measurements
private static int skips = 5; // skip measurements for first 5 socket.read() operations
// size in bytes
// start/end in ns
public static double harmonicMean(long start, long end, double size){
// check if we need to skip this initial value, since it might falsify our estimate
if(skips-- > 0) return 0;
// get current value of R
double curR = (size/(1024*1024))/(double)((end - start)*Math.pow(10, -9));
System.out.println(curR);
if(n == 0) {
// initial value
R = curR;
} else {
// use harmonic mean
R = (n+1)/((n/R)+(1/curR));
}
n++;
return R;
}
public static void main(String[] args)
{
// temporary buffer to hold bytes
byte[] buffer = new byte[1024*1024*10]; // 10MB buffer - just in case ...
Socket socket = null;
try {
// measurement done through file download from server
// prepare request
socket = new Socket("yourserver.com",80);
PrintWriter pw = new PrintWriter(socket.getOutputStream());
InputStream is = socket.getInputStream();
pw.println("GET /test_blob HTTP/1.1"); // a test file, e.g., 1MB big
pw.println("Host: yourserver.com");
pw.println("");
pw.flush();
// prepare measurement
long start,end;
double bytes = 0;
double totalBytes = 0;
start = System.nanoTime();
while((bytes = is.read(buffer)) != -1) {
// socket.read() occurred -> calculate harmonic mean
end = System.nanoTime();
totalBytes += bytes;
harmonicMean(start, end, totalBytes);
}
// clean up
is.close();
pw.close();
}
catch(Exception e){
e.printStackTrace();
}
finally {
if(socket != null) {
try{
socket.close();
}
catch(Exception e){
e.printStackTrace();
}
}
}
System.out.println(R+" MB/s");
}
}
Additionally, for the sake of completeness, as I already mentioned in the comments it is important that the test messages/files are big enough, so TCP reaches the full goodput potential of the link.
Please also note, that this is a simplified way to estimate the bandwidth. In this example we start measuring (taking the first timestamp) from when the request was sent, meaning we include the link propagation and server processing delay, which in return will reduce the overall estimated value. Anyways, since you seem to use a local network, I expect the sum of these delays to be rather small, which means they will not falsify the final estimate too much.
I wrote a small blog post concerning measuring TCP connection metrics inside an application layer. Everything is described in more detail there (though the code examples are in C).
I need your advice on a Java NIO package. I have an issue with delays while sending packets over network. The original code is actually my port of the SFML book source code to Java, but here I'll show you only a minimal working example, where the problem is reproduced. Though this code does contain some pieces from SFML library (actually creating a window and an event loop), I believe this has no impact on the issue.
Here I'll show only parts of the code, full version is available here.
So, the program has two entities: Server and Client. If you start an application in a server mode, then a Server is created, starts to listen for new connections, and a new Client is automatically created and tries to connect to the Server. In client mode only a Client is created and connects to the Server.
The application also creates a new basic GUI window and starts an event loop, where everything happens.
The Client sends packets to the Server. It handles them by just logging the fact of accepting. There are two types of packets the Client can send: periodical packet (with an incremental ID) and an event packet (application reacts to pressing SPACE or M buttons).
Client sends packets:
public void update(Time dt) throws IOException {
if (!isConnected) return;
if (tickClock.getElapsedTime().compareTo(Time.getSeconds(1.f / 20.f)) > 0) {
Packet intervalUpdatePacket = new Packet();
intervalUpdatePacket.append(PacketType.INTERVAL_UPDATE);
intervalUpdatePacket.append(intervalCounter++);
PacketReaderWriter.send(socketChannel, intervalUpdatePacket);
tickClock.restart();
}
}
public void handleEvent(Event event) throws IOException {
if (isConnected && (event.type == Event.Type.KEY_PRESSED)) {
KeyEvent keyEvent = event.asKeyEvent();
if (keyEvent.key == Keyboard.Key.SPACE) {
LOGGER.info("press SPACE");
Packet spacePacket = new Packet();
spacePacket.append(PacketType.SPACE_BUTTON);
PacketReaderWriter.send(socketChannel, spacePacket);
}
if (keyEvent.key == Keyboard.Key.M) {
LOGGER.info("press M");
Packet mPacket = new Packet();
mPacket.append(PacketType.M_BUTTON);
PacketReaderWriter.send(socketChannel, mPacket);
}
}
}
Server accepts packets:
private void handleIncomingPackets() throws IOException {
readSelector.selectNow();
Set<SelectionKey> readKeys = readSelector.selectedKeys();
Iterator<SelectionKey> it = readKeys.iterator();
while (it.hasNext()) {
SelectionKey key = it.next();
it.remove();
SocketChannel channel = (SocketChannel) key.channel();
Packet packet = null;
try {
packet = PacketReaderWriter.receive(channel);
} catch (NothingToReadException e) {
e.printStackTrace();
}
if (packet != null) {
// Interpret packet and react to it
handleIncomingPacket(packet, channel);
}
}
}
private void handleIncomingPacket(Packet packet, SocketChannel channel) {
PacketType packetType = (PacketType) packet.get();
switch (packetType) {
case INTERVAL_UPDATE:
int intervalId = (int) packet.get();
break;
case SPACE_BUTTON:
LOGGER.info("handling SPACE button");
break;
case M_BUTTON:
LOGGER.info("handling M button");
break;
}
}
Here is a PacketReaderWriter object:
package server;
import java.io.*;
import java.nio.ByteBuffer;
import java.nio.channels.SocketChannel;
public class PacketReaderWriter {
private static final int PACKET_SIZE_LENGTH = 4;
private static final ByteBuffer packetSizeReadBuffer = ByteBuffer.allocate(PACKET_SIZE_LENGTH);
private static ByteBuffer clientReadBuffer;
private static byte[] encode(Packet packet) throws IOException {
try (
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos)
) {
oos.writeObject(packet);
return baos.toByteArray();
}
}
private static Packet decode(byte[] encodedPacket) throws IOException, ClassNotFoundException {
try (ObjectInputStream oi = new ObjectInputStream(new ByteArrayInputStream(encodedPacket))) {
return (Packet) oi.readObject();
}
}
public static void send(SocketChannel channel, Packet packet) throws IOException {
byte[] encodedPacket = encode(packet);
ByteBuffer packetSizeBuffer = ByteBuffer.allocate(PACKET_SIZE_LENGTH).putInt(encodedPacket.length);
packetSizeBuffer.flip();
// Send packet size
channel.write(packetSizeBuffer);
// Send packet content
ByteBuffer packetBuffer = ByteBuffer.wrap(encodedPacket);
channel.write(packetBuffer);
}
public static Packet receive(SocketChannel channel) throws IOException, NothingToReadException {
int bytesRead;
// Read packet size
packetSizeReadBuffer.clear();
bytesRead = channel.read(packetSizeReadBuffer);
if (bytesRead == -1) {
channel.close();
throw new NothingToReadException();
}
if (bytesRead == 0) return null;
packetSizeReadBuffer.flip();
int packetSize = packetSizeReadBuffer.getInt();
// Read packet
clientReadBuffer = ByteBuffer.allocate(packetSize);
bytesRead = channel.read(clientReadBuffer);
if (bytesRead == -1) {
channel.close();
throw new NothingToReadException();
}
if (bytesRead == 0) return null;
clientReadBuffer.flip();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
baos.write(clientReadBuffer.array(), 0, bytesRead);
clientReadBuffer.clear();
try {
return decode(baos.toByteArray());
} catch (ClassNotFoundException e) {
e.printStackTrace();
return null;
}
}
}
And here is the problem: I have quite big delays between pressing a button (and sending a corresponding packet from the Client) and accepting this packet on the Server. If I start a new instance of the application in a client mode (just add a new Client in short), the delays become even bigger.
I don’t see any reason why these periodical packets create so much network load that other packets just cannot get through, but maybe I'm just missing something. Here I have to say that I’m not a Java expert, so don’t blame me too much for not seeing something obvious :)
Does anyone have any ideas?
Thanks!
I decided to take a look at the Github repo.
Your Server.run() looks like this.
public void run() {
while (isRunning) {
try {
handleIncomingConnections();
handleIncomingPackets();
} catch (IOException e) {
e.printStackTrace();
}
try {
// Sleep to prevent server from consuming 100% CPU
sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
The sleep(100) will result in approximately 10 calls to handleIncomingPackets() per second. handleIncomingPackets() in turn will select a Client channel and call handleIncomingPacket() on a single received Packet. In total the server will be able to handle 10 Packets/second per Client if I understand it correctly.
The Client on the other hand tries to send 20 packets per second of the type PacketType.INTERVAL_UPDATE. Either the Client must send fewer packets per second or the Server needs to be able to handle more packets per second.
The current sleep(100) means that there will always be a latency of up to around 100ms before the server can respond to a single packet, even in a non-overloaded situation. This might be fine though if you make sure you really read all packets available on the channel instead of just a single one each time.
In summary: the smallest change you'd have to do to improve response times is to decrease the sleep() time. 10 ms would be fine. But I'd also suggest trying to check if there's more than one packet available in each iteration.
Update:
In the c++ file you linked my hunch is that it's reading more than one packet per iteration.
<snip>
while (peer->socket.receive(packet) == sf::Socket::Done)
{
// Interpret packet and react to it
handleIncomingPacket(packet, *peer, detectedTimeout);
</snip>
The while loop will read all available packets. Compared to your Java version where you read a single packet per client per server iteration.
if (packet != null) {
// Interpret packet and react to it
handleIncomingPacket(packet, channel);
}
You need to make sure that you read all available packets the Java version also.
If you just want to convince yourself that the client code sends more packets than the server code can handle it's quickly done by setting the sleep() to 10 ms temporarily.
I have created the normal publishers and subscribers implemented using java , which works as reading the contents by size as 1MB of total size 5MB and published on every 1MB to the subscriber.Data is getting published successfully .Now 'm facing the issue on appending the content to the existing file .Finally i could find only the last 1MB of data in the file.So please let me to know how to solve this issue ? and also i have attached the source code for publisher and subscriber.
Publisher:
public class MessageDataPublisher {
static StringBuffer fileContent;
static RandomAccessFile randomAccessFile ;
public static void main(String[] args) throws IOException {
MessageDataPublisher msgObj=new MessageDataPublisher();
String fileToWrite="test.txt";
msgObj.towriteDDS(fileToWrite);
}
public void towriteDDS(String fileName) throws IOException{
DDSEntityManager mgr=new DDSEntityManager();
String partitionName="PARTICIPANT";
// create Domain Participant
mgr.createParticipant(partitionName);
// create Type
BinaryFileTypeSupport binary=new BinaryFileTypeSupport();
mgr.registerType(binary);
// create Topic
mgr.createTopic("Serials");
// create Publisher
mgr.createPublisher();
// create DataWriter
mgr.createWriter();
// Publish Events
DataWriter dwriter = mgr.getWriter();
BinaryFileDataWriter binaryWriter=BinaryFileDataWriterHelper.narrow(dwriter);
int bufferSize=1024*1024;
File readfile=new File(fileName);
FileInputStream is = new FileInputStream(readfile);
byte[] totalbytes = new byte[is.available()];
is.read(totalbytes);
byte[] readbyte = new byte[bufferSize];
BinaryFile binaryInstance;
int k=0;
for(int i=0;i<totalbytes.length;i++){
readbyte[k]=totalbytes[i];
k++;
if(k>(bufferSize-1)){
binaryInstance=new BinaryFile();
binaryInstance.name="sendpublisher.txt";
binaryInstance.contents=readbyte;
int status = binaryWriter.write(binaryInstance, HANDLE_NIL.value);
ErrorHandler.checkStatus(status, "MsgDataWriter.write");
ErrorHandler.checkStatus(status, "MsgDataWriter.write");
k=0;
}
}
if(k < (bufferSize-1)){
byte[] remaingbyte = new byte[k];
for(int j=0;j<(k-1);j++){
remaingbyte[j]=readbyte[j];
}
binaryInstance=new BinaryFile();
binaryInstance.name="sendpublisher.txt";
binaryInstance.contents=remaingbyte;
int status = binaryWriter.write(binaryInstance, HANDLE_NIL.value);
ErrorHandler.checkStatus(status, "MsgDataWriter.write");
}
is.close();
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
// clean up
mgr.getPublisher().delete_datawriter(binaryWriter);
mgr.deletePublisher();
mgr.deleteTopic();
mgr.deleteParticipant();
}
}
Subscriber:
public class MessageDataSubscriber {
static RandomAccessFile randomAccessFile ;
public static void main(String[] args) throws IOException {
DDSEntityManager mgr = new DDSEntityManager();
String partitionName = "PARTICIPANT";
// create Domain Participant
mgr.createParticipant(partitionName);
// create Type
BinaryFileTypeSupport msgTS = new BinaryFileTypeSupport();
mgr.registerType(msgTS);
// create Topic
mgr.createTopic("Serials");
// create Subscriber
mgr.createSubscriber();
// create DataReader
mgr.createReader();
// Read Events
DataReader dreader = mgr.getReader();
BinaryFileDataReader binaryReader=BinaryFileDataReaderHelper.narrow(dreader);
BinaryFileSeqHolder binaryseq=new BinaryFileSeqHolder();
SampleInfoSeqHolder infoSeq = new SampleInfoSeqHolder();
boolean terminate = false;
int count = 0;
while (!terminate && count < 1500) {
// To run undefinitely
binaryReader.take(binaryseq, infoSeq, 10,
ANY_SAMPLE_STATE.value, ANY_VIEW_STATE.value,ANY_INSTANCE_STATE.value);
for (int i = 0; i < binaryseq.value.length; i++) {
toWrtieXML(binaryseq.value[i].contents);
terminate = true;
}
try
{
Thread.sleep(200);
}
catch(InterruptedException ie)
{
}
++count;
}
binaryReader.return_loan(binaryseq,infoSeq);
// clean up
mgr.getSubscriber().delete_datareader(binaryReader);
mgr.deleteSubscriber();
mgr.deleteTopic();
mgr.deleteParticipant();
}
private static void toWrtieXML(byte[] bytes) throws IOException {
// TODO Auto-generated method stub
File Writefile=new File("samplesubscriber.txt");
if(!Writefile.exists()){
randomAccessFile = new RandomAccessFile(Writefile, "rw");
randomAccessFile.write(bytes, 0, bytes.length);
randomAccessFile.close();
}
else{
randomAccessFile = new RandomAccessFile(Writefile, "rw");
long i=Writefile.length();
randomAccessFile.seek(i);
randomAccessFile.write(bytes, 0, bytes.length);
randomAccessFile.close();
}
}
}
Thanks in advance
It is hard to give a conclusive answer to your question, because your issue could be the result of several different causes. Also, once the cause of the problem has been identified, you will probably have multiple options to mitigate it.
The first place to look is at the reader side. The code does a take() in a loop with a 200 millisecond pause between each take. Depending on your QoS settings on the DataReader, you might be facing a situation where your samples get overwritten in the DataReader while your application is sleeping for 200 milliseconds. If you are doing this over a gigabit ethernet, then a typical DDS product would be able to do those 5 chunks of 1 megabyte within that sleep period, meaning that your default, one-place buffer will get overwritten 4 times during your sleep.
This scenario would be likely if you used the default history QoS settings for your BinaryFileDataReader, which means history.kind = KEEP_LAST and history.depth = 1. Increasing the latter to a larger value, for example to 20, would result in a queue capable of holding 20 chunks of your file while you are sleeping. That should be sufficient for now.
If this does not resolve your issue, other possible causes can be explored.