So i have this small client side code
public class Client {
private static Socket socket;
private static ObjectOutputStream out;
public static void main(String[] args) {
while (true) {
try {
if (socket != null) {
out.writeObject("Hello...");
Thread.sleep(1500);
} else {
socket = new Socket("myhost", 1234);
out = new ObjectOutputStream(socket.getOutputStream());
System.out.println("connected to server");
}
} catch (final Exception e) {
//set socket to null for reconnecting
}
}
}
}
What bugs me is that when i run the code with javaw.exe, i see that the java is eating ~10kb more memory every 2-3 sec. So memory usage keeps growing and growing...
Is java really that bad or is there something else wrong?
I ran this code in while loop for a while and memory usage increased for 1000 kb.
Doesn't java gargabe collect the 'tmp' variable after it's used?
try {
if (socket == null) {
final Socket tmp = new Socket("localhost", 1234);
if (tmp != null) {
socket = tmp;
}
Thread.sleep(100);
}
} catch (final Exception e) {
}
So, I've written a simple test server for your client and I'm now running both, and there seems to be no increase in memory usage.
import java.net.*;
import java.io.*;
/**
* example class adapted from
* http://stackoverflow.com/questions/5122569/why-is-java-constantly-eating-more-memory
*/
public class Client {
private static Socket socket;
private static ObjectOutputStream out;
private static void runClient() {
while (true) {
try {
if (socket != null) {
out.writeObject("Hello...");
Thread.sleep(100);
System.out.print(",");
} else {
socket = new Socket("localhost", 1234);
out = new ObjectOutputStream(socket.getOutputStream());
System.out.println("connected to server");
}
} catch (final Exception e) {
//set socket to null for reconnecting
e.printStackTrace();
return;
}
}
}
private static void runServer() throws IOException{
ServerSocket ss = new ServerSocket(1234);
Socket s = ss.accept();
InputStream in = s.getInputStream();
byte[] buffer = new byte[500];
while(in.read(buffer) > 0) {
System.out.print(".");
}
}
public static void main(String[] args)
throws IOException
{
if(args.length > 0) {
runServer();
}
else {
runClient();
}
}
}
What are you doing different?
So, I've looked a bit more detailed at the memory usage of this program, and found for this a useful tool, the "Java Monitoring and Management console" hidden in the development menu of my system :-)
Here is a screenshot of the memory usage while running the client program some time (each 100 ms I send an object, remember) ...
We can see that the memory usage has a saw tooth curve - it is lineary increasing, then comes a garbage collection and it is falling down to the base usage. After some initial period the VM is doing the GC more often (and thus more quickly). For now, no problem.
Here is a variant program where I did not send the same string always, but a different one each time:
private static void runClient() {
int i = 0;
while (true) {
try {
i++;
if (socket != null) {
out.writeObject("Hello " + i + " ...");
Thread.sleep(100);
System.out.print(",");
(The rest is like above). I thought this would need more memory, since the ObjectOutputStream has to remember which Objects are already sent, to be able to reuse their identifiers in case they come again.
But no, it looks quite similar:
The little irregularity between 39 and 40 is a manual full GC made by the "Perform GC" button here - it did not change much, though.
I let the last program run a bit longer, and now we see that the ObjectOutputStream still is holding references to our Strings ...
In half an hour our program ate about 2 MB of memory (on a 64-bit-VM). In this time, it sent 18000 Strings. So, each of the Strings used in average about 100 bytes of memory.
Each of those Strings was between 11 and 17 chars long. the latter ones (about the half) are using in fact 32-char-arrays, the former ones 16-char-arrays, because of the allocation strategy of the StringBuilder. These take 64 or 32 bytes + array-overhead (at least 12 more bytes, more likely more). Additionally the String objects themselves take some memory overhead (at least 8+8+4+4+4 = 28 for class and the fields I remember, more likely more), so we have in average (at least) 88 bytes per String. In addition there likely is some overhead in the ObjectOutputStream to maintain these objects in some data structure.
So, not much more lost than in fact needed.
Ah, one tip of how to avoid the ObjectOutputStream (and the corresponding ObjectInputStream, too) storing the objects, if you don't plan on sending any of them again: Invoke its reset method every some thousand strings or so.
Here is a last screenshot before I kill the program, after a bit more than an hour:
For comparison, I added the named reset and let the program run two more hours (and a bit):
It still collects memory as before, but now when I click on "Perform GC" it cleans everything and goes back at the state before (just a bit over 1 MB). (It would do the same when coming at the end of Heap, but I didn't want to wait this long.)
Well the garbage collector never runs when a variable actually goes out of scope, or you'd spend most of your time in the GC code.
What it does instead (and this is quite a simplification) is it waits until your memory used reaches a threshold, and only then does it start releasing memory.
This is what you're seeing, your memory consumption is increasing so slowly that it'll take a long time to reach the next threshold and actually free memory.
I don't think adding close is your problem because from what i think you are trying to do is keep writing to the stream. Have you tried out.flush(). This flushes the content so that its not in memory anymore.
It looks like you never close the Socket or flush the ObjectOutputStream. Also note that Java garbage collection basically happens not when you want it to but when the garbage collector sees fit.
IO is cached by the Socket implementation until it is flushed. So either you really read the input / output from the socket (or call #flush() on your streams) or you close the socket.
To me logic itself is culPrit, there is no condition to come out of the while loop.
Again no flush.
ObjectOutputStream cached every object you send, in case you send it again. To clear this you need to call the reset() method
Reset will disregard the state of any objects already written to the stream. The state is reset to be the same as a new ObjectOutputStream. The current point in the stream is marked as reset so the corresponding ObjectInputStream will be reset at the same point. Objects previously written to the stream will not be refered to as already being in the stream. They will be written to the stream again.
BTW: 10 KB is worth about 0.1 cents of memory. One minute of your time at minimum wage is worth 100x times this. I suggest you consider what is the best use of your time.
Related
I developed an application using Java socket. I am exchanging messages with this application with the help of byte arrays. I have a message named M1, 1979 bytes long. My socket buffer length is 512 bytes. I read this message in 4 parts, each with 512 bytes, but the last one is of course 443 bytes. I will name these parts like A, B, C, and D. So ABCD is a valid message of mine respectively.
I have a loop with a thread which is like below.
BlockingQueue<Chunk> queue = new LinkedBlockingQueue<>();
InputStream in = socket.getInputStream()
byte[] buffer = new byte[512];
while(true) {
int readResult = in.read(buffer);
if(readResult != -1) {
byte[] arr = Arrays.copyOf(buffer, readResult);
Chunk c = new Chunk(arr);
queue.put(c);
}
}
I'm filling the queue with the code above. When the message sending starts, I see the queue fill up in ABCD form but sometimes I put the data in the queue as a BACD. But I know that this is impossible because the TCP connection guarantees the order.
I looked at the dumps with Wireshark. This message comes correctly with a single tcp package. So there is no problem on the sender side. I am 100% sure that the message has arrived correctly but the read method does not seem to read in the correct order and this situation doesn't always happen. I could not find a valid reason for what caused this.
When I tried the same code on two different computers I noticed that the problem was in only one. The jdk versions on these computers are different. I looked at the version differences between the two jdk versions. When the Jdk version is "JDK 8u202", I am getting the situation where it works incorrectly. When I tried it with jdk 8u271, there was no problem. Maybe it is related to that but I wasn't sure. Because I have no valid evidence.
I am open to all kinds of ideas and suggestions. It's really on its way to being the most interesting problem I've ever encountered.
Thank you for your help.
EDIT: I found similar question.
Blocking Queue Take out of Order
EDIT:
Ok, I have read all the answers given below. Thank you for providing different perspectives for me. I will try to supplement some missing information.
Actually I have 2 threads. Thread 1(SocketReader) is responsible for reading socket. It wraps the information it reads with a Chunk class and puts it on the queue in the other Thread 2. So queue is in Thread 2. Thread 2(MessageDecoder) is consuming the blocking queue. There are no threads other than these. Actually this is a simple example of a "producer consumer design patter".
And yes, other messages are sent, but other messages take up less than 512 bytes. Therefore, I can read in one go. I do not encounter any sort problem.
MessageDecoder.java
public class MessageDecoder implements Runnable{
private BlockingQueue<Chunk> queue = new LinkedBlockingQueue<>();
public MessageDecoder() {
}
public void run() {
while(true) {
Chunk c;
try {
c = queue.take();
System.out.println(c.toString());
} catch (InterruptedException e) {
e.printStackTrace();
}
decodeMessageChunk(c);
}
}
public void put(Chunk c) {
try {
queue.put(c);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
SocketReader.java
public class SocketReader implements Runnable{
private final MessageDecoder msgDec;
private final InputStream in;
byte[] buffer = new byte[512];
public SocketReader(InputStream in, MessageDecoder msgDec) {
this.in = in;
this.msgDec = msgDec;
}
public void run() {
while(true) {
int readResult = in.read(buffer);
if(readResult != -1) {
byte[] arr = Arrays.copyOf(buffer, readResult);
Chunk c = new Chunk(arr);
msgDec.put(c);
}
}
}
}
Even if it's a FIFO queue, the locking of the LinkedBloquingQueue is unfair, so you can't guarantee the ordering of elements. More info regarding this here
I'd suggest using an ArrayBlockingQueue instead. Like the LinkedBloquingQueue, the order is not guaranteed but offers a slightly different locking mechanism.
This class supports an optional fairness policy for ordering waiting
producer and consumer threads. By default, this ordering is not
guaranteed. However, a queue constructed with fairness set to true
grants threads access in FIFO order. Fairness generally decreases
throughput but reduces variability and avoids starvation.
In order to set fairness, you must initialize it using this constructor:
So, for example:
ArrayBlockingQueue<Chunk> fairQueue = new ArrayBlockingQueue<>(1000, true);
/*.....*/
Chunk c = new Chunk(arr);
fairQueue.add(c);
As the docs state, this should grant thread access in FIFO order, guaranteeing the retrievement of the elements to be consistent while avoiding possible locking robbery happening in LinkedBloquingQueue's lock mechanism.
I am using Netty to perform large file upload. It works fine but the RAM used by the client seems to increase with the size of the file. This is not the expected behaviour since everything is piped from the Reading the source file to writing the target file.
At first, I thought about a kind of adaptive buffer growing up until Xmx is reached but setting Xmx to a reasonable value (50M) would lead to an OutOfMemoryError soon after starting upload.
After some research using Eclipse Memory Analyzer, it appears that the object retaining the heap memory is:
org.jboss.netty.channel.socket.nio.NioSocketChannel$WriteRequestQueue
Is there any option for setting a limit to this queue or do I have to code my own queue using ChannelFutures to control the number of bytes and block the pipe when the limit is reached?
Thanks for your help,
Regards,
Renaud
Answer from #normanmaurer on Netty Github
You should use
Channel.isWritable()
to check if the "queue" is full. If so you will need to check if there is enough space to write more. So the effect you see can happen if you write data to quickly to get it send out to the clients.
You can get around this kind of problems when try to write a File via DefaultFileRegion or ChunkedFile.
#normanmaurer thank you I missed this method of the Channel!
I guess I need to read what's happening inside:
org.jboss.netty.handler.stream.ChunkedWriteHandler
UPDATED: 2012/08/30
This is the code I made to solve my problem:
public class LimitedChannelSpeaker{
Channel channel;
final Object lock = new Object();
long maxMemorySizeB;
long size = 0;
Map<ChannelBufferRef, Integer> buffer2readablebytes = new HashMap<ChannelBufferRef, Integer>();
public LimitedChannelSpeaker(Channel channel, long maxMemorySizeB) {
this.channel= channel;
this.maxMemorySizeB = maxMemorySizeB;
}
public ChannelFuture speak(ChannelBuffer buff) {
if (buff.readableBytes() > maxMemorySizeB) {
throw new IndexOutOfBoundsException("The buffer is larger than the maximum allowed size of " + maxMemorySizeB + "B.");
}
synchronized (lock) {
while (size + buff.readableBytes() > maxMemorySizeB) {
try {
lock.wait();
} catch (InterruptedException ex) {
throw new RuntimeException(ex);
}
}
ChannelBufferRef ref = new ChannelBufferRef(buff);
ref.register();
ChannelFuture future = channel.write(buff);
future.addListener(new ChannelBufferRef(buff));
return future;
}
}
private void spoken(ChannelBufferRef ref) {
synchronized (lock) {
ref.unregister();
lock.notifyAll();
}
}
private class ChannelBufferRef implements ChannelFutureListener {
int readableBytes;
public ChannelBufferRef(ChannelBuffer buff) {
readableBytes = buff.readableBytes();
}
public void unregister() {
buffer2readablebytes.remove(this);
size -= readableBytes;
}
public void register() {
buffer2readablebytes.put(this, readableBytes);
size += readableBytes;
}
#Override
public void operationComplete(ChannelFuture future) throws Exception {
spoken(this);
}
}
}
for a Desktop background application
Netty is designed for highly scalable servers e.g. around 10,000 connections. For a desktop application with less than a few hundred connections, I would use plain IO. You may find the code is much simpler and it should use less than 1 MB.
I have a Socket that I am both reading and writing to, via BufferedReaders and BufferedWriters. I'm not sure which operations are okay to do from separate threads. I would guess that writing to the socket from two different threads at the same time is a bad idea. Same with reading off the socket from two different threads at the same time. What about reading on one thread while writing on another?
I ask because I want to have one thread blocked for a long time on a read as it waits for more data, but during this wait I also have occasional data to send on the socket. I'm not clear if this is threadsafe, or if I should cancel the read before I write (which would be annoying).
Sockets are thread unsafe at the stream level. You have to provide synchronization. The only warranty is that you won't get copies of the exact same bytes in different read invocations no matter concurrency.
But at a Reader and, specially, Writer level, you might have some locking problems.
Anyway, you can handle read and write operations with the Socket's streams as if they were completely independent objects (they are, the only thing they share is their lifecyle).
Once you have provided correct synchronization among reader threads on one hand, and writer threads on the other hand, any number of readers and writers will be okay. This means that, yes, you can read on one thread and write on another (in fact that's very frequent), and you don't have to stop reading while writing.
One last advice: all of the operations involving threads have associated timeout, make sure that you handle the timeouts correctly.
You actually read from InputStream and write to OutputStream. They are fairly independent and for as long as you serialize access to each of them you are ok.
You have to correlate, however, the data that you send with data that you receive. That's different from thread safety.
Java java.net.Socket is not actually thread safe: Open the Socket source, and look at the (let say) connected member field and how it is used. You will see that is not volatile, read and updated without synchrinization. This indicates that Socket class is not designed to be used by multiple threads. Though, there is some locks and synchronization there, it is not consistent.`
I recommend not to do it. Eventually, use buffers(nio), and do socket reads/writes in one thread
For details go the the discussionv
You can have one thread reading the socket and another thread writing to it. You may want to have a number of threads write to the socket, in which case you have to serialize your access with synchronization or you could have a single writing thread which gets the data to write from a queue. (I prefer the former)
You can use non-blocking IO and share the reading and writing work in a single thread. However this is actually more complex and tricky to get right. If you want to do this I suggest you use a library to help you such as Netty or Mina.
Very interesting, the nio SocketChannel writes are synchronized
http://www.docjar.com/html/api/sun/nio/ch/SocketChannelImpl.java.html
The old io Socket stuff depends on the OS so you would have to look at the OS native code to know for sure(and that may vary from OS to OS)...
Just look at java.net.SocketOutputStream.java which is what Socket.getOutputStream returns.
(unless of course I missed something).
oh, one more thing, they could have put synchronization in the native code in every JVM on each OS but who knows for sure. Only the nio is obvious that synchronization exists.
This is how socketWrite in native code, so it's not thread safe from the code
JNIEXPORT void JNICALL
Java_java_net_SocketOutputStream_socketWrite0(JNIEnv *env, jobject this,
jobject fdObj,
jbyteArray data,
jint off, jint len) {
char *bufP;
char BUF[MAX_BUFFER_LEN];
int buflen;
int fd;
if (IS_NULL(fdObj)) {
JNU_ThrowByName(env, "java/net/SocketException", "Socket closed");
return;
} else {
fd = (*env)->GetIntField(env, fdObj, IO_fd_fdID);
/* Bug 4086704 - If the Socket associated with this file descriptor
* was closed (sysCloseFD), the the file descriptor is set to -1.
*/
if (fd == -1) {
JNU_ThrowByName(env, "java/net/SocketException", "Socket closed");
return;
}
}
if (len <= MAX_BUFFER_LEN) {
bufP = BUF;
buflen = MAX_BUFFER_LEN;
} else {
buflen = min(MAX_HEAP_BUFFER_LEN, len);
bufP = (char *)malloc((size_t)buflen);
/* if heap exhausted resort to stack buffer */
if (bufP == NULL) {
bufP = BUF;
buflen = MAX_BUFFER_LEN;
}
}
while(len > 0) {
int loff = 0;
int chunkLen = min(buflen, len);
int llen = chunkLen;
(*env)->GetByteArrayRegion(env, data, off, chunkLen, (jbyte *)bufP);
while(llen > 0) {
int n = NET_Send(fd, bufP + loff, llen, 0);
if (n > 0) {
llen -= n;
loff += n;
continue;
}
if (n == JVM_IO_INTR) {
JNU_ThrowByName(env, "java/io/InterruptedIOException", 0);
} else {
if (errno == ECONNRESET) {
JNU_ThrowByName(env, "sun/net/ConnectionResetException",
"Connection reset");
} else {
NET_ThrowByNameWithLastError(env, "java/net/SocketException",
"Write failed");
}
}
if (bufP != BUF) {
free(bufP);
}
return;
}
len -= chunkLen;
off += chunkLen;
}
if (bufP != BUF) {
free(bufP);
}
}
I'm going to use a SoftReference-based cache (a pretty simple thing by itself). However, I've came across a problem when writing a test for it.
The objective of the test is to check if the cache does request the previously cached object from the server again after the memory cleanup occurs.
Here I find the problem how to make system to release soft referenced objects. Calling System.gc() is not enough because soft references will not be released until the memory is low. I'm running this unit test on the PC so the memory budget for the VM could be pretty large.
================== Added later ==============================
Thank you all who took care to answer!
After considering all pro's and contra's I've decided to go the brute force way as advised by nanda and jarnbjo. It appeared, however, that JVM is not that dumb - it won't even attempt garbage collecting if you ask for a block which alone is bigger than VM's memory budget. So I've modified the code like this:
/* Force releasing SoftReferences */
try {
final List<long[]> memhog = new LinkedList<long[]>();
while(true) {
memhog.add(new long[102400]);
}
}
catch(final OutOfMemoryError e) {
/* At this point all SoftReferences have been released - GUARANTEED. */
}
/* continue the test here */
This piece of code forces the JVM to flush all SoftReferences. And it's very fast to do.
It's working better than the Integer.MAX_VALUE approach, since here the JVM really tries to allocate that much memory.
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
I now use this bit of code everywhere I need to unit test code using SoftReferences.
Update: This approach will indeed work only with less than 2G of max memory.
Also, one need to be very careful with SoftReferences. It's so easy to keep a hard reference by mistake that will negate the effect of SoftReferences.
Here is a simple test that shows it working every time on OSX. Would be interested in knowing if JVM's behavior is the same on Linux and Windows.
for (int i = 0; i < 1000; i++) {
SoftReference<Object> softReference = new SoftReferencelt<Object>(new Object());
if (null == softReference.get()) {
throw new IllegalStateException("Reference should NOT be null");
}
try {
Object[] ignored = new Object[(int) Runtime.getRuntime().maxMemory()];
} catch (OutOfMemoryError e) {
// Ignore
}
if (null != softReference.get()) {
throw new IllegalStateException("Reference should be null");
}
System.out.println("It worked!");
}
An improvement that will work for more than 2G max memory. It loops until an OutOfMemory error occurs.
#Test
public void shouldNotHoldReferencesToObject() {
final SoftReference<T> reference = new SoftReference<T>( ... );
// Sanity check
assertThat(reference.get(), not(equalTo(null)));
// Force an OoM
try {
final ArrayList<Object[]> allocations = new ArrayList<Object[]>();
int size;
while( (size = Math.min(Math.abs((int)Runtime.getRuntime().freeMemory()),Integer.MAX_VALUE))>0 )
allocations.add( new Object[size] );
} catch( OutOfMemoryError e ) {
// great!
}
// Verify object has been garbage collected
assertThat(reference.get(), equalTo(null));
}
Set the parameter -Xmx to a very
small value.
Prepare your soft
reference
Create as many object as
possible. Ask for the object everytime until it asked the object from server again.
This is my small test. Modify as your need.
#Test
public void testSoftReference() throws Exception {
Set<Object[]> s = new HashSet<Object[]>();
SoftReference<Object> sr = new SoftReference<Object>(new Object());
int i = 0;
while (true) {
try {
s.add(new Object[1000]);
} catch (OutOfMemoryError e) {
// ignore
}
if (sr.get() == null) {
System.out.println("Soft reference is cleared. Success!");
break;
}
i++;
System.out.println("Soft reference is not yet cleared. Iteration " + i);
}
}
You could explicitly set the soft reference to null in your test, and as such simulate that the soft reference has been released.
This avoids any complicated test setup that is memory and garbage collection dependend.
Instead of a long running loop (as suggested by nanda), it's probably faster and easier to simply create a huge primitive array to allocate more memory than available to the VM, then catch and ignore the OutOfMemoryError:
try {
long[] foo = new long[Integer.MAX_VALUE];
}
catch(OutOfMemoryError e) {
// ignore
}
This will clear all weak and soft references, unless your VM has more than 16GB heap available.
I know this is an absolute shot in the dark, but we're absolutely perplexed.
A perl (5.8.6) script run by Java (1.5) is taking more than an hour to complete. The same script, when run manually from the command line takes 12 minutes to complete. This is on a Linux host.
Logging is the same in both cases and the script is run with the same parameters in both cases.
The script does some complex stuff like Oracle DB access, some scp's, etc, but again, it does the exact same actions in both cases.
We're stumped. Has anyone ever run into a similar situation? If not and if you were faced with the same situation, how would you consider debugging it?
Sub-proceses which produce console output can block (and deadlock) if their stdout/stderror streams are not flushed. #gustafc, the code posed will eventually block the sub-process when it tries to write to stdout/stderror, and there is no room in the stream (and the stream is not being serviced by java).
Process p = startProcess();
final InputStream stdout = p.getInputStream();
final InputStream sterr = p.getErrorStream();
new Thread() {
public void run() {
int c;
while ((c = sterr.read()) != -1) {
System.out.print((char)c);
}
}
}.start();
new Thread() {
public void run() {
int c;
while ((c = sterr.read()) != -1) {
System.out.print((char)c);
}
}
}.start();
I assume you've discarded the possibility that the Java wrapper happens to run simultaneously as something else which causes huge contention over some scarce resource? Good.
If you have a simple class like this:
public class Exec {
public static void main(String[] args) throws Throwable{
class Transfer implements Runnable {
private final InputStream in;
private final OutputStream out;
public Transfer(InputStream i, OutputStream o){
in = i;
out = o;
}
public void run(){
try {
for (int i; (i = in.read()) != -1;) out.write(i);
out.close();
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Process proc = new ProcessBuilder(args).start();
new Thread(new Transfer(System.in, proc.getOutputStream())).start();
new Thread(new Transfer(proc.getInputStream(), System.out)).start();
new Thread(new Transfer(proc.getErrorStream(), System.err)).start();
System.exit(proc.waitFor());
}
}
... and you compare time perl script.pl insert args here and time java Exec perl script.pl insert args here, what happens? If the world is sane, they take about the same time (except that the second one needs a few seconds extra for Java to start), and if that's the case, gradually start adapting the Exec class to look more and more like your deployment environment, and see when it starts taking a really long time.
If Exec above really does take longer time, start logging like crazy in the Perl script, so you see which actions take longer time. And btw, log in the Java wrapper, too, so you see if the Perl startup takes a really long time or something.
One possibility is that you are making the system thrash by trying to run a large Java app and a large Perl app on a system that doesn't have enough memory.
It would be a good idea to use monitoring utilities like top vmstat -5 iostat -5 etc to try and figure out if the slowness corresponds to some OS-level pathology.
To bring this thread to a close, the eventual cause was rogue processes consuming too much CPU. When launched from the command-line, the script had normal priority. When launched from Java, the script had low priority and thus took forever to execute. What threw us off was that the Java code was not just executing the script, it was issuing the same commands via SSH that we issued interactively. Thus, we didn't expect the difference in priority.