I have built a distributed client-server-database Java application based on sockets. The clients sends serialized objects to the servers (currently there are two servers) and the servers deserializes the objects and stores some content of the object in a postgreSQL database.
Now I'm benchmarking my system and I measured the size of the serialized object as well as the throughput and I made a very strange discovery which I cannot explain.
Until the object size reaches around 1400 Bytes or a bit less the throughput decreases, but then from object size of around 1400 Bytes till 2000 Bytes (or a bit above) the throughput stays constant and from an object size of around 2000 Bytes till 2600 Bytes (I only measured it till 2600 Bytes) the throughput increases.
I cannot explain this behaviour. My thinking was that the throughput will always decrease with increasing object size and if the MTU of 1500 Bytes is reached the decrease will be much bigger. But this seems not to be true and especially the constant throughput and the increase I cannot explain at all.
Related
I have a serialized object on disk of a patricia trie(https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/trie/PatriciaTrie.html). On disk, it occupies roughly 7.4 GB. I am using a 64 GB RAM server. When deserialized, the memory consumption of the corresponding process goes up till 40 GB. Is this sensible, because the highest voted answer at Serialized object size vs in memory object size in Java says that "the size in memory will be usually between half and double the serializable size!" I was expecting the in memory size to not go beyond 15 GB, but 40 GB is too much as other processes would be loaded as well.
I thought of using http://docs.oracle.com/javase/7/docs/api/java/lang/instrument/Instrumentation.html for measuring size in memory, but Calculate size of Object in Java says that it "can be used to get the implementation specific approximation of object size." So, it would be again approximate measure only.
Is there something I am missing here. I am closing the file and bufferred reader as well. What could be hogging all the memory? I can't share the code for corporate policy reasons - any help or pointers would be highly appreciated. Thanks
Serialized size on disk has little to do with the size of the data in memory. Every object in Java has some memory overhead (which can vary depending on the JVM mode and version). An single array of bytes would have serialized and deserialized to about the same size/memory. However, an array of billion 8 byte arrays would not.
If you create a heap dump of the data after deserializing the data you should be able to determine exactly where the memory is going.
How to collect heap dumps of any java process
What is the memory consumption of an object in Java?
Trick behind JVM's compressed Oops
We are trying to run a Google Cloud Dataflow job in the cloud but we keep getting "java.lang.OutOfMemoryError: Java heap space".
We are trying to process 610 million records from a Big Query table and writing the processed records to 12 different outputs (main + 11 side outputs).
We have tried increasing our number of instances to 64 n1-standard-4 instances but we are still getting the issue.
The Xmx value on the VMs seem to be set at ~4GB(-Xmx3951927296), even though the instances have 15GB memory. Is there any way of increasing the Xmx Value?
The job ID is - 2015-06-11_21_32_32-16904087942426468793
You can't directly set the heap size. Dataflow, however, scales the heap size with the machine type. You can pick a machine with more memory by setting the flag "--machineType". The heap size should increase linearly with the total memory of the machine type.
Dataflow deliberately limits the heap size to avoid negatively impacting the shuffler.
Is your code explicitly accumulating values from multiple records in memory? Do you expect 4GB to be insufficient for any given record?
Dataflow's memory requirements should scale with the size of individual records and the amount of data your code is buffering in memory. Dataflow's memory requirements shouldn't increase with the number of records.
When using Java sockets, is msg always send with buffer size lenght? (When i send only 300bytes, is it anyway send in package with for example 1024 bytes size?)
And what size buffer is the best option? What is a diffrence betweend 512 bytes, and 8k bytes size?
I don't want to create few threads, so I will ask here. Is java standard tcp serversocket performance enought for handling 100 connestions with 10-20 msg/s smooth?
Most machines have an MTU of 1500 bytes. This means if you send some multiple of this, it will break it into packets. If you have less than this, it may hold the data for a short period of time to see if more data will be sent to reduce the overhead of sending small packets. see Nagle's algorithim for more.
There is not much difference between sending 16 lots of 512 bytes or 8 KB all at once as the OS and network adapter will do some coalescing by default.
What really matters is the bandwidth of your connection between you and the other end. if you have up to 2000 messages per second and they are 512 bytes, you need a 10 Mbit/s line (2000*512*8 = ~8 Mbit)
In terms of numbers, you shouldn't have a problem up to about 10,000 connections, or around 500,000 msg per second. If you have a 1 Gbit/sec line you should be able to get 100 MB/s easily. If you have a 10 Gbit/s line you should be able to get more but you might have trouble using all the bandwidth unless you are careful.
Background:
I have a Java application which does intensive IO on quite large
memory mapped files (> 500 MB). The program reads data, writes data,
and sometimes does both.
All read/write functions have similar computation complexity.
I benchmarked the IO layer of the program and noticed strange
performance characteristics of memory mapped files:
It performs 90k reads per second (read 1KB every iteration at random position)
It performs 38k writes per second (write 1KB every iteration sequentially)
It performs 43k writes per second (write 4 bytes every iteration at random position)
It performs only 9k read/write combined operation per second (read 12 bytes then write 1KB every iteration, at random position)
The programs on 64-bit JDK 1.7, Linux 3.4.
The machine is an ordinary Intel PC with 8 threads CPU and 4GB physical memory. Only 1 GB was assigned to JVM heap when conducting the benchmark.
If more details are needed, here is the benchmark code: https://github.com/HouzuoGuo/Aurinko2/blob/master/src/test/scala/storage/Benchmark.scala
And here is the implementation of the above read, write, read/write functions: https://github.com/HouzuoGuo/Aurinko2/blob/master/src/main/scala/aurinko2/storage/Collection.scala
So my questions are:
Given fixed file size and memory size, what factors affect memory mapped file random read performance?
Given fixed file size and memory size, what factors affect memory mapped file random write performance?
How do I explain the benchmark result of read/write combined operation? (I was expecting it to perform over 20K iterations per second).
Thank you.
The memory mapped file performance depends on disk performance, file system type, free memory available for file system cache and read/write block size. The page size on the linux is 4K. So you should expect most performance with 4k read/writes. An access at random position causes page fault if page is not mapped and will pull a new page read. Usually, you want memory mapped file if you want to see the files as a one memory array ( or ByteBuffer in Java ).
How should I tune GC (e.g. the type of collector, min/max size of heap, size of new/old gen, ratio of Eden to survivor etc.) for the following scenario? (I'm using Sun JDK 6, and available memory is 1GB).
1) a master machine running tomcat which in turn runs a web app to show status of various node machines under the master
2) a node status message includes things like its CPU, memory, IO usage; typical message size is 1KB
3) a node sends 3 such messages per second to the master, and a master has 10 nodes under it
4) the master has to keep status data for the last one hour into its memory, so total status data that the master has to keep in memory is: 3kb * 3600 * 10 = 108MB
5) the master will of course need memory for its other tasks, so the JVM can get upto half GB but not more.
My specific question is, if I keep a bigger new gen, a lot of copying would occur because these objects would live at least for an hour; and if I keep old gen bigger, then a lot of minor collections will happen, so how do I go about this?
Measure yourself!
Attach JVisualVM to the Tomcat process and see how it behaves under various loads. You will most likely find the Visual GC plugin very useful.
See http://techblog.zabuchy.net/2012/monitoring-of-tomcat-with-visualvm-and-visualgc/ for inspiration.
I wouldn't even bother thinking about GC tuning until you have identified a real performance issue. The defaults will probably work fine.
Basically this is a fairly GC-light application. You're talking only 100k per second or so of memory allocations, and a working set around 100mb. That's pretty trivial for a modern server to handle.