We Keep Coding

How to handle huge json over network call through pagination in java?

I have this existing API running in loop which creates a connection and fetches huge json(1 GB) in each iteration from a service through restTemplate as follow:
ResponseEntity<String> response = restTemplate.exchange(uri.get().toString(), HttpMethod.POST,entity,
String.class);
The response is then converted to a complex java object through GSON. The problem with above approach is, rest template converts the inputstream to String through StringBuffer which ends up creating lots of char[] eventually running out of memory (OOM) when the loop is iterated for too long which usually is the case. In place of RestEntity, I even used HttpClient, it too does the same (expand char array).
To solve OOM issue, I refactored the API to stream the data to file. Creating a temp file in each iteration and converting to objects as follow:
File targetFile = new File("somepath\\response.tmp");
FileUtils.copyInputStreamToFile(response.getEntity().getContent(), targetFile);
List<Object> objects = gson.fromJson(reader, new TypeToken<List<Object>>(){}.getType());
is this the way to go or is there any effective approach to solve such problems? Maybe pooling connections instead of creating new in each iteration (will this be considerable change?)
Also, on analysing the API on 4GB xmx, 2GB xms heap, jvisual shows the below:
As it can be seen, the running thread has been allocated huge bytes.
Heap size during the API runtime:

Spark streaming gc setup questions

My logic is as follows.
Use createDirectStream to get a topic by log type in Kafka.
After repartition, the log is processed through various processing.
Create a single string using combineByKey for each log type (use StringBuilder).
Finally, save to HDFS by log type.
There are a lot of operations that add strings, so GC happens frequently.
How is it better to set up GC in this situation?
//////////////////////
There are various logic, but I think there is a problem in doing combineByKey.
rdd.combineByKey[StringBuilder](
(s: String) => new StringBuilder(s),
(sb: StringBuilder, s: String) => sb.append(s),
(sb1: StringBuilder, sb2: StringBuilder) => sb1.append(sb2)
).mapValues(_.toString)

The simplest thing with greatest impact you can do with that combineByKey expression is to size the StringBuilder you create so that it does not have to expand its backing character array as you merge string values into it; the resizing amplifies the allocation rate and wastes memory bandwidth by copying from old to new backing array. As a guesstimate, I would say pick the 90th percentile of string length of the resulting data set's records.
A second thing to look at (after collecting some statistics on your intermediate values) would be for your combiner function to pick the StringBuilder instance that has room to fit in the other one when you call sb1.append(sb2).
A good thing to take care of would be to use Java 8; it has optimizations that make a significant difference when there's heavy work on strings and string buffers.
Last but not least, profile to see where you are actually spending your cycles. This workload (excluding any additional custom processing you are doing) shouldn't need to promote a lot of objects (if any) to old generation, so you should make sure that young generation has ample size and is collected in parallel.

Why is string concatenation faster in Delphi than in Java?

Problem
I wrote 2 programs, one in Delphi and one in Java, for string concatenation and I noticed a much faster string concatenation in Delphi compared to Java.
Java
String str = new String();
long t0 = System.nanoTime();
for (int i = 0; i < 50000; i++)
str += "abc";
long t1 = System.nanoTime();
System.out.println("String + String needed " + (t1 - t0) / 1000000 + "ms");
Delphi
Stopwatch.Start;
for i := 1 to 50000 do
str := str + 'abc';
Stopwatch.Stop;
ShowMessage('Time in ms: ' + IntToStr(Stopwatch.ElapsedMilliseconds));
Question
Both measure the time in milliseconds but the Delphi program is much faster with 1ms vs. Javas 2 seconds. Why is string concatenation so much faster in Delphi?
Edit: Looking back at this question with more experience I should have come to the conclusion that the main difference comes from Delphi being compiled and Java being compiled and then run in the JVM.

TLDR
There may be other factors, but certainly a big contributor is likely to be Delphi's default memory manager. It's designed to be a little wasteful of space in order to reduce how often memory is reallocated.
Considering memory manager overhead
When you have a straight-forward memory manager (you might even call it 'naive'), your loop concatenating strings would actually be more like:
//pseudo-code
for I := 1 to 50000 do
begin
if CanReallocInPlace(Str) then
//Great when True; but this might not always be possible.
ReallocMem(Str, Length(Str) + Length(Txt))
else
begin
AllocMem(NewStr, Length(Str) + Length(Txt))
Copy(Str, NewStr, Length(Str))
FreeMem(Str)
end;
Copy(Txt, NewStr[Length(NewStr)-Length(Txt)], Length(Txt))
end;
Notice that on every iteration you increase the allocation. And if you're unlucky, you very often have to:
Allocate memory in a new location
Copy the existing 'string so far'
Finally release the old string
Delphi (and FastMM)
However, Delphi has switched from the default memory manager used in it's early days to a previously 3rd party one (FastMM) that's designed run faster primarily by:
(1) Using a sub-allocator i.e. getting memory from the OS a 'large' page at a time.
Then performing allocations from the page until it runs out.
And only then getting another page from the OS.
(2) Aggressively allocating more memory than requested (anticipating small growth).
Then it becomes more likely the a slightly larger request can be reallocated in-place.
These techniques can thought it's not guaranteed increase performance.
But it definitely does waste space. (And with unlucky fragmentation, the wastage can be quite severe.)
Conclusion
Certainly the simple app you wrote to demonstrate the performance greatly benefits from the new memory manager. You run through a loop that incrementally reallocates the string on every iteration. Hopefully with as many in-place allocations as possible.
You could attempt to circumvent some of FastMM's performance improvements by forcing additional allocations in the loop. (Though sub-allocation of pages would still be in effect.)
So simplest would be to try an older Delphi compiler (such as D5) to demonstrate the point.
FWIW: String Builders
You said you "don't want to use the String Builder". However, I'd like to point out that a string builder obtains similar benefits. Specifically (if implemented as intended): a string builder wouldn't need to reallocate the substrings all the time. When it comes time to finally build the string; the correct amount of memory can be allocated in a single step, and all portions of the 'built string' copied to where they belong.

In Java (and C#) strings are immutable objects. That means that if you have:
string s = "String 1";
then the compiler allocates memory for this string. Haven then
s = s + " String 2"
gives us "String 1 String 2" as expected but because of the immutability of the strings, a new string was allocated, with the exactly size to contain "String 1 String 2" and the content of both strings is copied to the new location. Then the original strings are deleted by the garbage collector. In Delphi a string is more "copy-on-write" and reference counted, which is much faster.
C# and Java have the class StringBuilder with behaves a lot like Delphi strings and are quite faster when modifying and manipulating strings.

Slowness with creating a String from a ObjectNode with Jackson in Java

I have problem with creating a String from a JSON node.
Currently I'm doing it by node.toString() method. But in sometimes this takes 7-8 seconds to create the JSON string weighted 15MB-18MB.
I tried with mapper.writeValueAsString(node)) method too. But it shows some additional time for the test. This is very difficult to check the issue because it is also harder to reproduce.
I'm currently using only ObjectNode (not TextNode, BooleanNode etc) is it this will be effect to this? Or is there any better way to convert JSONNode to String ?
Sample Code :
JsonNodeFactory nodeFactory = JsonNodeFactory.instance;
ObjectNode node = nodeFactory.objectNode();
node.put("fnm", "Namal");
node.put("lnm", "Fernando");
node.put("age", 30);
for (int i = 0; i < 10000; i++) {
ObjectNode order = nodeFactory.objectNode();
order.put("id", (i+1000)+"");
order.put("nm", "ORD"+(i+1000));
order.put("ref", "RF-"+i);
node.put("order"+i, order);
}
long smili = System.currentTimeMillis();
System.out.println("main().Node : " + node.toString());
System.out.println("main().TIMING 1 : " + (System.currentTimeMillis() - smili) / 1000.0);;
long smili2 = System.currentTimeMillis();
ObjectMapper mapper = new ObjectMapper();
System.out.println("main().Node : " + mapper.writeValueAsString(node));
System.out.println("main().TIMING 2 : " + (System.currentTimeMillis() - smili2) / 1000.0);;

First things first: JsonNode.toString() should NOT be used for serialization, ever. It is useful for simple troubleshooting, but since it does not have access to contextual configuration, it will not necessarily produce valid JSON. This is by design.
Instead, you should use ObjectMapper or ObjectWriter to serialize it; this will produce valid JSON using exact configuration and settings that mapper has (or writer created by mapper).
Now: your timing comparison is flawed since you only do one call with ObjectMapper. There is overhead in first N calls; partly due to ObjectMapper initialization, partly due to JVM warmup (dynamic JIT compiler running to optimize and such). To get more usable results you would need to call method multiple times, and ideally let it run for couple of seconds.
But beyond this a common problem is to forget to have enough heap space.
In Java, Strings require memory at least equivalent to 2x length of String in characters (each char takes 2 bytes). But this is just the size of the result; during serialization a temporary buffer is also needed, using roughly comparable amount.
On disk, however, UTF-8 encoding typically uses just 1 byte per character (for ASCII characters).
So assuming you see a 15mB file, it could use 60 mB of memory during processing. If your heap size is set to small (say, 64 megs which is default in many cases) it would lead to very heavy garbage-collection processing. Solution there would be to either increase heap size.
Or, unless you actually need the String, to:
Write directly into file: use of intermediate String is an anti-pattern if the result goes into File or network connection
If you do require intermediate form, serialize as byte[]: this will only need same amount of memory as a file would (since it is UTF-8 encoded in memory, instead of being a wrapped char[]

java while loop memory leak

I used a while loop to fetch message from Amazon SQS. Partial code is as follows:
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl);
while (true) {
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
if (messages.size() > 0) {
MemcachedClient c = new MemcachedClient(new BinaryConnectionFactory(), AddrUtil.getAddresses(memAddress));
for (Message message : messages) {
// get message from aws sqs
String messageid = message.getBody();
String messageRecieptHandle = message.getReceiptHandle();
sqs.deleteMessage(new DeleteMessageRequest(myQueueUrl, messageRecieptHandle));
// get details info from memcache
String result = null;
String key = null;
key = "message-"+messageid;
result = c.get(key);
}
c.shutdown();
}
}
Will it cause memory leak in such case?
I checked using "ps aux". What I found is that the RSS (resident set size, the non-swapped physical memory that a task used) is growing slowly.

You can't evaluate whether your Java application has a memory leak simply based on the RSS of the process. Most JVMs are pretty greedy, they would rather take more memory from the OS than spend a lot of work on Garbage Collection.
That said your while loop doesn't seem like it has any obvious memory "leaks" either, but that depends on what some of the method calls do (which isn't included above). If you are storing things in static variables, that can be a cause of concern but if the only references are within the scope of the loop you're probably fine.
The simplest way to know if you have a memory leak in a certain area of code is to rigorously exercise that code within a single run of your application (potentially set with a relatively low maximum heap size). If you get an OutOfMemoryError, you probably have a memory leak.

Sorry, but I don't see here code to remove message from the message queue. Did you clean the message list? In case that DeleteRequest removes message from the queue then you try to modify message list which you itereate.
Also you can get better memory usage statistic with visualvm tool which is part of JDK now.