We Keep Coding

Java Standalone Application is behaving as inconsistency with long runs in windows-10

Can you please any one help on this problem?
we have a Java Desktop application(JDK8/JRE8-32bit) which will run on Windows-10
we are capturing times for each Runs for huge data.
Run1: 76min:24sec
Run2: 80min:34sec
Run3: 57min:8sec
Run4: 76min:50sec
we are not able to predict for next Run how much time we can get.
Windows 10-configuration
Socket-1
No.of Cores=8
No.of Logical Processors=8
Base Speed : 1.99 GHz
RAM: 16GB
Note: If we run the same application in 8GB Of windows-10 then it is taking very huge time nearly 3 hrs for each Run.
In code we have captured the timings for each line execution at attribute level, but it was showing the time difference in mills sec's and this difference was some time increasing/decreasing during executing of application.
CODE:
Block nextSubBlock = this.getNextSubBlock();
while (nextSubBlock != null && !this.endOfFile()) {
while (!this.blockReached(nextSubBlock) && !this.endOfFile()) {
this.processAttribute(this.next());
}
nextSubBlock.processBlock();
nextSubBlock = this.getNextSubBlock();
}
while (!this.endOfBlock() && !this.endOfFile()) {
this.processAttribute(this.next());
}
this.setStaticCounters();
this.processRepeatedBlocks()

Creating threads is slower on AMD systems compared to Intel systems

In a study project we should create 100.000 threads to get a feeling for the time it takes to create a lot of threads and why it's more efficient to use tasks instead.
However we found out, that the same "Create and start 100.000 threads" code runs a lot slower on a modern Ryzen AMD systems compared to some older (even notebook) Intel systems. We did some benchmarking with different JDKs, but all using Java 16 (older versions didn't make a difference).
public class ThreadCreator {
public static void main(String[] args) throws InterruptedException {
List<Thread> startedThreads = new ArrayList<>();
long startTime = System.currentTimeMillis();
for (int i = 0; i < 100_000; i++) {
Thread t = new Thread(() -> {});
t.start();
startedThreads.add(t);
}
for (Thread t : startedThreads) {
t.join();
}
System.out.println("Duration: " + (System.currentTimeMillis() - startTime));
}
}
The benchmark results:
AMD Ryzen 7 3700X System (Java 16, Ubuntu 20.04):
Adopt OpenJDK (Hotspot): 13882ms
Adopt OpenJDK (OpenJ9): 7521ms
Intel i7-8550U System (Fedora 34, Java 16):
Adopt OpenJDK (Hotspot): 5321ms
Adopt OpenJDK (OpenJ9): 3089ms
Intel i5-6600k System (Windows 10, Java 16):
Adopt OpenJDK (Hotspot): 29433ms (Maybe realted to low memory of this system)
Adopt OpenJDK (OpenJ9): 5119ms
The OpenJ9 JVM reduces the time on both systems to nearly the half. However the AMD system never reaches the time of the Intel systems. The AMD system only runs at 10% cpu utilisation during this test.
What might be the reason why creating threads is so much slower on AMD systems compared to Intel systems?

I have a Ryzen 3700 system running Windows 10 and I got the following results:
Duration: 5.813002900 seconds
100000 tasks completed.
The program I ran, using Ada is:
with Ada.Text_IO; use Ada.Text_IO;
with Ada.Calendar; use Ada.Calendar;
procedure Main is
protected counter is
procedure add;
function report return Natural;
private
count : Natural := 0;
end counter;
protected body counter is
procedure add is
begin
count := count + 1;
end add;
function report return Natural is
begin
return count;
end report;
end counter;
task type worker;
task body worker is
begin
counter.add;
end worker;
type worker_access is access worker;
type list is array (Positive range 1 .. 100000) of worker_access;
start_time : Time;
end_time : Time;
begin
start_time := Clock;
declare
The_List : list;
begin
for I in The_List'Range loop
The_List (I) := new worker;
end loop;
end;
end_time := Clock;
Put_Line
("Duration:" & Duration'Image (end_time - start_time) & " seconds");
Put_Line (Natural'Image (counter.report) & " tasks completed.");
end Main;
This program creates a protected object used to count the number of tasks (similar to Java threads) executed. The protected procedure named add only allows one task at a time to increment the count within the protected object.
The inner block within the main procedure achieves the effect of a Java join. Note that a timing of 5.813 seconds is the same as 5813 milliseconds.

How to get percentage of CPU usage of OS from java

I want to calculate percentage of CPU usage of OS from java code.
There are several ways to find it by unix command [e.g. using mpstat, /proc/stat etc...] and use it from Runtime.getRuntime().exec
But I don't want to use the system calls.
I tried ManagementFactory.getOperatingSystemMXBean()
OperatingSystemMXBean osBean =
(OperatingSystemMXBean) ManagementFactory.getOperatingSystemMXBean();
System.out.println(osBean.getSystemLoadAverage());
But it gives the cpu load but not the cpu usage. Is there anyway to find the usage percentage?

In Java 7 you can get it like so:
public static double getProcessCpuLoad() throws Exception {
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
ObjectName name = ObjectName.getInstance("java.lang:type=OperatingSystem");
AttributeList list = mbs.getAttributes(name, new String[]{ "ProcessCpuLoad" });
if (list.isEmpty()) return Double.NaN;
Attribute att = (Attribute)list.get(0);
Double value = (Double)att.getValue();
// usually takes a couple of seconds before we get real values
if (value == -1.0) return Double.NaN;
// returns a percentage value with 1 decimal point precision
return ((int)(value * 1000) / 10.0);
}

You can use the SIGAR API.
It is cross platform ( but I've only use it on Windows).
The Javadoc is available here and the binaries are here
It is licensed under the terms of the Apache 2.0 license.

Getting GCOverhead Limit exceeded

The following code is throwing OutofMemoryError on Linux 3.5 enterprise box running jdk1.6.0_14 but running fine on JDK 1.6.0_20 I am clueless why its happening.
while (rs.next()) {
for (TableMetaData tabMeta : metaList) {
rec.append(getFormattedString(rs, tabMeta));
}
rec.append(lf);
recCount++;
if (recCount % maxRecBeforWrite == 0) {
bOutStream.write(rec.toString().getBytes());
rec = null;
rec = new StringBuilder();
}
}
bOutStream.write(rec.toString().getBytes());
The getFormattedString() method goes here:
private String getFormattedString(ResultSet rs, TableMetaData tabMeta)
throws SQLException, IOException {
String colValue = null;
// check if it is a CLOB column
if (tabMeta.isCLOB()) {
// Column is a CLOB, so fetch it and retrieve first clobLimit chars.
colValue = String.format("%-" + clobLimit + "s", getCLOBString(rs,
tabMeta));
} else {
colValue = String.format("%-" + tabMeta.getColumnSize() + "s", rs
.getString(tabMeta.getColumnName()));
}
return colValue;
}
Below is the exception trace:
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Formatter$FormatSpecifier.justify(Formatter.java:2827)
at java.util.Formatter$FormatSpecifier.print(Formatter.java:2821)
at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2794)
at java.util.Formatter$FormatSpecifier.print(Formatter.java:2677)
at java.util.Formatter.format(Formatter.java:2433)
at java.util.Formatter.format(Formatter.java:2367)
at java.lang.String.format(String.java:2769)
at com.boa.cpal.cpal2repnet.main.CPALToReportNet.getFormattedString(Unknown Source)
I suspect that the use of String.format is the culprit, but not sure. How to overcome this issue?
Please note that this code has been written to query on the database that have huge tables to read the resultset and create extract files with specific formatting.

The exception you are getting refers to the GC overhead limit that is enabled by this HotSpot option:
-XX:+UseGCOverheadLimit -Use a policy that limits the proportion of the VM's time that is spent in GC before an OutOfMemory error is thrown. (Introduced in 6.)
So, my best guess is that your application is simply running out of heap space. As #Andreas_D's answer says, the default heap sizes were changed between jdk1.6.0_14 and JDK 1.6.0_20, and that could explain the different behaviour. Your options are:
Upgrade to the later JVM. (UPDATE - 2012/06 even JDK 1.6.0_20 is now very out of date. Later 1.6 and 1.7 releases have numerous security fixes.)
Explicitly set the heap dimensions -Xmx and -Xms options when launching the JVM. If you are already doing this (on the older JVM), increase the numbers so that the maximum heap size is larger.
You could also adjust the GC overhead limit, but that's probably a bad idea on a production server.
If this particular problem only happens after your server has been running for some time, then maybe you've got memory leaks.

Garbage collection has been improved significantly with JDK 1.6.0_18:
In the Client JVM, the default Java heap configuration has been modified to improve the performance of today's rich client applications. Initial and maximum heap sizes are larger and settings related to generational garbage collection are better tuned.
A quick look at the details in this release notes makes me believe that this is why you have less problems with 1.6.0_20.
The following part of the code is not consistent with the comment:
// Column is a CLOB, so fetch it and retrieve first clobLimit chars.
colValue = String.format("%-" + clobLimit + "s", getCLOBString(rs, tabMeta));
colValue doesn't get the first clobLimit bytes from the CLOB, it is left-justfied at the column clobLimit. I wasn't sure and tried
System.out.println(String.format("%-5s", "1234567890"));
and the output was
1234567890
To achieve what you tell in the comment, you can use the simpler form:
colValue = getCLOBString(rs, tabMeta).substring(0, clobLimit);

Can't get past 2542 Threads in Java on 4GB iMac OSX 10.6.3 Snow Leopard (32bit)

I am running the following program trying to figure out how to configure my JVM to get the maximum number of threads my machine can support. For those that might not know, Snow Leopard ships with Java 6.
I tried starting it with defaults, and the following command lines, I always get the Out of Memory Error at Thread 2542 no matter what the JVM options are set to.
java TestThreadStackSizes 100000
java -Xss1024 TestThreadStackSizes 100000
java -Xmx128m -Xss1024 TestThreadStackSizes 100000
java -Xmx2048m -Xss1024 TestThreadStackSizes 100000
java -Xmx2048m -Xms2048m -Xss1024 TestThreadStackSizes 100000
no matter what I pass it, I get the same results, Out of Memory Error at 2542
public class TestThreadStackSizes
{
public static void main(final String[] args)
{
Thread.currentThread().setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
public void uncaughtException(final Thread t, final Throwable e)
{
System.err.println(e.getMessage());
System.exit(1);
}
});
int numThreads = 1000;
if (args.length == 1)
{
numThreads = Integer.parseInt(args[0]);
}
for (int i = 0; i < numThreads; i++)
{
try
{
Thread t = new Thread(new SleeperThread(i));
t.start();
}
catch (final OutOfMemoryError e)
{
throw new RuntimeException(String.format("Out of Memory Error on Thread %d", i), e);
}
}
}
private static class SleeperThread implements Runnable
{
private final int i;
private SleeperThread(final int i)
{
this.i = i;
}
public void run()
{
try
{
System.out.format("Thread %d about to sleep\n", this.i);
Thread.sleep(1000 * 60 * 60);
}
catch (final InterruptedException e)
{
throw new RuntimeException(e);
}
}
}
}
Any ideas on how I can affect these results?
I wrote this program to figure out what a Windows Server 2003 is capable of, because I am getting these out of memory can't create native threads at very low numbers, like a couple of hundred. I need to see what a particular box was capable of with different -Xss parameters, then I run into this arbitrary limit on OSX.

2542 seems like an arbitrary number:
I shut all programs down except the one terminal window I was running my test from and I got to 2545, that told me it was an arbitrary limit.
To get the number of threads for OSX 10.6.3 you do:
> sysctl kern.num_threads
kern.num_threads: 2560
and
> sysctl kern.num_taskthreads
kern.num_taskthreads: 2560
The 2560 number matches up with the 2542 and 2545 because there are obviously other threads running in the background.
According to the official documentation kern.num_taskthreads can not be adjusted in the desktop version of OSX.

According to the Apple Developer doc the thread stack size should be at least 64K, so your -Xss 1014 is ignored. But even with 64K per thread, the thread stack memory consumption comes only to about 160MB, so this shouldn't be the problem. Threads could also consume memory from a more limited pool, or there could simply be limit on the number of thread you can have per process or user.

You need to find out the maximum number of threads the operating system supports on your system.
On linux you can do something like :
cat /proc/sys/kernel/threads-max
to get the max, and to set it you can do something like :
echo 10000 > /proc/sys/kernel/threads-max
Also try running with :
-XX:-UseBoundThreads
and report back the results.

Do you think you will have these much thread concurrently up to 1 hour? I don't think so. I have worked in application which processed hundreds of documents, convert them from and to diff. format, generates proper logs in DB and stores specific info also. Then also it finished in seconds.
The thing you should take care about it, code wisely to avoid making too much threads. Instead use a ThreadPool provided by Java, so that same threads can be utilized when needed. that will provide better performance. Also keep synchronization on minimal blocks to avoid bottle necks in your execution.
thanks.