We Keep Coding

Java command-line application somehow retains state

Foreword: I apologise if this is a very silly error or something that is in fact well-documented. To me right now it seems very strange and makes absolutely no sense.
The Application
I have a Java command-line application built in IntelliJ IDEA Ultimate on macOS 10.13.4 that makes use of four Maven libraries listed below. Its purpose is to download files from a website, and navigates across paginated results in doing so.
One of this application's features is the ability to keep running in a loop, checking for new results if enough time has passed by the time it finishes its current scan. To do this, it calls Thread.sleep(remainingMillis) as part of the while condition in a do-while block.
The Problem
The application worked without any issues, but after introducing the Thread.sleep() call (I suspect this is the troublesome line anyways), some very strange behaviour occurs: The application performs the first run without issues, fetching three items from the configured website; it is then configured to ensure that 60 seconds have passed before running again. However upon consequent runs, rather than scan the first page of results, logs indicate that it starts looking at page 31 (as an example), where it finds no results. Having failed to find anything, attempt two of three looks at page 32, and the final attempt looks at page 33; it then once again waits until 60 seconds have passed since the scan iteration began.
I can't confirm this, but it seems as though it then continues this count in subsequent scans: 34, 35, then 36, and waiting again. However, the code would suggest that this should have started at 1 again when another iteration of the while starts up.
This could have been IntelliJ or Java playing up, and it may have simply required cleaning out the bin/obj folders, but if this is something due to my code, I would much rather know about it so I don't encounter the same silly issue in the future.
The Observations
Having just run the application a few days later with the current configuration means that it doesn't call Thread.sleep(), as more than 60 seconds pass so it continues with the next iteration immediately; when this happens, the weird page index incrementing issue doesn't rear its head - instead the next iteration continues from page 1 as it should.
Afterwards, running it such that it did Thread.sleep() for several seconds before starting the next iteration didn't cause a problem either... very strange. Was this a dream?
The Code
Sidenote: I added Thread.currentThread().interrupt() to try and fix this issue, but it didn't seem to have an effect.
public static void main(String[] args) {
do {
startMillis = System.currentTimeMillis();
int itemsFetched = startFetching(agent, config, record, 1, 0);
} while (shouldRepeat(config.getRepeatSeconds(), startMillis));
}
private static boolean shouldRepeat(int repeatSeconds, long startMillis) {
long passedMillis = System.currentTimeMillis() - startMillis;
int repeatMillis = repeatSeconds * 1000;
boolean repeatSecondsReached = passedMillis >= repeatMillis;
if (repeatSeconds < 0) {
return false;
} else if (repeatSecondsReached) {
return true;
}
long remainingMillis = repeatMillis - passedMillis;
int remainingSeconds = (int) (remainingMillis / 1000);
try {
Thread.sleep(remainingMillis);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException(e);
}
return true;
}
private static int startFetching(Agenter agent, MyApplicationConfig config, MyApplicationRecord record, int pageIndex, int itemsFetched) {
String categoryCode = config.getCategoryCode();
List<Item> items = agent.getPageOfItems(categoryCode, pageIndex, config);
if (items == null) {
return itemsFetched;
}
int maxItems = config.getMaxItems();
try {
for (Item item : items) {
String itemURL = item.getURL();
agent.downloadItem(itemURL, config, item.getItemCount());
itemsFetched++;
if (maxItems > 0 && itemsFetched >= maxItems) {
return itemsFetched;
}
}
} catch (IOException e) {
// Log
}
return startFetching(agent, config, record, pageIndex + 1, itemsFetched);
}
}
Maven Libraries
commons-cli:commons-cli:1.4
org.apache.logging.log4j:log4j-api:2.11.0
org.apache.logging.log4j:log4j-core:2.11.0
org.jsoup:jsoup:1.11.2

Check your Agenter implementation, in the call of
agent.getPageOfItems the pageIndex is supplied but could be stored there in an instance variable or something like that. The error itself might be that on additional calls it probably didn't get reset (correctly).

Use of parallels in java shows no increase in performance

I recently embarked on a project to simulate a collection of stellar bodies with the use of LWJGL. The solution required many loop iterations per frame to accomplish. The program calculates the forces exerted on each body by every other body. I did not wish to implement any form of limitations, such as tree algorithms. The program itself is able to simulate 800 bodies of random mass (between 1 and 50) at around 15 fps. Here is the original code for calculating, then updating the position of each body.
public void updateAllBodies() {
for (Body b : bodies) {
for (Body c : bodies) {
if (b != c) {
double[] force = b.getForceFromBody(c, G);
b.velocity[0] += force[0];
b.velocity[1] += force[1];
b.velocity[2] += force[2];
b.updatePosition();
}
}
}
}
Recently I came across the subject of parallels and streams. Seeing that my original code used only one thread, I thought I might be able to improve the performance by converting the array to a stream, and executing it with the use of
.parallelStream()
I don't know much about multi-threading and parallelism, but here is the resulting code that I came up with.
public void updateAllBodies() {
Arrays.asList(bodies).parallelStream().forEach(i -> {
for(Body b: bodies){
if (i != b){
double[] force = i.getForceFromBody(b, G);
i.velocity[0] += force[0];
i.velocity[1] += force[1];
i.velocity[2] += force[2];
i.updatePosition();
}
}
});
}
Unfortunately, when executed, this new code resulted in the same 15 fps as the old one. I was able to confirm that there were 3 concurrent threads running with
Thread.currentThread().getName();
At this point, I have no idea as to what the cause could be. lowering the number of bodies does show a drastic increase in frame rate. Any help will be greatly appreciated.

I cant seem to find a way to mark a comment as the answer to a post, so I will state that the best answer was given by softwarenwebie7331.

In OS X, why does using println() cause my program to run faster than without println()

I've run into a really strange bug, and I'm hoping someone here can shed some light as it's way out of my area of expertise.
First, relevant background information: I am running OS X 10.9.4 on a Late 2013 Macbook Pro Retina with a 2.4GHz Haswell CPU. I'm using JDK SE 8u5 for OS X from Oracle, and I'm running my code on the latest version of IntelliJ IDEA. This bug also seems to be specific only to OS X, as I posted on Reddit about this bug already and other users with OS X were able to recreate it while users on Windows and Linux, including myself, had the program run as expected with the println() version running half a second slower than the version without println().
Now for the bug: In my code, I have a println() statement that when included, the program runs at ~2.5 seconds. If I remove the println() statement either by deleting it or commenting it out, the program counterintuitively takes longer to run at ~9 seconds. It's extremely strange as I/O should theoretically slow the program down, not make it faster.
For my actual code, it's my implementation of Project Euler Problem 14. Please keep in mind I'm still a student so it's not the best implementation:
public class ProjectEuler14
{
public static void main(String[] args)
{
final double TIME_START = System.currentTimeMillis();
Collatz c = new Collatz();
int highestNumOfTerms = 0;
int currentNumOfTerms = 0;
int highestValue = 0; //Value which produces most number of Collatz terms
for (double i = 1.; i <= 1000000.; i++)
{
currentNumOfTerms = c.startCollatz(i);
if (currentNumOfTerms > highestNumOfTerms)
{
highestNumOfTerms = currentNumOfTerms;
highestValue = (int)(i);
System.out.println("New term: " + highestValue); //THIS IS THE OFFENDING LINE OF CODE
}
}
final double TIME_STOP = System.currentTimeMillis();
System.out.println("Highest term: " + highestValue + " with " + highestNumOfTerms + " number of terms");
System.out.println("Completed in " + ((TIME_STOP - TIME_START)/1000) + " s");
}
}
public class Collatz
{
private static int numOfTerms = 0;
private boolean isFirstRun = false;
public int startCollatz(double n)
{
isFirstRun = true;
runCollatz(n);
return numOfTerms;
}
private void runCollatz(double n)
{
if (isFirstRun)
{
numOfTerms = 0;
isFirstRun = false;
}
if (n == 1)
{
//Reached last term, does nothing and causes program to return to startCollatz()
}
else if (n % 2 == 0)
{
//Divides n by 2 following Collatz rule, running recursion
numOfTerms = numOfTerms + 1;
runCollatz(n / 2);
}
else if (n % 2 == 1)
{
//Multiples n by 3 and adds one, following Collatz rule, running recursion
numOfTerms = numOfTerms + 1;
runCollatz((3 * n) + 1);
}
}
}
The line of code in question has been commented in with all caps, as it doesn't look like SO does line numbers. If you can't find it, it's within the nested if() statement in my for() loop in my main method.
I've run my code multiple times with and without that line, and I consistently get the above stated ~2.5sec times with println() and ~9sec without println(). I've also rebooted my laptop multiple times to make sure it wasn't my current OS run and the times stay consistent.
Since other OS X 10.9.4 users were able to replicate the code, I suspect it's due to a low-level bug with the compliler, JVM, or OS itself. In any case, this is way outside my knowledge. It's not a critical bug, but I definitely am interested in why this is happening and would appreciate any insight.

I did some research and some more with #ekabanov and here are the findings.
The effect you are seeing only happens with Java 8 and not with Java 7.
The extra line triggers a different JIT compilation/optimisation
The assembly code of the faster version is ~3 times larger and quick glance shows it did loop unrolling
The JIT compilation log shows that the slower version successfully inlined the runCollatz while the faster didn't stating that the callee is too large (probably because of the unrolling).
There is a great tool that helps you analyse such situations, it is called jitwatch. If it is assembly level then you also need the HotSpot Disassembler.
I'll post also my log files. You can feed the hotspot log files to the jitwatch and the assembly extraction is something that you diff to spot the differences.
Fast version's hotspot log file
Fast version's assembly log file
Slow version's hotspot log file
Slow version's assembly log file

New thread freezing application

I have an application which is quite resource intensive, it is using large images as input and some of the operations on these images can take a while. I am therefore looking to make some parts run in their own threads. To do this I have used the following code just to test out first:
Thread t1 = new Thread(new Runnable() {
public void run()
{
inputChooser.setFileFilter(filter);
inputChooser.addChoosableFileFilter(filter);
int img = inputChooser.showOpenDialog(this);
if (img == JFileChooser.APPROVE_OPTION) {
File file = inputChooser.getSelectedFile();
String filename = file.getName();
if (filename.contains("B10")) {
greenBand = 1;
}
if (filename.contains("B20")) {
greenBand = 2;
}
if (filename.contains("B30")) {
greenBand = 3;
}
if (filename.contains("B40")) {
greenBand = 4;
}
if (filename.contains("B50")) {
greenBand = 5;
}
if (filename.contains("B60")) {
greenBand = 6;
}
if (filename.contains("B70")) {
greenBand = 7;
}
try {
greenImage = ImageIO.read(file);
ImageIO.write(greenImage, "JPEG", new File("img2_tmp.jpeg"));
greenImage = ImageIO.read(new File("img2_tmp.jpeg"));
if (greenImage.getWidth() > 8000 | greenImage.getHeight() > 7000) {
greenImage = greenImage.getSubimage(1450, 1400, (greenImage.getWidth()-3200), (greenImage.getHeight()-3000));
}
update(greenImage, greenIcon, greenLabel);
loadingBar.setIndeterminate(false);
checkInput();
} catch (IOException e) {
JOptionPane.showMessageDialog(null, "Input Image Error", "Input Error", WARNING_MESSAGE);
}
}
}});
t1.start();
When I run the application it freezes when this code is called. However, I have managed to get it to work once, I am not sure how but it ran perfectly (not the first time, it froze a few times first and then randomly worked one time). I haven't changed any of the code just some of the indents and such to get it to fit with the rest of the code and ever since it just continues to freeze. A button action press calls this method where the above code is, as soon as the buttons pressed it freezes.
Is there a reason as to why this is happening?
Thanks in advance

You are calling a non-thread-safe code (swing (is NOT thread safe)) from both threads (newly created and main thread) at the same time.
Make sure that you have decoupled the logic before creating new threads.
For this specific use case, I'd suggest that you use SwingWorkers in stead of threads, they are easy to use, and work well within the limitations of swing.
More about SwingWorkers at http://docs.oracle.com/javase/tutorial/uiswing/concurrency/worker.html
Hope this helps.
Good luck.

It's difficult to say exactly, but I notice that the variable greenImage and greenBand are not declared anywhere. That makes me think they are global variables. If something else has access to them, it's possible that they're causing some manipulation that sends your code into an infinite loop or does other unexpected Bad Things.

Java code seems to only use two concurrent threads

I have approximately 40000 objects which might need to be repainted.
Most of them are not on the screen, so it seems that I could save a lot of work by doing the checks concurrently. But, my CPU never goes above 15% usage, so it seems that it is still only using one core. Have I implemented the threads correctly? If so, why aren't all my cores being used? And is there a better way which does utilize all my cores?
public void paintComponent(Graphics g)
{
super.paintComponent(g);
if (game.movables.size() > 10000)
{
final int size = game.drawables.size();
final Graphics gg = g;
Thread[] threads = new Thread[8];
for (int j = 0; j < 8; ++j)
{
final int n = j;
threads[j] = new Thread(new Runnable()
{
public void run()
{
Drawable drawMe;
int start = (size / 8) * n;
int end = (size / 8) * (n + 1);
if (n == 8) end = game.drawables.size(); // incase size
// % 8 != 0
for (int i = start; i < end; ++i)
{
drawMe = game.drawables.get(i);
if (drawMe.isOnScreen())
{
synchronized (gg)
{
drawMe.draw(gg);
}
}
}
}
});
threads[j].start();
}
try
{
for (int j = 0; j < 8; ++j)
threads[j].join();
}
catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
else
{
for (Drawable drawMe : game.drawables)
{
if (drawMe.isOnScreen())
{
drawMe.draw(g);
}
}
}
}

As has been pointed out, the synchronized (gg) is effectively serializing all the drawing, so you're probably going slower than single-threaded code due to thread creation and other overhead.
The main reason I'm writing however is that Swing, which this presumably is, is not thread safe. So the behavior of this program is not only likely to be bad, it's undefined.
Threading errors like this turn up as screwy behavior on some machines with some java runtime parameters and some graphics drivers. Been there. Done that. Not good.
JOGL will give you direct access to the GPU, the surest way to speed rendering.

To do this right, you might start by putting each drawMe in a (properly synchronized) list, then actually draw them in a loop after the joins are done. You can't speed the drawing (though if you've knocked out 99% of the drawMe's you've cut down the time needed dramatically), but if isOnScreen() is somewhat complicated, you'll get some real work out of your cores.
A ConcurrentLinkedQueue would save you the need to synchronize adds to the list.
The next step might be to use a blocking queue instead of a list, so the paint code could run in parallel with the visibility checks. With eight checks running, they should keep well ahead of the drawing. (But I think all the blocking queues either need synchronizing or do synching themselves. I'd skip this and stick with the CLQ and the first solution. Simpler and possibly faster.)
And (as Gene pointed out), everything Swing related starts on the EventQueue. Keep it there or life will get strange. Only your own code, not referencing the UI, should run in your threads.

Since you're already not drawing any objects that are off-screen, you're probably gaining very very little by doing what you're doing above.
I would also go as far as to say you're making it worse, but introducing synchronize which is slow and also introducing threads that cause context switches, which are expensive.
To improve performace you should perhaps look into using different drawing libraries, such as the Java2D drawing library, which is part of the JDK: http://java.sun.com/products/java-media/2D/index.jsp

I'm not sure how java will handle this, but other languages will blow up horribly and die if you reference something across scopes like you're doing with final int n (since it goes out of scope when the loop stops). Consider making it a field of the runnable object. Also, you're synchronizing on the graphics object while you're doing all of the real work. It's likely that you aren't getting any real performance increase from this. You might benefit from explicitly checking if the object is on the screen in parallel which is a read only operation, adding on-screen objects to a set or collection of some other sort, and then rendering sequentially.