We Keep Coding

Robot.mouseMove not moving to specified location properly

Whenever I run a mouseMove command for a robot, the mouse doesn't always go to the same location. For example, I have the following code:
import java.awt.Robot;
import java.util.concurrent.TimeUnit;
public class MainBot {
public static void main(String[] args){
try {
Robot screenWin = new Robot();
TimeUnit.SECONDS.sleep(2);
screenWin.mouseMove(100, 300);
} catch (Exception e) {
e.printStackTrace();
}
}
}
The code usually makes the mouse end up at the X:
First, I hit run (I am using eclipse) and move my mouse to a location (before the 2 second timer is up). Then the 2 second delay finishes and the mouse moves and then the script ends. The problem is, the mouse never seems to go to the same exact place twice. For example, the mouse should go to (100, 300) but it goes to something that looks like (0, 300) most of the time. Other times, however, if I move the mouse at the beginning to where it should roughly be, then it goes to the right spot.
I am getting where the mouse should be using Paint to get the pixel location of a screenshot but I don't think it is that because the location keeps changing.
Is there anything I'm missing how the coordinates for mouseMove work?
Edit: Basically, I hit start with that program, then I move the mouse to a new position (so there is a different initial position before the mouseMove function) and then mouseMove executes. Each time I do this, the mouse goes to a different location.

There's an open bug on OpenJDK, so this could be related:
https://bugs.openjdk.java.net/browse/JDK-8196030?jql=project%20in%20(JDK)%20AND%20component%20in%20(client-libs)%20AND%20Subcomponent%20in%20(java.awt)
The bug details that a problem may have been introduced in Windows 10 Fall Creators update, related to screen scaling and a mouse_move function.
In the meantime, you could try to set your screen scale to 100% instead of 125% and see if it helps.

I found a solution, you just have to move the mouse to the coordinate (0,0) then you can move it to the place you want.

I wrote a class to do proper cursor positioning.
This works under windows 10 scalings too.
Use the MoveMouseControlled(double, double) function to move the cursor to a specified position. It uses a [0,1] coordinate system. The (0,0) Point is the upper left corner of the screen.
import java.awt.AWTException;
import java.awt.Dimension;
import java.awt.MouseInfo;
import java.awt.Point;
import java.awt.Robot;
import java.awt.Toolkit;
public class MouseCorrectRobot extends Robot
{
final Dimension ScreenSize;// Primary Screen Size
public MouseCorrectRobot() throws AWTException
{
super();
ScreenSize = Toolkit.getDefaultToolkit().getScreenSize();
}
private static double getTav(Point a, Point b)
{
return Math.sqrt((double) ((a.x - b.x) * (a.x - b.x) + (a.y - b.y) * (a.y - b.y)));
}
public void MoveMouseControlled(double xbe, double ybe)// Position of the cursor in [0,1] ranges. (0,0) is the upper left corner
{
int xbepix = (int) (ScreenSize.width * xbe);
int ybepix = (int) (ScreenSize.height * ybe);
int x = xbepix;
int y = ybepix;
Point mert = MouseInfo.getPointerInfo().getLocation();
Point ElozoInitPont = new Point(0, 0);
int UgyanAztMeri = 0;
final int UgyanAZtMeriLimit = 30;
int i = 0;
final int LepesLimit = 20000;
while ((mert.x != xbepix || mert.y != ybepix) && i < LepesLimit && UgyanAztMeri < UgyanAZtMeriLimit)
{
++i;
if (mert.x < xbepix)
++x;
else
--x;
if (mert.y < ybepix)
++y;
else
--y;
mouseMove(x, y);
mert = MouseInfo.getPointerInfo().getLocation();
if (getTav(ElozoInitPont, mert) < 5)
++UgyanAztMeri;
else
{
UgyanAztMeri = 0;
ElozoInitPont.x = mert.x;
ElozoInitPont.y = mert.y;
}
}
}
}

I just had a similar problem, to solve it I’ve just done a loop :
Test position
Move
Test position
if not OK move again
And it always works in less than 2 loops
Point pd = new Point(X,Y); // X,Y where mouse must go
int n = 0;
while ((!pd.equals(MouseInfo.getPointerInfo().getLocation())) && (++n <= 5))
{
r.mouseMove(pd.x, pd.y);
}

It works well (correct location) in Full Screen mode with zoom=100%. press F-11 in chrome to full screen page.

How to do smooth-scrolling by Java robot?

I got used to my touch pad, that allows to scroll smoothly and very exactly, but I can not to simulate it by Java robot - mousewheel is getting only integer parameters and a scrolling carried by steps. Can I simulate smoothly scrolling in Java?
robot.mouseWheel(a); // int a

The unit of scrolls will always be by "notches of the wheel" (before you ask: that's how the measurement is named in the docs). This is simply how it's implemented by the hardware (to be more specific: the mouse). How many pixels are scrolled per "notch" is nothing but OS-configuration. You can't mess with that with pure java and I wouldn't recommend it, even if it was possible.
What you can do nevertheless is to slow down the speed at which the robot scrolls:
import java.awt.Robot;
public class Test{
public static void main(String[] args)
throws Exception
{
//time to switch to a specific window where the robot ought to be tested
try{ Thread.sleep(2000); }catch(InterruptedException e){}
Robot r = new Robot();
for(int i = 0; i < 20; i++){
//scroll and wait a bit to give the impression of smooth scrolling
r.mouseWheel(1);
try{ Thread.sleep(50); }catch(InterruptedException e){}
}
}
}

Java Audio Metronome | Timing and Speed Problems

I’m starting to work on a music/metronome application in Java and I’m running into some problems with the timing and speed.
For testing purposes I’m trying to play two sine wave tones at the same time at regular intervals, but instead they play in sync for a few beats and then slightly out of sync for a few beats and then back in sync again for a few beats.
From researching good metronome programming, I found that Thread.sleep() is horrible for timing, so I completely avoided that and went with checking System.nanoTime() to determine when the sounds should play.
I’m using AudioSystem’s SourceDataLine for my audio player and I’m using a thread for each tone that constantly polls System.nanoTime() in order to determine when the sound should play. I create a new SourceDataLine and delete the previous one each time a sound plays, because the volume fluctuates if I leave the line open and keep playing sounds on the same line. I create the player before polling nanoTime() so that the player is already created and all it has to do is play the sound when it is time.
In theory this seemed like a good method for getting each sound to play on time, but it’s not working correctly. I’m not sure if the timing problems are from running different threads or if it has to do with deleting and recreating the SourceDataLine or if it’s in playing sounds or what exactly...
At the moment this is just a simple test in Java, but my goal is to create my app on mobile devices (Android, iOS, Windows Phone, etc)...however my current method isn’t even keeping perfect time on a PC, so I’m worried that certain mobile devices with limited resources will have even more timing problems. I will also be adding more sounds to it to create more complex rhythms, so it needs to be able to handle multiple sounds going simultaneously without sounds lagging.
Another problem I’m having is that the max tempo is controlled by the length of the tone since the tones don’t overlap each other. I tried adding additional threads so that every tone that played would get its own thread...but that really screwed up the timing, so I took it out. I would like to have a way to overlap the previous sound to allow for much higher tempos.
Any help getting these timing and speed issues straightened out would be greatly appreciated!
Thanks.
SoundTest.java:
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.*;
import java.io.*;
import javax.sound.sampled.*;
public class SoundTest implements ActionListener {
static SoundTest soundTest;
// ENABLE/DISABLE SOUNDS
boolean playSound1 = true;
boolean playSound2 = true;
JFrame mainFrame;
JPanel mainContent;
JPanel center;
JButton buttonPlay;
int sampleRate = 44100;
long startTime;
SourceDataLine line = null;
int tickLength;
boolean playing = false;
SoundElement sound01;
SoundElement sound02;
public static void main (String[] args) {
soundTest = new SoundTest();
SwingUtilities.invokeLater(new Runnable() { public void run() {
soundTest.gui_CreateAndShow();
}});
}
public void gui_CreateAndShow() {
gui_FrameAndContentPanel();
gui_AddContent();
}
public void gui_FrameAndContentPanel() {
mainContent = new JPanel();
mainContent.setLayout(new BorderLayout());
mainContent.setPreferredSize(new Dimension(500,500));
mainContent.setOpaque(true);
mainFrame = new JFrame("Sound Test");
mainFrame.setContentPane(mainContent);
mainFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
mainFrame.pack();
mainFrame.setVisible(true);
}
public void gui_AddContent() {
JPanel center = new JPanel();
center.setOpaque(true);
buttonPlay = new JButton("PLAY / STOP");
buttonPlay.setActionCommand("play");
buttonPlay.addActionListener(this);
buttonPlay.setPreferredSize(new Dimension(200, 50));
center.add(buttonPlay);
mainContent.add(center, BorderLayout.CENTER);
}
public void actionPerformed(ActionEvent e) {
if (!playing) {
playing = true;
if (playSound1)
sound01 = new SoundElement(this, 800, 1);
if (playSound2)
sound02 = new SoundElement(this, 1200, 1);
startTime = System.nanoTime();
if (playSound1)
new Thread(sound01).start();
if (playSound2)
new Thread(sound02).start();
}
else {
playing = false;
}
}
}
SoundElement.java
import java.io.*;
import javax.sound.sampled.*;
public class SoundElement implements Runnable {
SoundTest soundTest;
// TEMPO CHANGE
// 750000000=80bpm | 300000000=200bpm | 200000000=300bpm
long nsDelay = 750000000;
int clickLength = 4100;
byte[] audioFile;
double clickFrequency;
double subdivision;
SourceDataLine line = null;
long audioFilePlay;
public SoundElement(SoundTest soundTestIn, double clickFrequencyIn, double subdivisionIn){
soundTest = soundTestIn;
clickFrequency = clickFrequencyIn;
subdivision = subdivisionIn;
generateAudioFile();
}
public void generateAudioFile(){
audioFile = new byte[clickLength * 2];
double temp;
short maxSample;
int p=0;
for (int i = 0; i < audioFile.length;){
temp = Math.sin(2 * Math.PI * p++ / (soundTest.sampleRate/clickFrequency));
maxSample = (short) (temp * Short.MAX_VALUE);
audioFile[i++] = (byte) (maxSample & 0x00ff);
audioFile[i++] = (byte) ((maxSample & 0xff00) >>> 8);
}
}
public void run() {
createPlayer();
audioFilePlay = soundTest.startTime + nsDelay;
while (soundTest.playing){
if (System.nanoTime() >= audioFilePlay){
play();
destroyPlayer();
createPlayer();
audioFilePlay += nsDelay;
}
}
try { destroyPlayer(); } catch (Exception e) { }
}
public void createPlayer(){
AudioFormat af = new AudioFormat(soundTest.sampleRate, 16, 1, true, false);
try {
line = AudioSystem.getSourceDataLine(af);
line.open(af);
line.start();
}
catch (Exception ex) { ex.printStackTrace(); }
}
public void play(){
line.write(audioFile, 0, audioFile.length);
}
public void destroyPlayer(){
line.drain();
line.close();
}
}

This sort of thing is difficult to get right. What you have to realise is that in order to even play a sound, it has to be loaded into an audio driver (and possibly a sound card). This takes time, and you have to account for that. There are basically two options for you:
Rather than counting down a delay between every beat, count down a delay from the start, when the metronome activates. As an example, say for instance that you want a beat every second. Because of the ~20ms delay, in your old method you'd get beats at 20ms, 1040, 2060, 3080, etc... If you count down from the start and place beats at 1000, 2000, 3000, etc. then they will play at 20ms, 1020, 2020, 3020, etc... There will still be some variance since the dalay itself varies a bit, but there should be about 1000ms between beats and it will not go out of sync (or at least, the problem will not get worse over time and likely can't be heard).
The better option, and the one that most of such programs use, is to generate larger pieces of music. Buffer for instance 20 seconds ahead and play that. The timing should be perfect during those 20 seconds. When those 20 seconds are almost over you must generate some new sound. If you can find out how to do this, you should append the new waveform to the old and have it play continuously. Otherwise, just generate a new 20 second soundbit and accept the delay between them.
Now as for your problem with the sounds not being able to overlap... I'm no expert and I don't really know an answer, but this I do know: Something has to mix the sounds if you need them to overlap. Either you can do that yourself in software by combining the waveform bytes (I think it's an addition in some logarithmic space), or you need to send the different overlapping sounds to different 'channels', in which case the audio driver or sound card does it for you. I don't know how this works in Java though, or I forgot, but I learned this through trial-and-error and working with .mod files.

Why is this Java program taking up so much memory?

I have a small piece of code that takes a screenshot of my desktop every five minutes.
However I'm a little confused by the amount of memory it takes up - often it will creep up to 200mb of RAM, which I'm sure is excessive... Can anyone tell me a) sensible ways to reduce the memory footprint or b) why it's going up at all?
/**
* Code modified from code given in http://whileonefork.blogspot.co.uk/2011/02/java-multi-monitor-screenshots.html following a SE question at
* http://stackoverflow.com/questions/10042086/screen-capture-in-java-not-capturing-whole-screen and then modified by a code review at http://codereview.stackexchange.com/questions/10783/java-screengrab
*/
package com.tmc.personal;
import java.awt.AWTException;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.TimeZone;
import java.util.concurrent.TimeUnit;
import javax.imageio.ImageIO;
class ScreenCapture {
static int minsBetweenScreenshots = 5;
public static void main(String args[]) {
int indexOfPicture = 1000;// should be only used for naming file...
while (true) {
takeScreenshot("ScreenCapture" + indexOfPicture++);
try {
TimeUnit.MINUTES.sleep(minsBetweenScreenshots);
} catch (Exception e) {
e.printStackTrace();
}
}
}
//from http://www.coderanch.com/t/409980/java/java/append-file-timestamp
private final static String getDateTime()
{
DateFormat df = new SimpleDateFormat("yyyy-MM-dd_hh:mm:ss");
df.setTimeZone(TimeZone.getTimeZone("PST"));
return df.format(new Date());
}
public static void takeScreenshot(String filename) {
Rectangle allScreenBounds = getAllScreenBounds();
Robot robot;
try {
robot = new Robot();
BufferedImage screenShot = robot.createScreenCapture(allScreenBounds);
ImageIO.write(screenShot, "jpg", new File(filename + getDateTime()+ ".jpg"));
} catch (AWTException e) {
System.err.println("Something went wrong starting the robot");
e.printStackTrace();
} catch (IOException e) {
System.err.println("Something went wrong writing files");
e.printStackTrace();
}
}
/**
* Okay so all we have to do here is find the screen with the lowest x, the
* screen with the lowest y, the screen with the higtest value of X+ width
* and the screen with the highest value of Y+height
*
* #return A rectangle that covers the all screens that might be nearby...
*/
private static Rectangle getAllScreenBounds() {
Rectangle allScreenBounds = new Rectangle();
GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();
GraphicsDevice[] screens = ge.getScreenDevices();
int farx = 0;
int fary = 0;
for (GraphicsDevice screen : screens) {
Rectangle screenBounds = screen.getDefaultConfiguration().getBounds();
// finding the one corner
if (allScreenBounds.x > screenBounds.x) {
allScreenBounds.x = screenBounds.x;
}
if (allScreenBounds.y > screenBounds.y) {
allScreenBounds.y = screenBounds.y;
}
// finding the other corner
if (farx < (screenBounds.x + screenBounds.width)) {
farx = screenBounds.x + screenBounds.width;
}
if (fary < (screenBounds.y + screenBounds.height)) {
fary = screenBounds.y + screenBounds.height;
}
allScreenBounds.width = farx - allScreenBounds.x;
allScreenBounds.height = fary - allScreenBounds.y;
}
return allScreenBounds;
}
}

The other answers are right that Java will use as much memory as it is allowed to, at which point it will garbage collect. To work around this, you can specify a smaller max heap size in the JVM settings. You do this with the -Xmx setting. For example, if you think you only need 32MB, run it as:
java -Xmx32M [your main class or jar here]
The heap of your program (the non-stack memory) will never take more than 32MB, but it will crash if it needs more than that at once (and that's where you'll need to profile). I don't see any obvious leaks in your program (assuming ImageIO doesn't require any cleanup), though, so I think you'll be fine.

JVM garbage collector will eventually clear your memory heap. For manually clearing that heap call Runtime.getRuntime().gc();, but I don't advise doing that for every 5 minutes.

For a modern computer, 200MB is not an excessive amount of memory. The JVM will let the heap grow for a while if you're creating and discarding lots of objects so that your program doesn't get bogged down with garbage collection. Let your program run for several hours and then check back if you think there's a problem.

Java Robot class simulating human mouse movement

I am working on a project about remote control, send conrdinate x and y of cursor from client to server.
But
robot.mouseMove(x,y);
will only move the cursor to the particular point without moving the cursor form origional point
I have find this simple algorthim to simulate the continuing movement of mouse
for (int i=0; i<100; i++){
int x = ((end_x * i)/100) + (start_x*(100-i)/100);
int y = ((end_y * i)/100) + (start_y*(100-i)/100);
robot.mouseMove(x,y);
}
But this algorthim still too simple, it just move from one point to other point slowly, which still unlike human behave.
I have read some open soruce code about remote control from web, and I find this project
http://code.google.com/p/java-remote-control/
is using the method call MosueMovement from MouseListener class, which they use to perform the "dragging".
I like to know is any one know the better way of doing this?

There are a few things to consider if you want to make the artificial movement natural, I think:
Human mouse movement is usually in a slight arc because the mouse hand pivots around the wrist. Also that arc is more pronounced for horizontal movements than vertical.
Humans tend to go in the general direction, often overshoot the target and then go back to the actual target.
Initial speed towards the target is quite fast (hence the aforementioned overshoot) and then a bit slower for precise targeting. However, if the cursor is close to the target initially the quick move towards it doesn't happen (and neither does the overshoot).
This is a bit complex to formulate in algorithms, though.

For anyone in the future: I developed a library for Java, that mimics human mouse movement. The noise/jaggedness in movement, sinusoidal arcs, overshooting the position a bit, etc. Plus the library is written with extension and configuration possibilities in mind, so anyone can fine tune it, if the default solution is not matching the case. Available from Maven Central now.
https://github.com/JoonasVali/NaturalMouseMotion

Take a look in this example that I wrote. You can improve this to simulate what Joey said. I wrote it very fast and there are lots of things that can be improved (algorithm and class design). Note that I only deal with left to right movements.
import java.awt.AWTException;
import java.awt.MouseInfo;
import java.awt.Point;
import java.awt.Robot;
public class MouseMoving {
public static void main(String[] args) {
new MouseMoving().execute();
}
public void execute() {
new Thread( new MouseMoveThread( 100, 50, 50, 10 ) ).start();
}
private class MouseMoveThread implements Runnable {
private Robot robot;
private int startX;
private int startY;
private int currentX;
private int currentY;
private int xAmount;
private int yAmount;
private int xAmountPerIteration;
private int yAmountPerIteration;
private int numberOfIterations;
private long timeToSleep;
public MouseMoveThread( int xAmount, int yAmount,
int numberOfIterations, long timeToSleep ) {
this.xAmount = xAmount;
this.yAmount = yAmount;
this.numberOfIterations = numberOfIterations;
this.timeToSleep = timeToSleep;
try {
robot = new Robot();
Point startLocation = MouseInfo.getPointerInfo().getLocation();
startX = startLocation.x;
startY = startLocation.y;
} catch ( AWTException exc ) {
exc.printStackTrace();
}
}
#Override
public void run() {
currentX = startX;
currentY = startY;
xAmountPerIteration = xAmount / numberOfIterations;
yAmountPerIteration = yAmount / numberOfIterations;
while ( currentX < startX + xAmount &&
currentY < startY + yAmount ) {
currentX += xAmountPerIteration;
currentY += yAmountPerIteration;
robot.mouseMove( currentX, currentY );
try {
Thread.sleep( timeToSleep );
} catch ( InterruptedException exc ) {
exc.printStackTrace();
}
}
}
}
}