I'm learning about AI and (just for fun and practice, not profit or anything evil) I'd like to write a little bot to play a rinky-dink Flash game for me.
As a Microsoft hater, I'm embarrassed to say that I know and have used the required function calls some time ago under Windows, programming in Delphi. But these days I try to do everything under Linux and I don't seem to have any idea.
Requirements:
to grab part of the screen's contents in image or bitmap form, periodically or on programmatic request;
to simulate mouse pointer moves and left clicks under programmatic control;
keyboard grabbing might come in handy too.
Ideally, I'd like to do this from a Clojure application running in the JVM, so a Java compatible library would be the cat's meow. But failing that, I could write myself a C program to mediate between X Window and my app via local TCP/IP.
You can always rely upon the good old java.awt.Robot class, which, i think, has all the features you're asking :
moving mouse
capturing a Rectangle on screen
clicking mouse
Related
I have made a light weight java web-server serving html files and static content (made with pure java library nanohttpd), i have successfully made a javaFX launcher window that has a single button, clicking button simply runs the server in the background and opens the localhost URL in Android/PC browser (I was unsuccessful in making an IOS version using javaFX)
I am thinking of using libGDX as "launcher window" because of IOS support and access to mobile specific hardware like SMS/GPS which javaFX don't have.
I am targeting IOS/Android/PC, I'd like to ask libGDX developers how possible is this given my target platforms?
Yes you can
There's a catch though, you may have to build the UI the game dev way
I've made a game or two using it and had rather steep learning curve in the beginning to get a hold on to how it worked. For example you'll have to provide the images for button, background and also, the pressed view of the button... like that. But your app is not a game. So you won't have to worry that much.
Once you learned how to place them in the screen successfully, there is not much to worry about because the API provides everything you need to carry on from there.
Also I found enough resources/tutorials online enough to make a game from ground up. So you'll definitely can.
And there's very little to worry about your multi-platform problem.
I want to create java code which will stimulate a mouse click on any window of a windows OS application - eg browser, word etc.
Can someone suggest how I can begin ? I don't know which API's are needed for this.
I also need to know if this is a very complex task and will require more than just
core concepts of Java.
Thanks in advance.
The class you would want to use for this task is the Robot class: http://docs.oracle.com/javase/7/docs/api/java/awt/Robot.html
It may require more knowledge than the core concepts (depending on what you consider core), but Java is a good language to deal with this kind of thing.
Depending on what you seek to do with this knowledge, this may end up a very complex task. For example, if you sought to click on a specific button on the screen, you would need some way to analyze the screen (a very hard task). If you know where every button is already, you just move to the x, y location and cause a click to occur.
First of all, im not looking for a bot, that's clicking on the screen so i can't use my computer.
I'm curios on how to interact with flash games without having to use awt.robot to click the screen.
Is this possible in Java, and how can i do it.
I prefer java, as that's what im best at, but if c# has some huge advantages im willing to use that :)
First of all, im not looking for a bot, that's clicking on the screen
so i can't use my computer.
You're mistaken if you think using AWT Robot clicking on the "screen" means you can't use your computer.
One way to do what you want by actually using Java's Robot would simply be to run another graphical display and have your bot act on that graphical display.
There are a lot of ways to do that. A virtual machine would be one. Spawning a second X server would be another (under Un*x most likely). Configuring Windows so that it can accept several graphical logins at once and opening a "remote desktop" session, etc.
There are actually bots abusing Flash games and one way to do it is precisely by doing what I just described ; )
Some context: I am hoping to make a program that can play a videogame based on visual input. While I could just point a webcam at my monitor whilst the game is on the screen, I'd prefer to just have some way to send whatever pixel information is going to the screen to my program.
Specifically, I'm hoping to sample screenshots at a rate ~30fps and compute on them. So far the only thing I can think of is to use Fraps to take screenshots whenever my program 'presses' a certain key, but those can only be taken at a maximum rate of one per second and require using this outside program. I'm hoping there is some way to intercept this screen information more directly.
I'm currently planning on using a combination of Java and Matlab, but I'd be happy to switch to whichever language has a nice way to grab screenshots rapidly. Oh, and I'm doing this on Windows 7, in case this screen grabbing operation is low-level enough for that to matter.
I'll take a stab at answering your question. The basic answer is that the screen data itself is memory mapped, so it resides somewhere in physical memory. However, your operating system has probably cut you off from accessing that memory directly through its virtual memory system. So, the only way to get access is to either:
a) Write a device driver to access it
or
b) Use a device driver written by someone else to access it
On Windows you could use DirectX to access the screen data. This website explains how to do that:
http://www.dgrigoriadis.net/post/2008/01/29/Taking-screenshots-with-DirectX-90.aspx
If it's a 2D Java game, you'd just send it to the framebuffer as well as your visual input program. If it's 3D and in Java, you could possibly do a similar thing with screenshotting functions in the library the game is written in. If it's a game in another window, you could try using java.awt.Robot (http://download.oracle.com/javase/1,5.0/docs/api/java/awt/Robot.html) and see if the screenshot capability works. You get a BufferedImage, which you would send to the visual input program (like if you were sending the game view from a Java 2D or 3D app).
You can do it , But with complex image processing, gesture recognition and machine learning algorithms.
And the response of the system should be real-time if you want to win that game which naturally makes your research complex.
But if the original game is developed by yourself , then you probably don't need any of image processing, ,webcam and FRAPS. In that case all you need is Machine Learning for game playing.
I am creating an application that is essentially a financial alerts site. I am a basic level Java programmer, and I have created some of the logic for alerts in Java.
I want to be able to have pop-ups appear on the desktop whenever something "interesting" happens (interesting depends on %change, liquidity and a few other simple factors).
What is the best combo of technology to implement something like this?
I would use the java.awt.SystemTray in Java SE 6. It's cross-platform and pretty easy to use.
Although some people hate the balloon notifications in Windows, they're the least obtrusive popups, since they can be ignored by the user or easily dismissed. Most importantly, they can't be missed by the user who has been away from the computer, because balloons (at least in Windows XP/Vista) use system idle timers to determine when's the right time to disappear.
Some prefer more traditional toast notifications, similar to those shown by Outlook - they show up and slowly fade out, giving the user some time to interact with them if needed.
I had the same problem and finally solved it using an undecorated, alwaysOnTop window.
And thanks to this blog entry I found the TimingFramework, and now it even is translucent, fades in and out, goes 100% opaque on mouse over etc. In conjunction with the SystemTray and TrayIcon the behavior is nearly as that of Outlook.
Oh, I have to note, that other than the second link, I do the fading out with
AWTUtilities.setWindowOpacity(window, op);
You could write a java program that resides in the system tray, but I am not sure if there are cross platform compatible ways to do this. maybe you have to use a platform specific library for Win, Mac, Linux, ...
I'd just create a message window and animate it. Then add SystemTray support and voila, you're done.
In Delphi you can do that pretty quickly, but you can't easily reuse your java logic
You can just run you program in "silent" mode, without creating any windows by default, maybe just a little icon in the taskbar which when double-clicked will open a settings window. The program will be running in the background and creating windows with the set focus whenever an event happens.
But in my opinion, a slide window or at least a balloon tooltip is a better idea.