Quite simply as the title states, is it possible for Java to read what is happening in a web-browser based Flash game?
For Example: Could I make a Java program which could play FarmVille for me by reading what is going on currently and make decisions based on a pre-determined set of actions?
FarmVille doesn't publish any sort of API, and you won't be able to access the RAM allocated to the Flash player (to my knowledge), so your only real course of action is to analyze the screen and automate input. In other words, there's no prewritten farmville.getUnplowedCells() to use, but you can write a program that takes screenshots, analyzes them to find the unplowed cells, and then generates the mouse/keyboard input that would make your character plow his unplowed cells. The Robot class can be used for this sort of programming.
This would be very time consuming and probably not worth the effort. Also as a side effect this would essentially "take control" of your mouse and keyboard and prevent you from using your computer normally. However this problem can be mitigated by running the program in a virtual machine.
No, there is no implementation for a Java-Flash bridge that allows anything more direct than IPC. Your best bet is decompilation (which would probably require deobfuscation) of the swf and interaction through common IPC mechanisms.
You could also tackle this using a different approach by recognizing elements on the screen by their pixel colors, for that you could use the Java Robot API which provides you with access to the keyboard, mouse and video. Since FarmVille is a 2D game, recognizing elements on the screen should be fairly straightforward -- performing image comparison, or partial image comparison (just the image's borders or a representative part) to increase the speed.
Aside, cheating in online games is not a nice thing to do and you can probably find activities that are more productive given you are smart enough to make software that works.
If you goal is to make a bot for FarmVille, then you're heading a wrong way. Keep in mind: it's ONLINE game. So, you don't have to read flash client's state, it'll be much more better, to read incoming packets, you don't have to simulate mouse/keyboard events in swf - just create correct packets, discribing your in-game actions. So I recomend you act like this:
1) Write a sniffer in Java (it's quite easy, because you have to write a simple one, not an ultimate tool, like Wireshark).
2) Use request's analytic (such as Firefox's Firebug) to catch some packets from your game. Reading them, you can retrieve all info, what you need: packets headers, game actions encoding, etc.
3) Code logic in your Java program: read incoming game packets and execute actions, sending requests.
4) Set user-agent options of your connection object similar to a popular browser (if you refuse doing that, game server may ignore your requests).
That's all. Remember: a game bot isn't an image recognizer, it's just another game client, with another GUI and logic.
P.S. If I misunderstood you, and your goal is to create a webpage with flash and applet, which comunicate with each other, you have to use JavaScript as a mediator between them. Flash calls JS using ExternalInterface.call, JS calls applet and vice versa.
Related
Some context: I am hoping to make a program that can play a videogame based on visual input. While I could just point a webcam at my monitor whilst the game is on the screen, I'd prefer to just have some way to send whatever pixel information is going to the screen to my program.
Specifically, I'm hoping to sample screenshots at a rate ~30fps and compute on them. So far the only thing I can think of is to use Fraps to take screenshots whenever my program 'presses' a certain key, but those can only be taken at a maximum rate of one per second and require using this outside program. I'm hoping there is some way to intercept this screen information more directly.
I'm currently planning on using a combination of Java and Matlab, but I'd be happy to switch to whichever language has a nice way to grab screenshots rapidly. Oh, and I'm doing this on Windows 7, in case this screen grabbing operation is low-level enough for that to matter.
I'll take a stab at answering your question. The basic answer is that the screen data itself is memory mapped, so it resides somewhere in physical memory. However, your operating system has probably cut you off from accessing that memory directly through its virtual memory system. So, the only way to get access is to either:
a) Write a device driver to access it
or
b) Use a device driver written by someone else to access it
On Windows you could use DirectX to access the screen data. This website explains how to do that:
http://www.dgrigoriadis.net/post/2008/01/29/Taking-screenshots-with-DirectX-90.aspx
If it's a 2D Java game, you'd just send it to the framebuffer as well as your visual input program. If it's 3D and in Java, you could possibly do a similar thing with screenshotting functions in the library the game is written in. If it's a game in another window, you could try using java.awt.Robot (http://download.oracle.com/javase/1,5.0/docs/api/java/awt/Robot.html) and see if the screenshot capability works. You get a BufferedImage, which you would send to the visual input program (like if you were sending the game view from a Java 2D or 3D app).
You can do it , But with complex image processing, gesture recognition and machine learning algorithms.
And the response of the system should be real-time if you want to win that game which naturally makes your research complex.
But if the original game is developed by yourself , then you probably don't need any of image processing, ,webcam and FRAPS. In that case all you need is Machine Learning for game playing.
The goal is to use Processing as a scripting environment for creating graphics and to have the output display on a custom display device that is something like an LED light panel. The server running the program will be on a 1U rack. The idea is all the LED stuff is custom hardware, but rather than reinvent the wheel, it would be better to use an existing stack to drive the display. The problem is getting java to display on this device.
My initial thoughts are:
1. Run Java in headless mode.
2. Use Xvbf as the framebuffer.
3. Have a program run that reads the framebuffer, unpacks it, and then displays it on the remote device, at 30 fps.
4. Use Processing scripts to generate the graphics.
Does this make sense? Is there a better way? I'm not that knowledgeable about this area, but it seems better than trying to create a new java.awt.
I routinely use JFreeChart in headless mode with VNC to generate charts in servlets using ChartUtilities. It seems like you could just download the pictures to a picture frame via USB.
Another proposed option is to use Processing's createGraphics() and write the result to a file. I don't know about the tradeoffs of this option. It doesn't support OPENGL. And I'm concerned that the write will be a synchronous operation so calculations can't be going on during the write, which would make it difficult to get a high frame rate I would think.
If the "remote device" is just something directly attached by USB or some PCI controller, this seems sound (and exactly the sort of thing xvfb is made for). But if the remote device is something connected by ethernet or wifi, depending on it's resolution you might find the naive approach of copying all the data each frame requires too much bandwidth and before you know it you'll be rolling your own frame-differencing image compression. If you find yourself going down that route look at the VNC/TightVNC class of software which (at least in the form it's normally used on headless servers) provides an Xvfb-like virtual framebuffer/X-server accessible by a TCP/IP protocol which can transmit the content reasonably efficiently using compression and display it with VNC client software.
Processing has Android support and it seems to be pretty awesome from my 10 minutes of playing with it. But I would like to make a regular (nongraphics) application like a twitter feed reader or something. So is there something like Processing that can do regular apps? Besides Titanium...
Basically I am looking for anything that will make coding for android easier, processing was so easy to get working that I was very happy with it, but it is for graphics only. Titanium didn't give me the same wow factor and it isn't open so that kind of takes away from it. What other tools are out there?
I'm going to give you the answer you are looking for and some advice.
Processing can do ANY of the things you are thinking about doing. If you want textboxes etc, you can use the Control P5 library. It's great. If you are an expert at Processing and just want to port over your Processing code to android, Processing for android is great.
But that's not what you want to do. You want to write an application. And you want to write it on Android. There are frameworks designed to give you a leg up in writing cross-platform mobile apps, but nothing is going to make writing an android application easier than learning Java and learning how the android stack works. It's actually really well designed and easy to follow once you start grokking "intents" and "bundles".
At the end of the day, you might even want to scale back a little further. Are you trying to write an application that needs to be used without internet access or that uses super special phone APIs? If you aren't, maybe you should try just writing your app as an html5 css3 website.
You can do plenty of input based stuff with processing. The original mouse events work as specified, except pass touches, but you can also access things like pressure and multiple fingers down. The hardware keys are also supported.
Hello I want write my own desktop sharing application in Java.
The application should have some very default features:
Capture screen;
Allow a remote connected user to click / edit fields.
I was thinking to use Java Robot class for mouse movements / key pressing.
The problem is i don't know what screen capture strategy to use.
Should I make sequentially screen captures (on the hosting computer) every second, and send those captures with UDP via network, so that the clients can intercept the data-grams ? Isn't this a little overkill for the network ?
What other strategies are available ? (Except trying an already existing app).
PS: If necessary I can even write native code using JNI (still that's the last thing I planning to do).
Later edit:
After some investigation I've come to the conclusion of #Thorbjørn Ravn Andersen . Java is probably not the best choice for this kind of application. I can try to use JNI, but that code will cover 75%+ of my project.
I will try to find other alternatives.
Have a good long look at the Ultra VNC project on SourceForge. Great place to start.
In pure Java you cannot access structured information on system windows, nor monitor all relevant system events, so the performance of the display synchronization will not be optimal. Also there are privileged windows, which do not accept mouse or key events from Robot. Remote video streaming is not an option!
With named restrictions your attempt with the Robot class is valid.
Robot.createScreenCapture(Rectangle) will put a Desktop section into a BufferedImage, which you can send to the client window.
On client side you can capture keyboard and mouse events and send them to a Robot object on the server side.
Without knowing the actual extent of system windows, it will makes sense to work on a grid of Desktop tiles:
captured BufferedImage-s of tiles may fit into the buffer of the transfer protocol.
capturing period may locally be optimized for tiles with high entropy (-> Capturing Strategy).
Further traffic minimization
Consider differences of subsequent Desktop contents, only
Compression of tiles by PNG or a PCX-like method
For sharing over internet, Peer-to-Peer connection may be established by
a public proxy server
UDP hole punching with, for connection establishing, a necessarily public mediator server
In any case the protocol needs to be secured and delivery asserted.
I am looking to create a video training program which records videos - via webcam, user screen capture and captures sound. Now the main problem is that I need a cross-platform (mac and windows) solutions.
I know its possible to use flash to record webcam + audio. But its not possible to record the user's screen via flash.
So am wonder if I should use Java (which i believe will work on mac & windows). I do not want to develop to separate versions because of the cost involved in developing two versions.
Please guide me as I am new to this.
Thank you.
UPDATE
Hello again,
I had a look at the following site: www.screencast-o-matic.com or www.screentoaster.com. I see that they have developed a java applet which helps interact with Windows/Mac to record the screen.
I am wondering how to go about developing something like that and integrating it with Flash (for webcam and audio recording).
Is this a better idea?
This is not an answer to your question, but I strongly recommend against using video for educational programmes. Our company delivers university courses on-line, and we long ago learned that video feeds are only effective under particular scenarios. In general, a talking head is a waste of bandwidth. You're much better off to put together a well designed powerpoint presentation, record a voice-over (and edit it!) and then assemble the whole thing as a flash presentation. This is a non-trivial amount of work, but it provides a much more interesting product for the student.
When to use video:
1) When you are demonstrating something dynamic - Mechanics or Chemistry for example.
2) When you are acting out a scenario or case as an illustration -- For example, threat de-escalation techniques for high school teachers.
When you solve the screen recording problem, seriously consider whether you need full motion or if you can get away with stills. Often the motion is distracting, and a still with good voice over can be more effective. (Hint: Replace mouse pointers with something HUGE before recording -- Like Fox did with hockey pucks)
Try CamStudio. I don't know, if it works on Mac, but on windows, it's the best solution I know. It's open source, so you can use it's source code, if you want to :)
If you're looking to build an application that does all of the recording and screen capture itself, then you might consider using Adobe AIR (essentially, Flash running on the desktop) in combination with Merapi. Merapi is essentially a bridge between Adobe AIR and Java. So for example, for your project, you might use Java to handle the lower-level (but still cross-platform) stuff you can't do natively in AIR, and use Merapi to wire the Java application to your AIR UI.
This is by no means a simple project. Lets get that said and out the way. There are open source (and cross-platform) options for each element, but nothing (I know of) that will do everything for you.
I think the "cleanest" option would be to use Flash for webcam and audio, as you said, and run a VNC server to send the screen video... The only closed-platform code will be the VNC launching code. That should be pretty simple to maintain!
That raises a problem because most people are behind NAT firewalls these days. Setting up port forwarding is a pain in the behind. I've used an app called Gitso before which allows people to connect to me and send their desktop to my screen (for tech support). Its VNC-based and all it really does is add another layer on top of the VNC connection so rather than me connecting to them, they connect to me. That makes the whole business of port forwarding a non-issue.
And once you've recorded everything, there's the final issue of syncing it all back together... Might not be so hard.
Well, Camtasia provides the solution to get your problem done. It can record the onscreen activity and also the webcam video and put them in the same player template. Another screen recorder DemoCreator can publish the screen recording as Flash movie, but can not record the webcam.