First of all, im not looking for a bot, that's clicking on the screen so i can't use my computer.
I'm curios on how to interact with flash games without having to use awt.robot to click the screen.
Is this possible in Java, and how can i do it.
I prefer java, as that's what im best at, but if c# has some huge advantages im willing to use that :)
First of all, im not looking for a bot, that's clicking on the screen
so i can't use my computer.
You're mistaken if you think using AWT Robot clicking on the "screen" means you can't use your computer.
One way to do what you want by actually using Java's Robot would simply be to run another graphical display and have your bot act on that graphical display.
There are a lot of ways to do that. A virtual machine would be one. Spawning a second X server would be another (under Un*x most likely). Configuring Windows so that it can accept several graphical logins at once and opening a "remote desktop" session, etc.
There are actually bots abusing Flash games and one way to do it is precisely by doing what I just described ; )
Related
I have made a light weight java web-server serving html files and static content (made with pure java library nanohttpd), i have successfully made a javaFX launcher window that has a single button, clicking button simply runs the server in the background and opens the localhost URL in Android/PC browser (I was unsuccessful in making an IOS version using javaFX)
I am thinking of using libGDX as "launcher window" because of IOS support and access to mobile specific hardware like SMS/GPS which javaFX don't have.
I am targeting IOS/Android/PC, I'd like to ask libGDX developers how possible is this given my target platforms?
Yes you can
There's a catch though, you may have to build the UI the game dev way
I've made a game or two using it and had rather steep learning curve in the beginning to get a hold on to how it worked. For example you'll have to provide the images for button, background and also, the pressed view of the button... like that. But your app is not a game. So you won't have to worry that much.
Once you learned how to place them in the screen successfully, there is not much to worry about because the API provides everything you need to carry on from there.
Also I found enough resources/tutorials online enough to make a game from ground up. So you'll definitely can.
And there's very little to worry about your multi-platform problem.
Quite simply as the title states, is it possible for Java to read what is happening in a web-browser based Flash game?
For Example: Could I make a Java program which could play FarmVille for me by reading what is going on currently and make decisions based on a pre-determined set of actions?
FarmVille doesn't publish any sort of API, and you won't be able to access the RAM allocated to the Flash player (to my knowledge), so your only real course of action is to analyze the screen and automate input. In other words, there's no prewritten farmville.getUnplowedCells() to use, but you can write a program that takes screenshots, analyzes them to find the unplowed cells, and then generates the mouse/keyboard input that would make your character plow his unplowed cells. The Robot class can be used for this sort of programming.
This would be very time consuming and probably not worth the effort. Also as a side effect this would essentially "take control" of your mouse and keyboard and prevent you from using your computer normally. However this problem can be mitigated by running the program in a virtual machine.
No, there is no implementation for a Java-Flash bridge that allows anything more direct than IPC. Your best bet is decompilation (which would probably require deobfuscation) of the swf and interaction through common IPC mechanisms.
You could also tackle this using a different approach by recognizing elements on the screen by their pixel colors, for that you could use the Java Robot API which provides you with access to the keyboard, mouse and video. Since FarmVille is a 2D game, recognizing elements on the screen should be fairly straightforward -- performing image comparison, or partial image comparison (just the image's borders or a representative part) to increase the speed.
Aside, cheating in online games is not a nice thing to do and you can probably find activities that are more productive given you are smart enough to make software that works.
If you goal is to make a bot for FarmVille, then you're heading a wrong way. Keep in mind: it's ONLINE game. So, you don't have to read flash client's state, it'll be much more better, to read incoming packets, you don't have to simulate mouse/keyboard events in swf - just create correct packets, discribing your in-game actions. So I recomend you act like this:
1) Write a sniffer in Java (it's quite easy, because you have to write a simple one, not an ultimate tool, like Wireshark).
2) Use request's analytic (such as Firefox's Firebug) to catch some packets from your game. Reading them, you can retrieve all info, what you need: packets headers, game actions encoding, etc.
3) Code logic in your Java program: read incoming game packets and execute actions, sending requests.
4) Set user-agent options of your connection object similar to a popular browser (if you refuse doing that, game server may ignore your requests).
That's all. Remember: a game bot isn't an image recognizer, it's just another game client, with another GUI and logic.
P.S. If I misunderstood you, and your goal is to create a webpage with flash and applet, which comunicate with each other, you have to use JavaScript as a mediator between them. Flash calls JS using ExternalInterface.call, JS calls applet and vice versa.
I'm working on a game which would work both on the web, as an applet, and on the Android phone, as an app.
Is that possible to do, and if so, what do I need to be aware of to make that work (i.e if there are any settings that I shouldn't hard code and instead determine them based on the user's device when the game is run, or any java libraries that I shouldn't use?).
Also, the game needs to accept touchscreen as input for the Android app. Is that possible to build into the same game which will also be run as an applet? May be so at run time, the applet decides whether to use Mouse or Touchscreen for the input when it is run?
Although Android apps are written in Java, the framework around the app is extremely different of the framework wrapped around an applet. You won't be able to have one .jar file that you can include as an applet and throw at an Android device because that's just not how it works.
You will however probably be able to create all the game logic and objects and have them in be shared with the applet code and android app. You can probably even get away with having them in one repository and project (although it's probably going to have to be an Android project that you then wedge in your app build scripts).
In order to tackle the different controls for your game you are probably going to have to abstract away the input, and have your game/level object have a call back like userHasPoked(int x, int y) and then have the applet call that method on click of the mouse and the android app calls it on touch (which is oddly still called onClick).
I think it'll be a long road, but much easier than rewriting the whole thing. It'll probably seem like a lot more work up front, but once you are done wedging your code into an applet and an Android app, you'll probably "never" have to touch that code again and can just keep adding to the game.
I wouldn't underestimate the task, but that sounds like a very fun programming exercise. Good luck!
What kind of game do you develop? It may be the better approach to develop an Javascript game.
That can be installed with phonegap (cordova) onto an android device.
Let me break this for you....
Model - The Business Logic and Data
View - The Display of the Output of the Model
Controller - On which the action is done.
The advantage of using this MVC architecture is that, you can keep the same model and keep changing the Views.
So keeping this idea in mind, you can have the same model for both the Web App and the Android App, and then implement each others Views to it respectively.
Some context: I am hoping to make a program that can play a videogame based on visual input. While I could just point a webcam at my monitor whilst the game is on the screen, I'd prefer to just have some way to send whatever pixel information is going to the screen to my program.
Specifically, I'm hoping to sample screenshots at a rate ~30fps and compute on them. So far the only thing I can think of is to use Fraps to take screenshots whenever my program 'presses' a certain key, but those can only be taken at a maximum rate of one per second and require using this outside program. I'm hoping there is some way to intercept this screen information more directly.
I'm currently planning on using a combination of Java and Matlab, but I'd be happy to switch to whichever language has a nice way to grab screenshots rapidly. Oh, and I'm doing this on Windows 7, in case this screen grabbing operation is low-level enough for that to matter.
I'll take a stab at answering your question. The basic answer is that the screen data itself is memory mapped, so it resides somewhere in physical memory. However, your operating system has probably cut you off from accessing that memory directly through its virtual memory system. So, the only way to get access is to either:
a) Write a device driver to access it
or
b) Use a device driver written by someone else to access it
On Windows you could use DirectX to access the screen data. This website explains how to do that:
http://www.dgrigoriadis.net/post/2008/01/29/Taking-screenshots-with-DirectX-90.aspx
If it's a 2D Java game, you'd just send it to the framebuffer as well as your visual input program. If it's 3D and in Java, you could possibly do a similar thing with screenshotting functions in the library the game is written in. If it's a game in another window, you could try using java.awt.Robot (http://download.oracle.com/javase/1,5.0/docs/api/java/awt/Robot.html) and see if the screenshot capability works. You get a BufferedImage, which you would send to the visual input program (like if you were sending the game view from a Java 2D or 3D app).
You can do it , But with complex image processing, gesture recognition and machine learning algorithms.
And the response of the system should be real-time if you want to win that game which naturally makes your research complex.
But if the original game is developed by yourself , then you probably don't need any of image processing, ,webcam and FRAPS. In that case all you need is Machine Learning for game playing.
I am looking to create a video training program which records videos - via webcam, user screen capture and captures sound. Now the main problem is that I need a cross-platform (mac and windows) solutions.
I know its possible to use flash to record webcam + audio. But its not possible to record the user's screen via flash.
So am wonder if I should use Java (which i believe will work on mac & windows). I do not want to develop to separate versions because of the cost involved in developing two versions.
Please guide me as I am new to this.
Thank you.
UPDATE
Hello again,
I had a look at the following site: www.screencast-o-matic.com or www.screentoaster.com. I see that they have developed a java applet which helps interact with Windows/Mac to record the screen.
I am wondering how to go about developing something like that and integrating it with Flash (for webcam and audio recording).
Is this a better idea?
This is not an answer to your question, but I strongly recommend against using video for educational programmes. Our company delivers university courses on-line, and we long ago learned that video feeds are only effective under particular scenarios. In general, a talking head is a waste of bandwidth. You're much better off to put together a well designed powerpoint presentation, record a voice-over (and edit it!) and then assemble the whole thing as a flash presentation. This is a non-trivial amount of work, but it provides a much more interesting product for the student.
When to use video:
1) When you are demonstrating something dynamic - Mechanics or Chemistry for example.
2) When you are acting out a scenario or case as an illustration -- For example, threat de-escalation techniques for high school teachers.
When you solve the screen recording problem, seriously consider whether you need full motion or if you can get away with stills. Often the motion is distracting, and a still with good voice over can be more effective. (Hint: Replace mouse pointers with something HUGE before recording -- Like Fox did with hockey pucks)
Try CamStudio. I don't know, if it works on Mac, but on windows, it's the best solution I know. It's open source, so you can use it's source code, if you want to :)
If you're looking to build an application that does all of the recording and screen capture itself, then you might consider using Adobe AIR (essentially, Flash running on the desktop) in combination with Merapi. Merapi is essentially a bridge between Adobe AIR and Java. So for example, for your project, you might use Java to handle the lower-level (but still cross-platform) stuff you can't do natively in AIR, and use Merapi to wire the Java application to your AIR UI.
This is by no means a simple project. Lets get that said and out the way. There are open source (and cross-platform) options for each element, but nothing (I know of) that will do everything for you.
I think the "cleanest" option would be to use Flash for webcam and audio, as you said, and run a VNC server to send the screen video... The only closed-platform code will be the VNC launching code. That should be pretty simple to maintain!
That raises a problem because most people are behind NAT firewalls these days. Setting up port forwarding is a pain in the behind. I've used an app called Gitso before which allows people to connect to me and send their desktop to my screen (for tech support). Its VNC-based and all it really does is add another layer on top of the VNC connection so rather than me connecting to them, they connect to me. That makes the whole business of port forwarding a non-issue.
And once you've recorded everything, there's the final issue of syncing it all back together... Might not be so hard.
Well, Camtasia provides the solution to get your problem done. It can record the onscreen activity and also the webcam video and put them in the same player template. Another screen recorder DemoCreator can publish the screen recording as Flash movie, but can not record the webcam.