Hello I want write my own desktop sharing application in Java.
The application should have some very default features:
Capture screen;
Allow a remote connected user to click / edit fields.
I was thinking to use Java Robot class for mouse movements / key pressing.
The problem is i don't know what screen capture strategy to use.
Should I make sequentially screen captures (on the hosting computer) every second, and send those captures with UDP via network, so that the clients can intercept the data-grams ? Isn't this a little overkill for the network ?
What other strategies are available ? (Except trying an already existing app).
PS: If necessary I can even write native code using JNI (still that's the last thing I planning to do).
Later edit:
After some investigation I've come to the conclusion of #Thorbjørn Ravn Andersen . Java is probably not the best choice for this kind of application. I can try to use JNI, but that code will cover 75%+ of my project.
I will try to find other alternatives.
Have a good long look at the Ultra VNC project on SourceForge. Great place to start.
In pure Java you cannot access structured information on system windows, nor monitor all relevant system events, so the performance of the display synchronization will not be optimal. Also there are privileged windows, which do not accept mouse or key events from Robot. Remote video streaming is not an option!
With named restrictions your attempt with the Robot class is valid.
Robot.createScreenCapture(Rectangle) will put a Desktop section into a BufferedImage, which you can send to the client window.
On client side you can capture keyboard and mouse events and send them to a Robot object on the server side.
Without knowing the actual extent of system windows, it will makes sense to work on a grid of Desktop tiles:
captured BufferedImage-s of tiles may fit into the buffer of the transfer protocol.
capturing period may locally be optimized for tiles with high entropy (-> Capturing Strategy).
Further traffic minimization
Consider differences of subsequent Desktop contents, only
Compression of tiles by PNG or a PCX-like method
For sharing over internet, Peer-to-Peer connection may be established by
a public proxy server
UDP hole punching with, for connection establishing, a necessarily public mediator server
In any case the protocol needs to be secured and delivery asserted.
Related
Quite simply as the title states, is it possible for Java to read what is happening in a web-browser based Flash game?
For Example: Could I make a Java program which could play FarmVille for me by reading what is going on currently and make decisions based on a pre-determined set of actions?
FarmVille doesn't publish any sort of API, and you won't be able to access the RAM allocated to the Flash player (to my knowledge), so your only real course of action is to analyze the screen and automate input. In other words, there's no prewritten farmville.getUnplowedCells() to use, but you can write a program that takes screenshots, analyzes them to find the unplowed cells, and then generates the mouse/keyboard input that would make your character plow his unplowed cells. The Robot class can be used for this sort of programming.
This would be very time consuming and probably not worth the effort. Also as a side effect this would essentially "take control" of your mouse and keyboard and prevent you from using your computer normally. However this problem can be mitigated by running the program in a virtual machine.
No, there is no implementation for a Java-Flash bridge that allows anything more direct than IPC. Your best bet is decompilation (which would probably require deobfuscation) of the swf and interaction through common IPC mechanisms.
You could also tackle this using a different approach by recognizing elements on the screen by their pixel colors, for that you could use the Java Robot API which provides you with access to the keyboard, mouse and video. Since FarmVille is a 2D game, recognizing elements on the screen should be fairly straightforward -- performing image comparison, or partial image comparison (just the image's borders or a representative part) to increase the speed.
Aside, cheating in online games is not a nice thing to do and you can probably find activities that are more productive given you are smart enough to make software that works.
If you goal is to make a bot for FarmVille, then you're heading a wrong way. Keep in mind: it's ONLINE game. So, you don't have to read flash client's state, it'll be much more better, to read incoming packets, you don't have to simulate mouse/keyboard events in swf - just create correct packets, discribing your in-game actions. So I recomend you act like this:
1) Write a sniffer in Java (it's quite easy, because you have to write a simple one, not an ultimate tool, like Wireshark).
2) Use request's analytic (such as Firefox's Firebug) to catch some packets from your game. Reading them, you can retrieve all info, what you need: packets headers, game actions encoding, etc.
3) Code logic in your Java program: read incoming game packets and execute actions, sending requests.
4) Set user-agent options of your connection object similar to a popular browser (if you refuse doing that, game server may ignore your requests).
That's all. Remember: a game bot isn't an image recognizer, it's just another game client, with another GUI and logic.
P.S. If I misunderstood you, and your goal is to create a webpage with flash and applet, which comunicate with each other, you have to use JavaScript as a mediator between them. Flash calls JS using ExternalInterface.call, JS calls applet and vice versa.
I am talking about components that can be externally attached to a computer system via some port or other means, not about any of the component that is part of or peripheral of computer itself.
Actually, working on a college project for controlling traffic lights and boom barrier at railway crossing. I've got knowledge in Java but I do not know how can I get the traffic lights and boom barrier working on events in a Swing based application?
One thing is I can create a electronic circuit which can read the small output voltages at computer ports such as a USB port and used them as a trigger for controlling the devices. But how can I generate that small voltages using Java application?
Is JavaPOS can be the solution? or something else?
Any ideas? Suggestions? Articles? Samples?
I'd work backwards from the external device. Answer this 1st: What's the easiest way to communicate with it? If you say USB, ok, use usb. Then ask, what's the easiest way to interface with USB. Then build in whatever language you find to be easiest this USB interface. Finally, call from your Java swing application to this USB-wrapping application... it could be that simple invoke the app using something like ProcessBuilder.
In other words, I think it might be a mistake to solve the problem of interfacing to something like this device with Java, unless it's easy to do so directly.
Have you considered communicating with these external devices by sending digital signals to a serial port using Java? It's then a simple matter of either using those digital signals directly, or using an Analog-to-digital converter to get a voltage of desirable magnitude.
Same for input from the serial port. The RXTX library can help you do this (communicate with the serial port).
On the other hand, if you have access to MATLAB, then this sort of stuff is a piece of cake. Take a look at the Data Acquisition Toolbox and Instrument Control Toolbox.
I think your looking at this the wrong way. Most lights are them selves computer controlled. The lights are running on a computer system. If your project is to write this start to end, then you need to write a loaded to the light controller that does many things, one controls light color and direction, also allow connections via an out side computer. This connection could be USB, Ethernet ext. Now write a program facilitating connecting to the lights and pass commands to the light controller.
The goal is to use Processing as a scripting environment for creating graphics and to have the output display on a custom display device that is something like an LED light panel. The server running the program will be on a 1U rack. The idea is all the LED stuff is custom hardware, but rather than reinvent the wheel, it would be better to use an existing stack to drive the display. The problem is getting java to display on this device.
My initial thoughts are:
1. Run Java in headless mode.
2. Use Xvbf as the framebuffer.
3. Have a program run that reads the framebuffer, unpacks it, and then displays it on the remote device, at 30 fps.
4. Use Processing scripts to generate the graphics.
Does this make sense? Is there a better way? I'm not that knowledgeable about this area, but it seems better than trying to create a new java.awt.
I routinely use JFreeChart in headless mode with VNC to generate charts in servlets using ChartUtilities. It seems like you could just download the pictures to a picture frame via USB.
Another proposed option is to use Processing's createGraphics() and write the result to a file. I don't know about the tradeoffs of this option. It doesn't support OPENGL. And I'm concerned that the write will be a synchronous operation so calculations can't be going on during the write, which would make it difficult to get a high frame rate I would think.
If the "remote device" is just something directly attached by USB or some PCI controller, this seems sound (and exactly the sort of thing xvfb is made for). But if the remote device is something connected by ethernet or wifi, depending on it's resolution you might find the naive approach of copying all the data each frame requires too much bandwidth and before you know it you'll be rolling your own frame-differencing image compression. If you find yourself going down that route look at the VNC/TightVNC class of software which (at least in the form it's normally used on headless servers) provides an Xvfb-like virtual framebuffer/X-server accessible by a TCP/IP protocol which can transmit the content reasonably efficiently using compression and display it with VNC client software.
I am looking to create a video training program which records videos - via webcam, user screen capture and captures sound. Now the main problem is that I need a cross-platform (mac and windows) solutions.
I know its possible to use flash to record webcam + audio. But its not possible to record the user's screen via flash.
So am wonder if I should use Java (which i believe will work on mac & windows). I do not want to develop to separate versions because of the cost involved in developing two versions.
Please guide me as I am new to this.
Thank you.
UPDATE
Hello again,
I had a look at the following site: www.screencast-o-matic.com or www.screentoaster.com. I see that they have developed a java applet which helps interact with Windows/Mac to record the screen.
I am wondering how to go about developing something like that and integrating it with Flash (for webcam and audio recording).
Is this a better idea?
This is not an answer to your question, but I strongly recommend against using video for educational programmes. Our company delivers university courses on-line, and we long ago learned that video feeds are only effective under particular scenarios. In general, a talking head is a waste of bandwidth. You're much better off to put together a well designed powerpoint presentation, record a voice-over (and edit it!) and then assemble the whole thing as a flash presentation. This is a non-trivial amount of work, but it provides a much more interesting product for the student.
When to use video:
1) When you are demonstrating something dynamic - Mechanics or Chemistry for example.
2) When you are acting out a scenario or case as an illustration -- For example, threat de-escalation techniques for high school teachers.
When you solve the screen recording problem, seriously consider whether you need full motion or if you can get away with stills. Often the motion is distracting, and a still with good voice over can be more effective. (Hint: Replace mouse pointers with something HUGE before recording -- Like Fox did with hockey pucks)
Try CamStudio. I don't know, if it works on Mac, but on windows, it's the best solution I know. It's open source, so you can use it's source code, if you want to :)
If you're looking to build an application that does all of the recording and screen capture itself, then you might consider using Adobe AIR (essentially, Flash running on the desktop) in combination with Merapi. Merapi is essentially a bridge between Adobe AIR and Java. So for example, for your project, you might use Java to handle the lower-level (but still cross-platform) stuff you can't do natively in AIR, and use Merapi to wire the Java application to your AIR UI.
This is by no means a simple project. Lets get that said and out the way. There are open source (and cross-platform) options for each element, but nothing (I know of) that will do everything for you.
I think the "cleanest" option would be to use Flash for webcam and audio, as you said, and run a VNC server to send the screen video... The only closed-platform code will be the VNC launching code. That should be pretty simple to maintain!
That raises a problem because most people are behind NAT firewalls these days. Setting up port forwarding is a pain in the behind. I've used an app called Gitso before which allows people to connect to me and send their desktop to my screen (for tech support). Its VNC-based and all it really does is add another layer on top of the VNC connection so rather than me connecting to them, they connect to me. That makes the whole business of port forwarding a non-issue.
And once you've recorded everything, there's the final issue of syncing it all back together... Might not be so hard.
Well, Camtasia provides the solution to get your problem done. It can record the onscreen activity and also the webcam video and put them in the same player template. Another screen recorder DemoCreator can publish the screen recording as Flash movie, but can not record the webcam.
is it possible to create java application that will
work as background process on symbian smartphones?
You can approximate it but J2ME (the version of java on mobile phones) may not be the right technology to do this.
starting a MIDlet (a Java application for mobile phones) when the phone is switched on is tricky at best without coding a small Symbian OS C++ module that will start it for you. If you want to try anyway, look at the PushRegistry class in the MIDP specs
(http://java.sun.com/javame/reference/apis/jsr118/). The Content Handling API might provide some way to do it too (http://java.sun.com/javame/reference/apis/jsr211). When you are ready to give up, do it in C++.
Backgrounding a MIDlet isn't hard. The phone's "menu" key will do it for you. Programatically, Canvas.setCurrent(null) has a good chance of working. Trying to trick the phone by providing a fully transparent GUI and not handling any keypad activity will absolutely not work. Creating and starting a separate Thread in the MIDlet should allow you to keep something running even after your overload of MIDlet.pauseApp() has been called by the application management system.
The real issue is that the MIDlet will not have any Inter Process Communication system unless you make one. The usual way of doing that is a loopback socket connection over which you transfer data. Not a nice or efficient way of simulating IPC. Sharing a RMS record can only be done from within the same MIDlet suite (you can package several MIDlets into the same .jar file), I think. The code to create a provider/consumer data flow over a file connection is even uglier and will raise security issues.
Without any more information about what you want to use it for, my answer is : maybe but you should probably not try.
You will have in-built MIDP support for background MIDlets in MIDP 3.0 (http://jcp.org/en/jsr/detail?id=271). Don't hold your breath for devices to appear, however - might be some time.
(Note that a few Symbian OS devices have more than just MIDP - the S-E p990 for instance, https://developer.sonyericsson.com/site/global/products/phonegallery/p990/p_p990.jsp).
As already pointed out, it might be helpful to have more information on what product functionality you are trying to implement - often more than one way to skin a cat.