I want to to render an application (e.g. a browser) in memory (not on screen) and stream the result of this rendering to a couple of remote desktops.
What options do I have? Is there a Java framework (maybe based on OpenGL) that I can use for this?
OpenGL is a drawing API; totally unsuited for your demands; just telling you to clear things up.
The whole "do the thing off-screen" requires to hook into the lower parts of the graphics system. With Java this means serious tinkering with the JNI and probably also some native binary code. IMHO not worth the effort, as there is a much nicer solution:
Xorg has a X server that is backed by a VNC framebuffer (Xvnc). You can start a browser in such and if there's no window manager and desktop environment running, the browser will be the only thing visible, no decorations, titlebar or the like. Then you connect using VNC to that server and will see the picture of the browser only. Technically VNC is just a video stream of JPEG images, and there are tools to create a regular video stream from VNC.
X provides you an additional possibility: The Composite extension. X Composite provides a mechanism to transfer a window into an offscreen rendering area. The contents rendered to the offscreen area must be composited to the screen by a so called compositor. It is possible, though quite inefficient, to copy those offscreen images into process memory and from there build a video stream.
Depending on what you are trying to do, you could run you application on a linux box and stream the output of the application over SSH to an xserver on another machine. I believe that there are xservers available for Windows and MacOS, but this gets you pretty far away from Java.
http://www.vanemery.com/Linux/XoverSSH/X-over-SSH2.html
Related
How can I make my Java Swing app GUI scale properly to users on high-DPI screens?
See the screenshot below. At the top you can see how tiny the app looks compared to Ubuntu's file explorer icons and gedit. In the bottom left you can see roughly what size the app should look like (and does look on regular DPI monitors). So I'm looking for a way to scale the GUI of the app properly when a high DPI monitor is in use. For example, gedit looks the same on both regular DPI and high DPI monitors. I want my app to behave like this.
Here is source code for the app: https://github.com/baobabKoodaa/baopass
This is an extremely common problem affecting many apps. Roughly half of the apps I run on Ubuntu are scaled properly without any actions from the user, the other half are not scaled and look really tiny. Note that I'm mainly looking for a solution that doesn't require actions from the user (although any help is appreciated at this point - I haven't found any ways to scale it at all).
According to this scaling should already work out of the box. It doesn't. I'm not sure if this is a bug or if there is some additional step I'm supposed to do besides running the app on Java 9?
You have to tell the drawing libraries to scale the app up.
GDK_SCALE=2 ./application
Will have the appropriate information set in the environment and the widgets will render scaled up such that each pixel effectively takes four pixels of footprint.
Note that the splash screen (if you use Java's splash screen support) isn't presented after the entire Swing libraries are loaded, so it won't scale regardless of the settings you attempt.
In some platforms, like the Linux distribution of Fedora, partial scaling is also possible, such that you can set GDK_SCALE=1.5. Just keep in mind that it's not universally available, and it is not settable to any scaling ratio you might want.
Finally, the older JVMs ignore this setting completely, so if you aren't launching with Java 9 or higher, odds are it won't work. And, of course, the way that things are tweaked for older JVMs and different operating systems tend to vary (if they work at all).
It looks like you're using Linux. You can use a command line switch
java -Dsun.java2d.uiScale=2 -jar some_application.jar
From https://wiki.archlinux.org/index.php/HiDPI#Java_applications
I am setting up a home server on an old PC, but I am not using a server like apache, instead making a really basic one in java. I have got a extremely basic linux kernel compiled and working on it. But I want a gui along with my program. Is it possible for me to do it without installing X11 or wayland on my system?Note: I dont have anything on my system apart from the necessary java files.
You can run Java applications on headless mode on server environment.
From Oracle docs:
Headless mode is a system configuration in which the display device, keyboard, or mouse is lacking. Sounds unexpected, but actually you can perform different operations in this mode, even with graphic data.
It means you can run some AWT graphic stuff, even without X11, but obviously, not visible in your screen. As example, you can create graphics with java.awt.Canvas and save/export as a image. Available AWT classes in headless mode are: Canvas, Font, Image (and subclasses), Print classes and Beep.
See detailed info here: Headless Java SE
Some context: I am hoping to make a program that can play a videogame based on visual input. While I could just point a webcam at my monitor whilst the game is on the screen, I'd prefer to just have some way to send whatever pixel information is going to the screen to my program.
Specifically, I'm hoping to sample screenshots at a rate ~30fps and compute on them. So far the only thing I can think of is to use Fraps to take screenshots whenever my program 'presses' a certain key, but those can only be taken at a maximum rate of one per second and require using this outside program. I'm hoping there is some way to intercept this screen information more directly.
I'm currently planning on using a combination of Java and Matlab, but I'd be happy to switch to whichever language has a nice way to grab screenshots rapidly. Oh, and I'm doing this on Windows 7, in case this screen grabbing operation is low-level enough for that to matter.
I'll take a stab at answering your question. The basic answer is that the screen data itself is memory mapped, so it resides somewhere in physical memory. However, your operating system has probably cut you off from accessing that memory directly through its virtual memory system. So, the only way to get access is to either:
a) Write a device driver to access it
or
b) Use a device driver written by someone else to access it
On Windows you could use DirectX to access the screen data. This website explains how to do that:
http://www.dgrigoriadis.net/post/2008/01/29/Taking-screenshots-with-DirectX-90.aspx
If it's a 2D Java game, you'd just send it to the framebuffer as well as your visual input program. If it's 3D and in Java, you could possibly do a similar thing with screenshotting functions in the library the game is written in. If it's a game in another window, you could try using java.awt.Robot (http://download.oracle.com/javase/1,5.0/docs/api/java/awt/Robot.html) and see if the screenshot capability works. You get a BufferedImage, which you would send to the visual input program (like if you were sending the game view from a Java 2D or 3D app).
You can do it , But with complex image processing, gesture recognition and machine learning algorithms.
And the response of the system should be real-time if you want to win that game which naturally makes your research complex.
But if the original game is developed by yourself , then you probably don't need any of image processing, ,webcam and FRAPS. In that case all you need is Machine Learning for game playing.
The goal is to use Processing as a scripting environment for creating graphics and to have the output display on a custom display device that is something like an LED light panel. The server running the program will be on a 1U rack. The idea is all the LED stuff is custom hardware, but rather than reinvent the wheel, it would be better to use an existing stack to drive the display. The problem is getting java to display on this device.
My initial thoughts are:
1. Run Java in headless mode.
2. Use Xvbf as the framebuffer.
3. Have a program run that reads the framebuffer, unpacks it, and then displays it on the remote device, at 30 fps.
4. Use Processing scripts to generate the graphics.
Does this make sense? Is there a better way? I'm not that knowledgeable about this area, but it seems better than trying to create a new java.awt.
I routinely use JFreeChart in headless mode with VNC to generate charts in servlets using ChartUtilities. It seems like you could just download the pictures to a picture frame via USB.
Another proposed option is to use Processing's createGraphics() and write the result to a file. I don't know about the tradeoffs of this option. It doesn't support OPENGL. And I'm concerned that the write will be a synchronous operation so calculations can't be going on during the write, which would make it difficult to get a high frame rate I would think.
If the "remote device" is just something directly attached by USB or some PCI controller, this seems sound (and exactly the sort of thing xvfb is made for). But if the remote device is something connected by ethernet or wifi, depending on it's resolution you might find the naive approach of copying all the data each frame requires too much bandwidth and before you know it you'll be rolling your own frame-differencing image compression. If you find yourself going down that route look at the VNC/TightVNC class of software which (at least in the form it's normally used on headless servers) provides an Xvfb-like virtual framebuffer/X-server accessible by a TCP/IP protocol which can transmit the content reasonably efficiently using compression and display it with VNC client software.
Hello I want write my own desktop sharing application in Java.
The application should have some very default features:
Capture screen;
Allow a remote connected user to click / edit fields.
I was thinking to use Java Robot class for mouse movements / key pressing.
The problem is i don't know what screen capture strategy to use.
Should I make sequentially screen captures (on the hosting computer) every second, and send those captures with UDP via network, so that the clients can intercept the data-grams ? Isn't this a little overkill for the network ?
What other strategies are available ? (Except trying an already existing app).
PS: If necessary I can even write native code using JNI (still that's the last thing I planning to do).
Later edit:
After some investigation I've come to the conclusion of #Thorbjørn Ravn Andersen . Java is probably not the best choice for this kind of application. I can try to use JNI, but that code will cover 75%+ of my project.
I will try to find other alternatives.
Have a good long look at the Ultra VNC project on SourceForge. Great place to start.
In pure Java you cannot access structured information on system windows, nor monitor all relevant system events, so the performance of the display synchronization will not be optimal. Also there are privileged windows, which do not accept mouse or key events from Robot. Remote video streaming is not an option!
With named restrictions your attempt with the Robot class is valid.
Robot.createScreenCapture(Rectangle) will put a Desktop section into a BufferedImage, which you can send to the client window.
On client side you can capture keyboard and mouse events and send them to a Robot object on the server side.
Without knowing the actual extent of system windows, it will makes sense to work on a grid of Desktop tiles:
captured BufferedImage-s of tiles may fit into the buffer of the transfer protocol.
capturing period may locally be optimized for tiles with high entropy (-> Capturing Strategy).
Further traffic minimization
Consider differences of subsequent Desktop contents, only
Compression of tiles by PNG or a PCX-like method
For sharing over internet, Peer-to-Peer connection may be established by
a public proxy server
UDP hole punching with, for connection establishing, a necessarily public mediator server
In any case the protocol needs to be secured and delivery asserted.