We Keep Coding

Recognizing text on screen with Java

I am building a program that records a "kill feed" in the top right of a full-screen game and performs two actions as a result:
Update a counter I've painted to the screen with an always-on-top, transparent window that represents the number of players alive on each team (in the style of "2 v 1"), and
render a "hitmarker" graphic when my in-game username appears in the killfeed, indicating that I've killed my opponent
I've been programming in Java for three years, but I still don't know how to achieve this. I am able to use Graphics2D to paint graphics such as a "hitmarker" and draw strings such as the player count I mentioned above. However, I do not know how to recognize colors or text on screen, aside from Java's built-in Java's built-in Robot's getPixelColor() method (which I've never used before).
My best guess is to use java.util.Robot's createScreenCapture() method to save an image of the screen 60 times per second and do some OCR magic to recognize my username, then perform the two actions I detailed above.
Because I'm relatively inexperienced, specific and detailed answers would be much appreciated.
EDIT: I know it's not feasible to save the screen 60 times a second. My intention was for another method to be proposed. Also, as clarification, I am drawing over a game, and reading a killfeed that is a part of the game, not my program. I'm not rendering the feed, just the player counter and hitmarker (killmarker) because it's not a feature of the game like the killfeed is.

Double buffering and libgdx

I have a very simple application in libgdx. It currently is supposed to cover half of the screen with pink texture. And it does that, the texture however is blinking. Basing on my own understanding of subject, as well as this thread:
LibGDX blinking
I need to enable double buffering. Or alternatively, write to my buffer. The thing is, either I am inept with google or it's very difficult to find any information how to manipulate buffers in libgdx. If someone could provide me with resources, explaining how to manipulate buffers, and how to create one, how to swap them with current display etc. I would be very grateful.
EDIT:
After searching the web some more, I found post on badlogic forum, claiming that libgdx has double buffering set by default. The question is, how does it work? When are the buffers swapped?

Libgdx has double buffering enabled by default (its really part of the underlying OpenGL infrastructure, not Libgdx). I don't think Libgdx exposes any controls over double buffering. There are some controls to disable vsync which might create some tearing, but are unlikely to be related.
The more likely way to get this behavior is if, somehow, your render method is only invoked once. This will draw into one buffer. Then, assuming render has become a no-op, nothing is drawn into the other buffer, they're switched, nothing is drawn into the original buffer, they're switched, etc. This can lead to the blinking behavior you're seeing.
If that guess doesn't cover it you'll have to provide more details. Which platform are you running on? Are you running a stable version of Libgdx or one of the nightly builds? How are you setting up the libgdx app configuration? What does your render method actually look like?

How to learn mouse movement?

I've been attempting to develop a means of synthesizing human-like mouse movement in an application of mine for the past few weeks. At the start I used simple techniques like polynomial and spline interpolation, however even with a little noise the result still failed to appear sufficiently human-like.
In an effort to remedy this issue, I've been researching into ways of applying machine learning algorithms on real human mouse movement biometrics in order to synthesize mouse movements by learning from recorded real human ones. Users would be compiling a profile of recorded movements that would trainh= the program for synthesis purposes.
I've been searching for a few weeks and read several articles on application of inverse biometrics in generating mouse dynamics, such as Inverse Biometrics for Mouse Dynamics; they tend to focus, however, on generating realistic time from randomly-generated dynamics, while I was hoping to generate a path from specifically A to B. Plus, I still need to actually need to come up with a path, not just a few dynamics measured from one.
Does anyone have a few pointers to help a noob?
Currently, testing is done by recording movements and having I and several other developers watch the playback. Ideally the movement will be able to trick both an automatic biometric classifier, as well as a real, live, breathing Homo sapien, too.

Fitt's law gives a very good estimation of the time needed to position the mouse pointer. In the derivation section there is a simple explanation I think you could use this as one of the basic building blocks of your app. Start with big movements, put some inacurracy both in the direction and the length of the movement, then do a smaller correction movement and so on...

First, i guess you record human mouse movements from A to B. Because otherwise, trying to synthesize a model for such movement does not seem possible to me.
Second, how about measuring the deviations from the "direct" path, maybe in relation to time. I actually suspect that movements look different for different angles, path lengths etc., but maybe you can try a normalized model first, that you just stretch (in space and time) and rotate like you need it.
Third, the learning. The easiest thing would be to just have a collection of real moves (in the form i discussed above), and sample from that collection. Evaluate how that looks like. If you really want a probabilistic model, then you have to evaluate what kind of models fit. is it enough to blurr the direct path with gaussian noise whose parameters you learn from your training set? Or some (sin-)wavy deviation? Or seperate models for "getting near the button" and "final corrections". Fitts law might be useful for evaluation.

This question reminded me of a website I knew about years ago, so I visited it and found this in-depth discussion on the topic.
The timing is so similar as to make me think this question is related in some way. In fact, someone in the thread linked to the same article you did. If it's not related, well, there's a link to a lot of people discussing exactly what you're thinking about.

I don't think the problem is all that well defined. There is a important notion not mentioned so far, which is context. The mouse movement on my screen when Chrome has focus is massively different that the motion when Vim has focus.

The way a mouse moves varies based on the type of the device, the type of action, the UI elements involved, familiarity with the UI, the speed at which the user is attempting to complete their task, the skill of the user, initial failure of the user (eg miss-clicks), the user's emotional state (as well as many other factors). Do you plan on creating several pathing strategies to correspond to different contexts? Also how well do you know the algorithm you are trying to fool? I assume not extensively or you would simply program directly against that algorithm.
If a human is looking at the pathing, they might be able to identify the state associated with a pathing strategy and may be more inclined to be fooled if they identify it as a human state (eg user is rushing, miss-clicks, quickly closes a resulting popup, tries again slower). UI comes into play with not just size and position. I often quickly point to a toolbar, then slide across the options until I get to my target. Another example is that I typically pause on menu items while I am scanning for my target or hover over text I am reading. Are you attempting to emulate human behavior or just their mouse movements (because I think they are joined at the hip)?

Are you wanting to simulate human-like mouse movement because you are doing real-time online training for your game? If your training sequences are static, just record your mouse movements and play a mouse clicking sound effect whenever you click the mouse button. No mouse movement is going to feel "real enough" to you more than your own.
Personally, I feel experts in software move their mice too quickly in training videos. I prefer an approach taken by screencast video software I've seen that always moves the mouse linearly from point A --> B. The trick was, every mouse move made in the video always took the same amount of time regardless of distance, say 3/4 of a second and then followed by a mouse click sound effect.
I believe they moved the mouse in this way because then the viewer could anticipate the landing area of the mouse by the direction and velocity the mouse moved at the start. In a training situation, I suppose that regular movements like this are gentler on the eye and perhaps easier to retain/recall.

Have you considered adding mouse tracking to your application so you essentially record how the user moves the mouse and then analyze the recordings?
I have not looked into this recently but I believe that a MouseListener in a Swing application get the information you need.

Java graphics performance

I was programming a simple game and wanted to display it via the Java Graphics + Swing API. It felt a little slow, of course, so I measured how long it took to repaint, which was around 32ms. Then I read about accelerated Java graphics and used the method described here: Space Invaders
However, somehow this is even slower. It now takes around 98ms to redraw. Why is that?
Note that I am aware of libraries like LWGL and JOGL, but I don't want to use a full-blown OpenGL wrapper for such a graphically simple game.

A few hints to improve performance:
Clip and just draw the area that has changed
Don't scale anything when drawing
Use the correct buffer strategy
Use full screen exclusive mode if appropriate.
Swing isn't inherently slow. Most of the confusion about Swing comes from people who don't know or understand about the Event Dispatch Thread (and yes, it's a lot of effort to get it right).
Regarding your times, are you just calling g.drawImage between the time settings? If so, what size are you repainting, and are you doing any scaling on this call?
EDIT: Forgot to mention Pulp Core - a very nice gaming framework for Java.

Read this about timers
Use an always sleeping thread in the background to lower the system interrupt rate to get much more precise timers!
You can take a look at my code in here
A small game I've created can be found here
long renderStart = System.nanoTime()
// render stuff here
long renderTime = (System.nanoTime() - renderStart) / 1000000;

Swing will be slow for a game. Your best bet is probably an SDL wrapper, such as jsdl or sdljava. SDL is quite lightweight and supported on many platforms.
The nice thing is, well-implemented 2d graphics library wrappers implement the Graphics interface, so you should be able to try several and see which one performs best for your app without modifying your code.

I am not an expert, but Java2D supports different rendering pipelines. On my machine, switching to the OpenGL one by setting the system property
sun.java2d.opengl=true
yielded a substantial increase in rendering speed.

You may look at processing.

I agree with Nicolas, for graphics / interactivity processing is great

Try JGame. It's a simple engine for 2D games which uses GL when its available and can produce games that run on anything from a mobile phone to a desktop system.

First of all, you should check and see how much time is spent in your own code vs. how much time is spent actually drawing the frame. You may have some bug or glitch that makes it run more slowly.
You should be able to get much better performance then 98ms to draw your screen. Those game libraries will probably be using the same graphics calls as you would. But which containers you use can have a big impact on how fast things draw. I remember a couple months ago working on some double buffered graphics code, and how the back buffer was created made a huge difference in performance.
In particular, make sure you only create background image once or at least, only when your canvas resizes. In my draw code I do this:
//create a graphics backplane
if(backplane == null || !backplaneSize.equals(this.getSize())){
backplane = this.createImage((int)this.getSize().getWidth(), (int)this.getSize().getHeight());
backplaneSize = this.getSize();
}
So the back plane only gets created if it's new, or if the component is resized. This has a huge impact on speed.
I've been doing graphics programming in java since JDK 1.0. Actually I'd never heard of this BufferStrategy thing. Heh.
I've never had any trouble getting good frame rates with basic java graphics calls. Another thing to look out for is making sure that Swing isn't trying to clear the background of your component for you. That can be another source of slowdown.

Java2D performance is a bit of a dark art. It can vary between platforms, but it possible to get it to perform well.
Sometimes doing seemingly innocuous things can result in much slower performance. You want to make sure that you are using accelerated images as much as possible.
The type of the image can also be significant. I don't know the full details, but I do know that my programs were much faster using type 1 images (INT RGB) than other types such as type 5 (3BYTE BGR) because of conversion issues.

I can't tell exactly what you are doing. Your question is about Swing but you mention a Canvas and Bufferstrategy. Well in Swing you would use a JPanel which is automatically double buffered so you don't have to worry about that.
I tested a simple animation with 1000 moving balls. The Timer was set to fire every 100ms. When the Timer fired, the 1000 ball where randomly moved to a new position and a new BufferedImage was created and the panel displaying the image was repainted. The difference in time between the repainting was 110ms implying it only took 10ms to create an new BufferedImage and do the painting. The painting was done a a full screen so no clipping was done.
This posting show the example code I tested.

How to generate simple 2D graphics in real time?

For my internship on Brain-Computer Interfacing I need to generate some very fast flickering squares on a CRT monitor (flickering = alternating between two colors). The monitor's refresh rate is 85Hz and we would like this to be the bottleneck, which means that repainting all squares can take at most 1000/85 = 11ms.
My language of preference for GUI/graphics programming is Java, so I tried making a prototype with AWT, because it's synchronous (unlike Swing). I now seem to have two problems: the first is that time measurements show that the repainting of even 9 squares simply takes too long. My algorithm takes the desired frequency and calculates the times at which the system should repaint in advance and then uses a loop (with no sleep/wait delay) that checks everytime if the next 'time' was reached and if so, loops through all the squares to repaint them. The way I implemented it now is that the squares are Panels with background color A and are contained in another Panel with background color B and the flickering happens because the Panels' visibility is changed. I figured that this would be faster than one Panel that has to draw Rectangles all the time.
I don't have a decent profiling tool (can't get Eclipse TPTP or NetBeans profiler to work) so I can't be sure, but I have the feeling that the bottleneck is actually not in the repainting, but in the looping (with conditional checking etc.). Can you recommend anything about what I should do?
The second problem is that it seems like the squares are rendered top-to-bottom. It's like they unroll really fast, but still visibly. This is unacceptable. What I'm wondering though, is what causes this. Is it Java/AWT, or Windows, or just me writing a slow algorithm?
Can you recommend some things for me to try? I prefer to use Java, but I will use C (or something) if I must.

I would avoid any kind of high-level "components", like JPanels and the like. Try getting a Graphics2D representing the window's contents, and use its fillRect() method.
Failing that, you could probably do this easy enough in C and OpenGL. rasonly.c is a standard template program that sets up OpenGL to work as a "rasterizer" only, i.e. 2D mode. Using this as a starting point, you should be able to get something running that draws your desired "squares" without too much trouble.
You don't describe your desired scene very well, it sounds from that equation as if you want to draw 100 squares, each having a different color. For maximum performance in OpenGL, you should draw all squares of the same color together, to minimize the "state changes" between drawing calls. This is probably a purely theoretical point though, as drawing 100 2D squares at 85 Hz really shouldn't tax OpenGL.
UPDATE: Oh, so it's been a bunch of years, and nowadays you probably need to take the above with a grain of salt and read some modern tutorial. Things have changed. Look up the Vulkan API.

(I remember a demonstration of this using the BBC micro and palette switching, though that would be 50fps rather than 85, as it was a British domestic TV)
I'd switch to jogl and use display lists. They get very high fps rates in Java.