Screen Capture in separate thread making Java application slow/unresponsive - java

I'm working on an application that records the users screen, webcam and microphone whilst he/she is performing certain activities. It will be used for research purposes. The application has been successfully tested on Windows, but on Mac OS X (Maverick with Java 7.0.45) the application becomes slow and unresponsive when recording is started.
This is why I find this difficult to comprehend:
The recording is done in a separate thread, so how could it influence the responsiveness of another thread? Especially as after each run either Thread.yield() or Thread.sleep(...) are called.
Logs show that whilst attempting to record at 15 FPS, the resulting frame rate was 2 FPS. So it seems the code that does the capturing of a single frame might be too slow. But why then does it work fine on Windows?
Just a quick note: the application was successfully tested by tons of users on Windows, but I only got to test it on a single Mac. However, that one was just formatted and got a clean install of OS X Maverick, Java (and Netbeans).
Below you will find the code that records the screen and writes it to a video using Xuggler. The code for recording the webcam is similar, and I'd doubt recording the audio has anything to do with it. My question is:
What might be the cause of the application becoming unresponsive?, and
How could the code be made more efficient and so improve FPS?
IMediaWriter writer = ToolFactory.makeWriter(file.getAbsolutePath());
Dimension size = Globals.sessionFrame.getBounds().getSize();
Rectangle screenRect;
BufferedImage capture;
BufferedImage mousePointImg;
writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264, size.width, size.height);
int i = 0;
while (stop == false) {
// Get mouse cursor to draw over screen image.
PointerInfo mousePointer = MouseInfo.getPointerInfo();
Point mousePoint = mousePointer.getLocation();
Point screenPoint = new Point((int) (mousePoint.getX() -
Globals.sessionFrame.getBounds().getX()), (int) (mousePoint.getY() -
Globals.sessionFrame.getBounds().getY()));
// Get the screen image.
try {
screenRect = new Rectangle(Globals.sessionFrame.getBounds());
capture = new Robot().createScreenCapture(screenRect);
} catch ( ... ) { ... }
// Convert and resize the screen image.
BufferedImage image = ConverterFactory.convertToType(capture,
BufferedImage.TYPE_3BYTE_BGR);
IConverter converter = ConverterFactory.createConverter(image,
IPixelFormat.Type.YUV420P);
// Draw the mouse cursor if necessary.
if (mouseWithinScreen()) {
Graphics g = image.getGraphics();
g.drawImage(mousePointImg, (int) screenPoint.getX(),
(int) screenPoint.getY(), null);
}
// Prepare the frame.
IVideoPicture frame = converter.toPicture(image, (System.currentTimeMillis() -
startTimeMillis()) * 1000);
frame.setKeyFrame(i % (getDesiredFPS() * getDesiredKeyframeSec()) == 0);
// Write to the video
writer.encodeVideo(0, frame);
// Delay the next capture if we are at the desired FPS.
try {
if (atDesiredFPS()) {
Thread.yield();
} else {
Thread.sleep(1000 / getDesiredFPS());
}
} catch ( ... ) { ... }
i++;
}
writer.close();

There are several architectural issues that I can see in your code:
First if you want to execute something at a fixed rate, use the ScheduledThreadPoolExecutor.scheduleAtFixedRate(...) function. It will make your entire delay code part obsolete as well as ensuring that certain OS timing issues will not interfere with your scheduling.
Then to make things faster you need to take your code apart a bit. As far as I can see you have 3 tasks: the capture, the mouse-drawing/conversion and the stream writing. If you put the capture part in a scheduled Runnable, the conversion into multi-parallel execution as Callables into an Executor, and then in a 3rd thread take the results from a result list and write it into the stream, you can fully utilize multi-cores.
Pseudocode:
Global declarations (or hand them over to the various classes):
final static Executor converterExecutor = Executors.newFixedThreadPoolExecutor(Runtime.getRuntime().availableProcessors());
final static LinkedBlockingQueue<Future<IVideoPicture>> imageQueue = new LinkedBlockingQueue<>();
// ...
Capture Runnable (scheduled at fixed rate):
capture = captureScreen();
final Converter converter = new Converter(capture);
final Future<IVideoPicture> conversionResult = converterExecutor.submit(converter);
imageQueue.offer(conversionResult); // returns false if queue is full
Conversion Callable:
class Converter implements Callable<IVideoPicture> {
// ... variables and constructor
public IVideoPicture call() {
return convert(this.image);
}
}
Writer Runnable:
IVideoPicture frame;
while (this.done == false) {
frame = imageQueue.get();
writer.encodeVideo(0, frame);
}
You can ensure that the imageQueue does not overflow with images to render if the CPU is too slow by limiting the size of this queue, see the constructor of LinkedBlockingQueue.

Related

Timer problems with swing GUI components

This snippet of code essentially reveals the back side of the cards when clicked. The showFace function sets the icon and text and therefore implying the front of the card. Light gray background is the back. If a non matching card is clicked, I first intend to showFace for a brief moment ( 2 seconds) than revert to the "back side of the card." However, upon clicking a non matching card, the icon and text flashes instantaneously and reverts to its gray background.
Tried changing the 2000 milliseconds to something higher but no avail. Any help is appreciated.
else if (currentMode == 1){
//matched cards
if(checkCards(currentCard, names)){
showFace();
currentMode = 0;
currentCard = "";
deafTo(this);
}
//else non match, still checking mode
else{
showFace();
var timer: Timer = null;
val action = new ActionListener(){
override def actionPerformed(event : ActionEvent){
icon = null;
text = "";
background = Color.DARK_GRAY;
timer.stop();
}
};
timer = new Timer (2000, action);
timer.setInitialDelay(0);
timer.start();
}
}
def showFace()={
text = names;
horizontalTextPosition = Alignment.Center;
verticalTextPosition = Alignment.Bottom;
background = Color.WHITE;
val icons = new ImageIcon(path);
val newIc = icons.getImage();
val newIcons = newIc.getScaledInstance(100, 75,
java.awt.Image.SCALE_SMOOTH);
icon = new ImageIcon(newIcons);
repaint();
}
This is because you set an initial delay of 2000ms in the constructor
timer = new Timer(2000, action)
But then you overwrite it to 0ms by:
timer.setInitialDelay(0);
Remove this line and you should be good.
You can check here Swing Timer API.
And see some examples here.
Javadoc http://docs.oracle.com/javase/7/docs/api/java/util/Timer.html
This class does not offer real-time guarantees: it schedules tasks using the Object.wait(long) method.
Java 5.0 introduced the java.util.concurrent package and one of the concurrency utilities therein is the ScheduledThreadPoolExecutor which is a thread pool for repeatedly executing tasks at a given rate or delay. It is effectively a more versatile replacement for the Timer/TimerTask combination, as it allows multiple service threads, accepts various time units, and doesn't require subclassing TimerTask (just implement Runnable). Configuring ScheduledThreadPoolExecutor with one thread makes it equivalent to Timer.

Slow Down Object movement in java JOGL

Using Thread.sleep(milliseconds) would delay execution of the whole program for the specified milliseconds. My problem is how do I slow down (not delay) object movement from one container into another container in my java code (implemented using JOGL - from canvas context) , so that the movement is noticeable which otherwise happens so fast?
Here is my approach for the problem: I use Stack to represent the amount of objects on each containers. Whenever user clicks on either of the container[source], the containers Stack is to be popped until it is empty. At the same time Stack of the destination container is to be pushed. for each iteration, the containers are redrawn containing objects represented by their corresponding Stack size.
fraction of the code relevant for this Question:
public void moveObjects(Container source, Container destination) {
while (!source.stack.isEmpty()) {
destination.stack.push(destination.stack.size() + 1);
source.stack.pop();
//redraw both containers with their new amount of objects represnted using their stack size
}
}
or
public void moveObject(int source, int dest) {
while (!this.container[source].stack.isEmpty()) {
this.container[dest].stack.push(this.container[dest].stack.size() + 1);
this.container[source].stack.pop();
try {
Thread.sleep(100);
} catch (InterruptedException ie) {
// Handle exception here
}
}
}
I have only managed to move the objects from the source to destination one by one but the movement is too fast to be noticed by eye. How do I aproach this problem?
If I understand you correctly you want to move multiple objects visibly on the screen without this happening instantaneous.
As you have noticed Thread.sleep(time) is a bad idea as it freezes the whole application.
A way around this is to supply your update logic with the elapsed time:
long lastTime = System.nanoTime();
while(running)
{
long now = System.nanoTime();
updateLogic(now-lastTime); //this handles your application updates
lastTime = now;
}
void updateLogic(long delta)
{
...
long movementTimespan +=delta
if(movementTimespan >= theTimeAfterWhichObjectsMove)
{
//move objects here
//...
movementTimespan -= theTimeAfterWhichObjectsMove;
}
//alternatively you could calculate how far the objects have moved directly from the delta to get a smooth animation instead of a jump
//e.g. if I want to move 300 units in 2s and delta is x I have to move y units now
}
This allows you to track the time since the last update/frame and if you use it for animation it makes the speed of the object movement independent from the executing systems spees.

Timer loop to constantly loop series of images [Java-NetBeans]

I'm trying to create a loop using Java Swing Timer to constantly cycle through a set of images (i1, i2, i3....in where n is total number of images).
Each of the images is exactly the same size and must be displayed on a Label (say, l1).
There must be a delay of ten seconds between each image being displayed.
Any idea how I can go about this without using the Java TumbleItem applet> It's seems much too complicated for a simple implementation such as mine. (Displaying special deals posters on an online storefront application for school).
I am open to this being achieved in any other way.
Help would be greatly appreciated. Thanks in advance!
I'm trying to create a loop using Java Swing Timer to constantly cycle through a set of images
When you use a Timer you don't use a loop. When the Timer fires you just change the image. So somewhere you would need to keep a List of the images to display and an index of the currently displayed image.
Any idea how I can go about this without using the Java TumbleItem applet> It's seems much too complicated for a simple implementation such as mine
How is it complicated? It displays a series of images, which is close to what you want.
Yes, there is some extra code that loads the images and doesn't start the animation until all the images are loaded. So you could easily simplify the code by not worry about that. Also, there is code that does animation from from left-to-right and then right-to-left. You also don't need that part of the code. Also, there is code that configures the animation speed. Again you can hard code that.
So if you start with that example and then simplify the code you will have a simple solution. Give it a try and then post your code when you encounter a problem.
This is very simple. Use a timer like this:
Timer timer = new Timer();
timer.scheduleAtFixedRate(new TimerTask() {
public void run() {
//codehere
}
}, 0, delayInMillis)
Use can use an integer to specify in image.
public int image = 1;
in the run() function, use this to switch between the image
if(image = 1) {
image = 2;
} else if(image = 2) {
image = 3;
} if(image = 3) {
image = 0;
}
Now, wherever you are drawing your images, use this:
if(image == 1) {
//draw first image
} else if(image == 2) {
//draw second image
} else if(image == 3) {
//draw third image
}

How to get fps/frames per second of a videocapture object

I am using a videocapture object to capture and process frames of a video in opencv/javacv.
I dont know how to get the frame rate.I want a timer that runs in the background during the live video capture.It should pause on a face being detected and continue later.
Due to the processing of haarcascade file,it is taking much time for each rame to process. how to adjust the frame rate.
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
VideoCapture camera = new VideoCapture(0);
You can extract various parameter from VideoCapture like frame rate, frame height, frame width etc.
cv::VideoCapture input_video;
if(input_video.open(my_device))
{
std::cout<<"Video file open "<<std::endl;
}
else
{
std::cout<<"Not able to Video file open "<<std::endl;
}
int fps = input_video.get(CV_CAP_PROP_FPS);
int frameCount = input_video.get(CV_CAP_PROP_FRAME_COUNT);
double fheight = input_video.get(CV_CAP_PROP_FRAME_HEIGHT);
double fwidth = input_video.get(CV_CAP_PROP_FRAME_WIDTH);
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
VideoCapture VC = new VideoCapture(0);
//First you requried open Camera.
VC.open();
//Now for geting 'Frame per secand"
VC.get(Videoio.CAP_PROP_FPS); // it returns FPS(Frame per secand)
//Now for seting 'Frame per secand"
VC.set(Videoio.CAP_PROP_FPS,10.0);//in this 10.0 is value for FPS,its double value.
VC.relase();
This answer helps me. There is a list of constants and their values. Just put value to VideoCapture's get() method. Ex. videoCapture.get(5), will return FPS of video.

Improving performance when processing images in Java

I am writing a program which among other things takes a folder of images (Typically around 2000 jpeg images) resizes them, and adds them to a timeline of images. The result of this being as follows:
This works fine, however the way I have done this seems very inefficient. The code which processes these images is shown below:
public void setTimeline(Vector<String> imagePaths){
int numberOfImages = imagePaths.size();
JLabel [] TotalImages = new JLabel[numberOfImages];
setGridPanel.setLayout(new GridLayout(1, numberOfImages, 10, 0));
Dimension image = new Dimension(96, 72);
if (imagePaths != null){
for(int i = 0; i <numberOfImages; i++){
TotalImages[i] = new JLabel("");
TotalImages[i].setPreferredSize(image);
ImageIcon tempicon = new ImageIcon(imagePaths.elementAt(i));
Image tempimage = tempicon.getImage();
Image newimg = tempimage.getScaledInstance(96, 72, java.awt.Image.SCALE_SMOOTH);
ImageIcon newIcon = new ImageIcon(newimg);
TotalImages[i].setIcon(newIcon);
setGridPanel.add(TotalImages[i]);
}
}
}
As can be seen, this code loops through each image path, adds it to a label and adds it to the panel - performing exactly as it should with the correct output.
However, the time taken to do this is substantial. Typically around 5 minutes for 2000 images (depending on the machine). I wondered if there is any way I could improve this performance by using different techniques?
Any help is greatly appreciated.
Save your scaled instances and load them direct. Hard drive space is cheap. This won't get around the initial cost of generating the thumbs, but any subsequent appearances will be lightning-fast.
takes a folder of images
with processes by using tempimage.getScaledInstance(96, 72, java.awt.Image.SCALE_SMOOTH);
use JTable, with reduced funcionality you can use JList too
Typically around 5 minutes for 2000 images
Image.getScaledInstance is simple asynchonous, witouth guarantee an fast and performance, then you have to redirect loading of images to the Background task
advantage first part of images are available immediatelly
dis_advantage required dispalying statuses of loading for user, very good knowledge about Swing and Event Dispatch Thread
I'd suggest to look at Runnable#Thread, and output put to the DefaultTableModel, notice this output must be wrapped into invokeLater
another and most complex way is use SwingWorker, but required very good knowledge about Java and Swing too
To add to mKorbel's excellent answer, I would definitely use a background thread such as a SwingWorker. This may not make the program any faster, but it will seem a lot faster, and that can make all the difference. Something like:
// use List<String> not Vector<String> so you can use Vector now, or change your
// mind and use ArrayList later if desired
// pass dimensions and components in as parameters to be cleaner
public void setTimeLine2(List<String> imagePaths, Dimension imgSize,
JComponent imgDisplayer) {
if (imagePaths != null && imgSize != null && imgDisplayer != null) {
// are you sure you want to set the layout in here?
imgDisplayer.setLayout(new GridLayout(1, 0, 10, 0));
// create your SwingWorker, passing in parameters that it will need
ImageWorker imgWorker = new ImageWorker(imagePaths, imgSize,
imgDisplayer);
imgWorker.execute(); // then ask it to run doInBackground on a background thread
} else {
// TODO: throw exception
}
}
private class ImageWorker extends SwingWorker<Void, ImageIcon> {
private List<String> imagePaths;
private JComponent imgDisplayer;
private int imgWidth;
private int imgHeight;
public ImageWorker(List<String> imagePaths, Dimension imgSize,
JComponent imgDisplayer) {
this.imagePaths = imagePaths;
this.imgDisplayer = imgDisplayer;
imgWidth = imgSize.width;
imgHeight = imgSize.height;
}
// do image creation in a background thread so as not to lock the Swing event thread
#Override
protected Void doInBackground() throws Exception {
for (String imagePath : imagePaths) {
BufferedImage bImg = ImageIO.read(new File(imagePath));
Image scaledImg = bImg.getScaledInstance(imgWidth, imgHeight,
Image.SCALE_SMOOTH);
ImageIcon icon = new ImageIcon(scaledImg);
publish(icon);
}
return null;
}
// but do all Swing manipulation on the event thread
#Override
protected void process(List<ImageIcon> chunks) {
for (ImageIcon icon : chunks) {
JLabel label = new JLabel(icon);
imgDisplayer.add(label);
}
}
}
Use tiles. Which means than rather than operating on images which are not shown in the screen, you only operated when the image has to be shown on the screen.
You need to maintain the logical position of the timeline, as well as displayed images.
When the user move the cursor to a previously hidden position, you compute which image(s) need to be shown next. If the images are not already processed, you process them. That's the same technique web-browsers use for performance.
A first thing you could do would be to add the images asynchronously, instead of trying to add all of them at once. Loop over them as you do, add them to the panel and render it every few images or so the user doesn't need to wait for a long initialization time.
Reuse image objects. A flyweight pattern would come to mind, and possibly limit the screen redraws to only the portions where you add a new image in your asynchronous loading.
If you are likely to have the same images redrawn (or to reload the same folders) in the future, you might want to consider caching some of the image objects, and maybe to save to file the resized thumbnails (many photo viewers do this and will store thumbnails versions - and some useful metadata - in hidden files or folders, so they can reload them faster the next time around.
what you could do to make it faster is by making 4 threads, and have them computing simultaneously the images. i dont know if the vm will spread them over multiple cpu cores though. something to look into because that would boost perfotrmace on a multicore pc

Categories