I've been programming sound with simple Swing graphics for a little while, but my frame rates are choppy for some reason.
Generally I'm doing something like the following on a background thread:
for(;;) {
// do some drawing
aPanel.updateABufferedImage();
// ask for asynchronous repaint
aPanel.repaint();
// write the sound
aSourceDataLine.write(bytes, 0, bytes.length);
}
Through debugging, I think I've already traced the problem to the blocking behavior of SourceDataLine#write. Its doc states the following:
If the caller attempts to write more data than can currently be written [...], this method blocks until the requested amount of data has been written.
So, what this seems to mean is SourceDataLine actually has its own buffer that it is filling when we pass our buffer to write. It only blocks when its own buffer is full. This seems to be the holdup: getting it to block predictably.
To demonstrate the issue, here's a minimal example which:
writes 0's to a SourceDataLine (no audible sound) and times it.
draws an arbitrary graphic (flips each pixel color) and times the repaint cycle.
import javax.swing.*;
import java.awt.*;
import java.awt.event.*;
import java.awt.image.*;
import javax.sound.sampled.*;
class FrameRateWithSound implements Runnable {
public static void main(String[] args) {
SwingUtilities.invokeLater(new FrameRateWithSound());
}
volatile boolean soundOn = true;
PaintPanel panel;
#Override
public void run() {
JFrame frame = new JFrame();
JPanel content = new JPanel(new BorderLayout());
final JCheckBox soundCheck = new JCheckBox("Sound", soundOn);
soundCheck.addActionListener(new ActionListener() {
#Override
public void actionPerformed(ActionEvent e) {
soundOn = soundCheck.isSelected();
}
});
panel = new PaintPanel();
content.add(soundCheck, BorderLayout.NORTH);
content.add(panel, BorderLayout.CENTER);
frame.setContentPane(content);
frame.pack();
frame.setLocationRelativeTo(null);
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.setVisible(true);
new Thread(new Worker()).start();
}
class Worker implements Runnable {
#Override
public void run() {
AudioFormat fmt = new AudioFormat(
AudioFormat.Encoding.PCM_SIGNED,
44100f, 8, 1, 1, 44100f, true
);
// just 0's
byte[] buffer = new byte[1000];
SourceDataLine line = null;
try {
line = AudioSystem.getSourceDataLine(fmt);
line.open(fmt);
line.start();
for(;;) {
panel.drawNextPixel();
panel.repaint();
if(soundOn) {
// time the write
long t = System.currentTimeMillis();
line.write(buffer, 0, buffer.length);
t = ( System.currentTimeMillis() - t );
System.out.println("sound:\t" + t);
}
// just so it doesn't fly off the handle
Thread.sleep(2);
}
} catch(Exception e) {
// lazy...
throw new RuntimeException(e);
} finally {
if(line != null) {
line.close();
}
}
}
}
class PaintPanel extends JPanel {
Dimension size = new Dimension(200, 100);
BufferedImage img = new BufferedImage(
size.width, size.height, BufferedImage.TYPE_INT_RGB);
int x, y;
int repaints;
long begin, prev;
String fps = "0";
PaintPanel() {
setPreferredSize(size);
setOpaque(false);
Graphics2D g = img.createGraphics();
g.setColor(Color.LIGHT_GRAY);
g.fillRect(0, 0, size.width, size.height);
g.dispose();
}
synchronized void drawNextPixel() {
img.setRGB(x, y, img.getRGB(x, y) ^ 0xFFFFFF); // flip
if( ( ++x ) == size.width ) {
x = 0;
if( ( ++y ) == size.height ) {
y = 0;
}
}
}
#Override
protected synchronized void paintComponent(Graphics g) {
super.paintComponent(g);
g.drawImage(img, 0, 0, size.width, size.height, null);
long curr = System.currentTimeMillis();
// time this cycle
long cycle = ( curr - prev );
System.out.println("paint:\t" + cycle);
++repaints;
// time FPS every 1 second
if(curr - begin >= 1000) {
begin = curr;
fps = String.valueOf(repaints);
repaints = 0;
}
prev = curr;
g.setColor(Color.RED);
g.drawString(fps, 12, size.height - 12);
}
}
}
I recommend actually running the example if you are curious about this.
A typical System.out feed during "playback" is something like the following:
sound: 0
paint: 2
sound: 0
paint: 2
sound: 0
paint: 3
sound: 0
paint: 2
paint: 2
sound: 325 // <- 'write' seems to be blocking here
sound: 0
paint: 328
sound: 0
paint: 2
This shows the behavior of write pretty clearly: it spins a majority of the time, then blocks for an extended period, at which point the repaints chug as well. The FPS meter typically displays ~45 during playback, but the animation is obviously choppy.
When sound is turned off, FPS climbs and the animation is smooth.
So is there a way to fix it? What am I doing wrong? How can I get write to block at a regular interval?
This behavior is apparent both on Windows and OSX environments.
One thing I've tried is using Thread.sleep to regulate it, but it's not very good. It's still choppy.
The solution seems to be to use open(AudioFormat, int) to open the line with a specified buffer size.
line.open(fmt, buffer.length);
Timing it again, we can see that write blocks much more consistently:
sound: 22
paint: 24
sound: 21
paint: 24
sound: 20
paint: 22
sound: 21
paint: 23
sound: 20
paint: 23
And the animation is smooth.
I seriously doubt that the sound playback is the culprit here. Please see my comment on the main question. The blocking that occurs in the audio write() method pertains to the rate at which the audio is presented to the playback system. Since audio processing is usually an order of magnitude faster than the audio system can play back (limited to 44100 fps), a majority of the time is spent blocking, for BOTH SourceDataLine and Clip. During this form of blocking, the CPU is FREE to do other things. It does not hang.
I'm more suspicious of the your use of synchronization for the images, and the editing being done to the image. I'm pretty sure editing on a bit level will wipe out the default graphics acceleration for that image.
You might check out this link on Graphics2D optimizations at Java-Gaming.org
http://www.java-gaming.org/topics/java2d-clearing-up-the-air/27506/msg/0/view/topicseen.html#new
I found it very helpful for optimizing my 2D graphics.
I'm not sure why you are getting coalescence in your particular case. The few times it has been an issue for me is when the looping code for the frame and the component were in the same class. By just putting the "game loop" code and the component in separate classes, the problem always went away for me, so I never bothered to think about it further. As a consequence I don't have a clear understanding of why that worked, or if that action was even a factor.
[EDIT: just looked more closely at your audio code. I think there is room for optimization. There are calculations being redone needlessly and could be consuming cpu. For example, since you have final values in your inner loop, why recalculate that part every iteration? Take the constant part and calculate that into a value once, and only calculate the unknowns in the inner loop. I recommend refactoring to allow avoiding all synchronizing and optimizing the data generation of the audio data, and then seeing if there is still a problem.]
Related
I'm writing a map making tool for a project that I am working on and I recently put in a JComboBox as a drop down selector for types of textures that can be applied to a tile. I'm custom drawing on an extended JPanel (labeled as DrawPanel) by overriding paintComponent() and calling repaint() for the graphical part of the editor and I'm also using various Swing components for the menus. Recently I started to notice that clicking on and opening the JComboBox caused the performance of the DrawPanel to slow to a crawl. I narrowed down the problem a little bit and figured out that it only happens under certain use cases (will explain them later). When the draw canvas is slowed down it can only be fixed by closing the window and re-launching or sometimes it happens intermittently when you resize the window. I suspect this works because the components get resized and re-validated but manually calling revalidate() didn't seem to work.
While looking into what exactly was being slowed down I put in some time measurments. I found that everything in the logic loop, i.e. update() and window.repaint(), all fire like normal: 10ms for the update method and as fast as possible with a 1ms sleep for the repaint() calls. When I timed inside of the paintComponent() method however I found that the time between the start of the method and next start of the same method had increased from an average of 1ms to an average of 30—180ms depending on how large the window was. The time it took to complete the draw functions didn't change though. I looked into it a bit more and to my understanding calling repaint() on the component schedules with the AWT thread using an event to paint the component again, but it's ultimately up to the AWT thread when it wants to actually do the repaint. This is to keep it concurrent and is also the reason why it's not recommended to directly override the paint() or update() method of a component. I'm not the best with everything Swing so please correct me if I got that wrong.
With this understanding though It seems like something is causing the AWT thread to either slowdown or focus on something else and doesn't get around to repainting the canvas as fast as I'd like it to. From now on we will just refer to the over all issue as a slowdown.
On to the use cases. I found that there are a number of use cases with JComboBoxes in which the slowdown occurs. Under all circumstances it only happens when the JComboBox draws the drop down outside of the JFrame. It also only happens when the JComboBox is unable to fill the area around it, whether that be by setting setMaximumSize() directly or if it is limited in size indirectly (such as placing it in a JPanel and letting it size to the width and height of it's contents). Oddly enough if the maximum number of displayed rows exceeds 10 the slowdown does not occur even if displayed outside of the JFrame.
Here's a Test class that I cooked up to show the behavior I mentioned. Depending on your hardware it might be noticeable a lot or it might not be all that noticeable. I tried to give the graphics card at least a little work so as to show the slowdown when it occurs. I ran this on my rig with a Ryzen 3800x and an RTX 2070.
import javax.swing.*;
import java.awt.*;
public class TestClass{
public boolean running = true;
public long tickRegulator, displayTimer;
public long UPSFrameTimeMark, UPSFrameTime, FPSFrameTimeMark, FPSFrameTime, UPSDisplay, FPSDisplay;
public JFrame window;
public DrawPanel canvas;
public float time = 0;
public static void main(String[] args){new TestClass();}
public TestClass()
{
initialize();
logicLoop();
}
private void initialize()
{
//look and feel + defaulting the timings
UPSFrameTimeMark = UPSFrameTime = FPSFrameTime = FPSFrameTimeMark = System.currentTimeMillis();
try{UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName());}
catch(Exception e){e.printStackTrace();}
//below is the frame setup
window = new JFrame("Test Frame");
window.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
//DrawPanel extends JPanel and uses an overridden paintComponent() to draw
canvas = new DrawPanel();
canvas.setPreferredSize(new Dimension(400, 400));
window.add(canvas, BorderLayout.NORTH);
//below are all the use cases for the JComboBoxes
int itemCount = 20;
String[] options = new String[itemCount];
for(int i = 0; i < itemCount; i++){options[i]=""+i;}
//The first two JComboBoxes are tested in a toolbar
JToolBar toolBar = new JToolBar();
toolBar.setBackground(Color.WHITE);
toolBar.setFloatable(false);
window.getContentPane().add(toolBar, BorderLayout.PAGE_END);
//bottom left - max size not set (filling available space) - causes no issue
JComboBox mapTextureOptions = new JComboBox(options);
mapTextureOptions.setForeground(Color.BLUE);
toolBar.add(mapTextureOptions);
//bottom right - max size set to 15x15 (although only the height is respected) - causes issue
JComboBox mapTextureOptions2 = new JComboBox(options);
mapTextureOptions2.setForeground(Color.RED);
mapTextureOptions2.setMaximumSize(new Dimension(15, 15));
toolBar.add(mapTextureOptions2);
//the second two JComboBoxes are tested in a JPanel
JPanel panel = new JPanel();
window.add(panel, BorderLayout.CENTER);
//top left - max rows set to 11 + no max size is explicitly set - causes no issues
JComboBox mapTextureOptions3 = new JComboBox(options);
mapTextureOptions3.setForeground(Color.YELLOW);
mapTextureOptions3.setMaximumRowCount(11);
panel.add(mapTextureOptions3);
//top right - max rows set to 10 + no max size is explicitly set - causes issues
JComboBox mapTextureOptions4 = new JComboBox(options);
mapTextureOptions4.setForeground(Color.GREEN);
mapTextureOptions4.setMaximumRowCount(10);
panel.add(mapTextureOptions4);
window.pack();
window.setVisible(true);
}
private void logicLoop()
{
long timeMarker = System.currentTimeMillis();
while(running){
tickRegulator += (System.currentTimeMillis() - timeMarker);
displayTimer += (System.currentTimeMillis() - timeMarker);
timeMarker = System.currentTimeMillis();
if(displayTimer > 100)
{
FPSDisplay = FPSFrameTime;
UPSDisplay = UPSFrameTime;
displayTimer = 0;
}
try{
while(tickRegulator > 10)
{
UPSFrameTime = System.currentTimeMillis() - UPSFrameTimeMark;
UPSFrameTimeMark = System.currentTimeMillis();
update();
tickRegulator-=10;
}
window.repaint();
}catch(Exception e)
{
e.printStackTrace();
}
try{Thread.sleep(1);}catch(Exception e){e.printStackTrace();}
}
}
private void update()
{
time += 0.02f;
}
private void draw(Graphics2D g)
{
FPSFrameTime = System.currentTimeMillis() - FPSFrameTimeMark;
FPSFrameTimeMark = System.currentTimeMillis();
//draw background
g.setColor(Color.DARK_GRAY);
g.fillRect(0, 0, window.getWidth(), window.getHeight());
float zoom = 0.05f;
for(int x = 0; x < Math.max((canvas.getWidth() * 1.2f) / zoom, 500); x += 100)
{
g.setColor(new Color(27, 255, 0, Math.round(80 * 0.8f)));
if((x - (0 - (0 % 100))) % 1000 != 0)
{
g.setColor(new Color(0, 255, 80, Math.round(28 * 0.8f)));
}
if((x - (0 - (0 % 100))) % 10000 == 0)
{
g.setColor(new Color(27, 255, 0, Math.round(128 * 0.8f)));
}
g.drawLine(Math.round((x + (0 % 100)) * zoom), 0, Math.round((x + (0 % 100)) * zoom), canvas.getHeight());
}
for(int y = 0; y < Math.max((canvas.getHeight() * 1.2) / zoom, 500); y += 100)
{
g.setColor(new Color(27, 255, 0, Math.round(80 * 0.8f)));
if((y - (0 - (0 % 100))) % 1000 != 0)
{
g.setColor(new Color(0, 255, 80, Math.round(28 * 0.8f)));
}
if((y - (0 - (0 % 100))) % 10000 == 0)
{
g.setColor(new Color(27, 255, 0, Math.round(128 * 0.8f)));
}
g.drawLine(0, Math.round((y + (0 % 100)) * zoom), canvas.getWidth(), Math.round((y + (0 % 100)) * zoom));
}
g.setColor(Color.RED);
g.fillOval((int)(Math.cos(time) * (canvas.getWidth() / 2 - 25)) + canvas.getWidth() / 2 - 25,
(int)(Math.sin(time) * (canvas.getHeight() / 2 - 25)) + canvas.getHeight() / 2 - 25,
50, 50);
g.drawString("UPS Frame Time: " + (UPSDisplay) + "ms", 5, 20);
g.drawString("FPS Frame Time: " + (FPSDisplay) + "ms", 5, 35);
}
class DrawPanel extends JPanel
{
public DrawPanel() {
setBorder(BorderFactory.createLineBorder(Color.black));
}
public void paintComponent(Graphics g) {
super.paintComponent(g);
draw((Graphics2D) g);
}
}
}
This is all of the necessary code to get the problem to occur. The test program includes a moving sphere so it's easy visualize the slowdowns as well as the UPS timings and the FPS timings. Pay attention to the FPS timings when looking for the slowdown. The two JComboBoxes that are leftmost do not cause the slowdown and the two that are rightmost do.
As you can see the FPS timings are at 1ms here even though the JComboBox is displaying outside of the JFrame.
In this image the drop box that I set to be a maximum size of 15x15 pixels causes the slowdown to occur and the frame time jumps to 12ms. That doesn't seem like much but if I increase the window size to be the size of my monitor (2560x1080) it slows down to a whopping 70+ms (in this example due to the JPanel taking up half the window at that resolution it isn't that bad but with the normal program it makes the frame time jump to 180+ms; nearly 5fps).
I don't know if there's something I'm missing here. Whether this is due to behaviors in Swing that I don't know about or if I've stumbled across a bug. Either way it's slowing down my application like crazy and I need a fix. (For now I'm just setting the max row size above 10 because that seems to work). Does anybody know why this is happening?
I’m starting to work on a music/metronome application in Java and I’m running into some problems with the timing and speed.
For testing purposes I’m trying to play two sine wave tones at the same time at regular intervals, but instead they play in sync for a few beats and then slightly out of sync for a few beats and then back in sync again for a few beats.
From researching good metronome programming, I found that Thread.sleep() is horrible for timing, so I completely avoided that and went with checking System.nanoTime() to determine when the sounds should play.
I’m using AudioSystem’s SourceDataLine for my audio player and I’m using a thread for each tone that constantly polls System.nanoTime() in order to determine when the sound should play. I create a new SourceDataLine and delete the previous one each time a sound plays, because the volume fluctuates if I leave the line open and keep playing sounds on the same line. I create the player before polling nanoTime() so that the player is already created and all it has to do is play the sound when it is time.
In theory this seemed like a good method for getting each sound to play on time, but it’s not working correctly. I’m not sure if the timing problems are from running different threads or if it has to do with deleting and recreating the SourceDataLine or if it’s in playing sounds or what exactly...
At the moment this is just a simple test in Java, but my goal is to create my app on mobile devices (Android, iOS, Windows Phone, etc)...however my current method isn’t even keeping perfect time on a PC, so I’m worried that certain mobile devices with limited resources will have even more timing problems. I will also be adding more sounds to it to create more complex rhythms, so it needs to be able to handle multiple sounds going simultaneously without sounds lagging.
Another problem I’m having is that the max tempo is controlled by the length of the tone since the tones don’t overlap each other. I tried adding additional threads so that every tone that played would get its own thread...but that really screwed up the timing, so I took it out. I would like to have a way to overlap the previous sound to allow for much higher tempos.
Any help getting these timing and speed issues straightened out would be greatly appreciated!
Thanks.
SoundTest.java:
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.*;
import java.io.*;
import javax.sound.sampled.*;
public class SoundTest implements ActionListener {
static SoundTest soundTest;
// ENABLE/DISABLE SOUNDS
boolean playSound1 = true;
boolean playSound2 = true;
JFrame mainFrame;
JPanel mainContent;
JPanel center;
JButton buttonPlay;
int sampleRate = 44100;
long startTime;
SourceDataLine line = null;
int tickLength;
boolean playing = false;
SoundElement sound01;
SoundElement sound02;
public static void main (String[] args) {
soundTest = new SoundTest();
SwingUtilities.invokeLater(new Runnable() { public void run() {
soundTest.gui_CreateAndShow();
}});
}
public void gui_CreateAndShow() {
gui_FrameAndContentPanel();
gui_AddContent();
}
public void gui_FrameAndContentPanel() {
mainContent = new JPanel();
mainContent.setLayout(new BorderLayout());
mainContent.setPreferredSize(new Dimension(500,500));
mainContent.setOpaque(true);
mainFrame = new JFrame("Sound Test");
mainFrame.setContentPane(mainContent);
mainFrame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
mainFrame.pack();
mainFrame.setVisible(true);
}
public void gui_AddContent() {
JPanel center = new JPanel();
center.setOpaque(true);
buttonPlay = new JButton("PLAY / STOP");
buttonPlay.setActionCommand("play");
buttonPlay.addActionListener(this);
buttonPlay.setPreferredSize(new Dimension(200, 50));
center.add(buttonPlay);
mainContent.add(center, BorderLayout.CENTER);
}
public void actionPerformed(ActionEvent e) {
if (!playing) {
playing = true;
if (playSound1)
sound01 = new SoundElement(this, 800, 1);
if (playSound2)
sound02 = new SoundElement(this, 1200, 1);
startTime = System.nanoTime();
if (playSound1)
new Thread(sound01).start();
if (playSound2)
new Thread(sound02).start();
}
else {
playing = false;
}
}
}
SoundElement.java
import java.io.*;
import javax.sound.sampled.*;
public class SoundElement implements Runnable {
SoundTest soundTest;
// TEMPO CHANGE
// 750000000=80bpm | 300000000=200bpm | 200000000=300bpm
long nsDelay = 750000000;
int clickLength = 4100;
byte[] audioFile;
double clickFrequency;
double subdivision;
SourceDataLine line = null;
long audioFilePlay;
public SoundElement(SoundTest soundTestIn, double clickFrequencyIn, double subdivisionIn){
soundTest = soundTestIn;
clickFrequency = clickFrequencyIn;
subdivision = subdivisionIn;
generateAudioFile();
}
public void generateAudioFile(){
audioFile = new byte[clickLength * 2];
double temp;
short maxSample;
int p=0;
for (int i = 0; i < audioFile.length;){
temp = Math.sin(2 * Math.PI * p++ / (soundTest.sampleRate/clickFrequency));
maxSample = (short) (temp * Short.MAX_VALUE);
audioFile[i++] = (byte) (maxSample & 0x00ff);
audioFile[i++] = (byte) ((maxSample & 0xff00) >>> 8);
}
}
public void run() {
createPlayer();
audioFilePlay = soundTest.startTime + nsDelay;
while (soundTest.playing){
if (System.nanoTime() >= audioFilePlay){
play();
destroyPlayer();
createPlayer();
audioFilePlay += nsDelay;
}
}
try { destroyPlayer(); } catch (Exception e) { }
}
public void createPlayer(){
AudioFormat af = new AudioFormat(soundTest.sampleRate, 16, 1, true, false);
try {
line = AudioSystem.getSourceDataLine(af);
line.open(af);
line.start();
}
catch (Exception ex) { ex.printStackTrace(); }
}
public void play(){
line.write(audioFile, 0, audioFile.length);
}
public void destroyPlayer(){
line.drain();
line.close();
}
}
This sort of thing is difficult to get right. What you have to realise is that in order to even play a sound, it has to be loaded into an audio driver (and possibly a sound card). This takes time, and you have to account for that. There are basically two options for you:
Rather than counting down a delay between every beat, count down a delay from the start, when the metronome activates. As an example, say for instance that you want a beat every second. Because of the ~20ms delay, in your old method you'd get beats at 20ms, 1040, 2060, 3080, etc... If you count down from the start and place beats at 1000, 2000, 3000, etc. then they will play at 20ms, 1020, 2020, 3020, etc... There will still be some variance since the dalay itself varies a bit, but there should be about 1000ms between beats and it will not go out of sync (or at least, the problem will not get worse over time and likely can't be heard).
The better option, and the one that most of such programs use, is to generate larger pieces of music. Buffer for instance 20 seconds ahead and play that. The timing should be perfect during those 20 seconds. When those 20 seconds are almost over you must generate some new sound. If you can find out how to do this, you should append the new waveform to the old and have it play continuously. Otherwise, just generate a new 20 second soundbit and accept the delay between them.
Now as for your problem with the sounds not being able to overlap... I'm no expert and I don't really know an answer, but this I do know: Something has to mix the sounds if you need them to overlap. Either you can do that yourself in software by combining the waveform bytes (I think it's an addition in some logarithmic space), or you need to send the different overlapping sounds to different 'channels', in which case the audio driver or sound card does it for you. I don't know how this works in Java though, or I forgot, but I learned this through trial-and-error and working with .mod files.
I'm working on an application that records the users screen, webcam and microphone whilst he/she is performing certain activities. It will be used for research purposes. The application has been successfully tested on Windows, but on Mac OS X (Maverick with Java 7.0.45) the application becomes slow and unresponsive when recording is started.
This is why I find this difficult to comprehend:
The recording is done in a separate thread, so how could it influence the responsiveness of another thread? Especially as after each run either Thread.yield() or Thread.sleep(...) are called.
Logs show that whilst attempting to record at 15 FPS, the resulting frame rate was 2 FPS. So it seems the code that does the capturing of a single frame might be too slow. But why then does it work fine on Windows?
Just a quick note: the application was successfully tested by tons of users on Windows, but I only got to test it on a single Mac. However, that one was just formatted and got a clean install of OS X Maverick, Java (and Netbeans).
Below you will find the code that records the screen and writes it to a video using Xuggler. The code for recording the webcam is similar, and I'd doubt recording the audio has anything to do with it. My question is:
What might be the cause of the application becoming unresponsive?, and
How could the code be made more efficient and so improve FPS?
IMediaWriter writer = ToolFactory.makeWriter(file.getAbsolutePath());
Dimension size = Globals.sessionFrame.getBounds().getSize();
Rectangle screenRect;
BufferedImage capture;
BufferedImage mousePointImg;
writer.addVideoStream(0, 0, ICodec.ID.CODEC_ID_H264, size.width, size.height);
int i = 0;
while (stop == false) {
// Get mouse cursor to draw over screen image.
PointerInfo mousePointer = MouseInfo.getPointerInfo();
Point mousePoint = mousePointer.getLocation();
Point screenPoint = new Point((int) (mousePoint.getX() -
Globals.sessionFrame.getBounds().getX()), (int) (mousePoint.getY() -
Globals.sessionFrame.getBounds().getY()));
// Get the screen image.
try {
screenRect = new Rectangle(Globals.sessionFrame.getBounds());
capture = new Robot().createScreenCapture(screenRect);
} catch ( ... ) { ... }
// Convert and resize the screen image.
BufferedImage image = ConverterFactory.convertToType(capture,
BufferedImage.TYPE_3BYTE_BGR);
IConverter converter = ConverterFactory.createConverter(image,
IPixelFormat.Type.YUV420P);
// Draw the mouse cursor if necessary.
if (mouseWithinScreen()) {
Graphics g = image.getGraphics();
g.drawImage(mousePointImg, (int) screenPoint.getX(),
(int) screenPoint.getY(), null);
}
// Prepare the frame.
IVideoPicture frame = converter.toPicture(image, (System.currentTimeMillis() -
startTimeMillis()) * 1000);
frame.setKeyFrame(i % (getDesiredFPS() * getDesiredKeyframeSec()) == 0);
// Write to the video
writer.encodeVideo(0, frame);
// Delay the next capture if we are at the desired FPS.
try {
if (atDesiredFPS()) {
Thread.yield();
} else {
Thread.sleep(1000 / getDesiredFPS());
}
} catch ( ... ) { ... }
i++;
}
writer.close();
There are several architectural issues that I can see in your code:
First if you want to execute something at a fixed rate, use the ScheduledThreadPoolExecutor.scheduleAtFixedRate(...) function. It will make your entire delay code part obsolete as well as ensuring that certain OS timing issues will not interfere with your scheduling.
Then to make things faster you need to take your code apart a bit. As far as I can see you have 3 tasks: the capture, the mouse-drawing/conversion and the stream writing. If you put the capture part in a scheduled Runnable, the conversion into multi-parallel execution as Callables into an Executor, and then in a 3rd thread take the results from a result list and write it into the stream, you can fully utilize multi-cores.
Pseudocode:
Global declarations (or hand them over to the various classes):
final static Executor converterExecutor = Executors.newFixedThreadPoolExecutor(Runtime.getRuntime().availableProcessors());
final static LinkedBlockingQueue<Future<IVideoPicture>> imageQueue = new LinkedBlockingQueue<>();
// ...
Capture Runnable (scheduled at fixed rate):
capture = captureScreen();
final Converter converter = new Converter(capture);
final Future<IVideoPicture> conversionResult = converterExecutor.submit(converter);
imageQueue.offer(conversionResult); // returns false if queue is full
Conversion Callable:
class Converter implements Callable<IVideoPicture> {
// ... variables and constructor
public IVideoPicture call() {
return convert(this.image);
}
}
Writer Runnable:
IVideoPicture frame;
while (this.done == false) {
frame = imageQueue.get();
writer.encodeVideo(0, frame);
}
You can ensure that the imageQueue does not overflow with images to render if the CPU is too slow by limiting the size of this queue, see the constructor of LinkedBlockingQueue.
I am making a 2D platform in Java, and I'm using a Swing timer, it is set at 5 milliseconds interval. However, sometimes it is smooth movement at a good speed, but then randomly it will become super fast and sometimes super slow. What could be the reason for this?
Code:
public Board() {
addKeyListener(new KeyListener());
setFocusable(true);
setBackground(new Color(204,250,255));
//draws the object off the screen in memory, then brings it in
setDoubleBuffered(true);
...(other code not relevant)...
timer = new Timer(5, this);
timer.start();
}
public void paint(Graphics g) {
super.paint(g);
Graphics2D g2d = (Graphics2D) g;
//draw platform
for(int i = 0; i < platform.length; i++) {
//g2d.setColor(new Color(0,0,0));
//g2d.drawRect(platform[i].getX(), platform[i].getY(), platform[i].getWidth(), platform[i].getHeight());
g2d.drawImage(platform[i].getImage(), platform[i].getX(), platform[i].getY(), this);
}
//draw guy
g2d.drawImage(guy.getImage(), guy.getX(), guy.getY(), this);
//destroy unneeded process
Toolkit.getDefaultToolkit().sync();
g.dispose();
}
Basically, the cycle function decides whether the character needs to move or not, and moves it if it does.
It is hard to make Java work as a real-time systems…. because of unknown factors like when the OS will schedule your process to run and GC. Since you are trying to update the game at 200 fps (if my math is correct) you might want to try the following approach. To make the game smoother I would try to find the time between the current event fired and the last event fired, now theoretically that might be 5ms but sometimes it will be 7ms (going super slow) and sometimes it will be 3ms (going super fast). Ones you have the time between the two event I would calculate where the animation should be, so if the time is 3ms then the animation will be a bit before the 5ms time. Lets take a simple example where your are trying to update the X position of a image 5 px every 5ms.
//your current code
newX = X + 5px
//new code
//this wont always be 5ms
timeDiff = currentEventTime – lastEventTime;
newX = X + 5px (timeDiff/5ms)
EDIT TWO
To prevent snarky comments and one-line answers missing the point: IFF it is as simple as calling setDoubleBuffered(true), then how do I get access to the current offline buffer so that I can start messing with the BufferedImage's underlying pixel databuffer?
I took the time to write a running piece of code (which looks kinda fun too) so I'd really appreciate answers actually answering (what a shock ;) my question and explaining what/how this is working instead of one-liners and snarky comments ;)
Here's a working piece of code that bounces a square across a JFrame. I'd like to know about the various ways that can be used to transform this piece of code so that it uses double-buffering.
Note that the way I clear the screen and redraw the square ain't the most efficient but this is really not what this question is about (in a way, it's better for the sake of this example that it is somewhat slow).
Basically, I need to constantly modify a lot pixels in a BufferedImage (as to have some kind of animation) and I don't want to see the visual artifacts due to single-buffering on screen.
I've got a JLabel whose Icon is an ImageIcon wrapping a BufferedImage. I want to modify that BufferedImage.
What has to be done so that this becomes double-buffered?
I understand that somehow "image 1" will be shown while I'll be drawing on "image 2". But then once I'm done drawing on "image 2", how do I "quickly" replace "image 1" by "image 2"?
Is this something I should be doing manually, like, say, by swapping the JLabel's ImageIcon myself?
Should I be always drawing in the same BufferedImage then do a fast 'blit' of that BufferedImage's pixels in the JLabel's ImageIcon's BufferedImage? (I guess no and I don't see how I could "synch" this with the monitor's "vertical blank line" [or equivalent in flat-screen: I mean, to 'synch' without interfering with the moment the monitor itselfs refreshes its pixels, as to prevent shearing]).
What about the "repaint" orders? Am I suppose to trigger these myself? Which/when exactly should I call repaint() or something else?
The most important requirement is that I should be modifying pixels directly in the images's pixel databuffer.
import javax.swing.*;
import java.awt.event.WindowAdapter;
import java.awt.event.WindowEvent;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferInt;
public class DemosDoubleBuffering extends JFrame {
private static final int WIDTH = 600;
private static final int HEIGHT = 400;
int xs = 3;
int ys = xs;
int x = 0;
int y = 0;
final int r = 80;
final BufferedImage bi1;
public static void main( final String[] args ) {
final DemosDoubleBuffering frame = new DemosDoubleBuffering();
frame.addWindowListener(new WindowAdapter() {
public void windowClosing( WindowEvent e) {
System.exit(0);
}
});
frame.setSize( WIDTH, HEIGHT );
frame.pack();
frame.setVisible( true );
}
public DemosDoubleBuffering() {
super( "Trying to do double buffering" );
final JLabel jl = new JLabel();
bi1 = new BufferedImage( WIDTH, HEIGHT, BufferedImage.TYPE_INT_ARGB );
final Thread t = new Thread( new Runnable() {
public void run() {
while ( true ) {
move();
drawSquare( bi1 );
jl.repaint();
try {Thread.sleep(10);} catch (InterruptedException e) {}
}
}
});
t.start();
jl.setIcon( new ImageIcon( bi1 ) );
getContentPane().add( jl );
}
private void drawSquare( final BufferedImage bi ) {
final int[] buf = ((DataBufferInt) bi.getRaster().getDataBuffer()).getData();
for (int i = 0; i < buf.length; i++) {
buf[i] = 0xFFFFFFFF; // clearing all white
}
for (int xx = 0; xx < r; xx++) {
for (int yy = 0; yy < r; yy++) {
buf[WIDTH*(yy+y)+xx+x] = 0xFF000000;
}
}
}
private void move() {
if ( !(x + xs >= 0 && x + xs + r < bi1.getWidth()) ) {
xs = -xs;
}
if ( !(y + ys >= 0 && y + ys + r < bi1.getHeight()) ) {
ys = -ys;
}
x += xs;
y += ys;
}
}
EDIT
This is not for a full-screen Java application, but a regular Java application, running in its own (somewhat small) window.
---- Edited to address per pixel setting ----
The item blow addresses double buffering, but there's also an issue on how to get pixels into a BufferedImage.
If you call
WriteableRaster raster = bi.getRaster()
on the BufferedImage it will return a WriteableRaster. From there you can use
int[] pixels = new int[WIDTH*HEIGHT];
// code to set array elements here
raster.setPixel(0, 0, pixels);
Note that you would probably want to optimize the code to not actually create a new array for each rendering. In addition, you would probably want to optimized the array clearing code to not use a for loop.
Arrays.fill(pixels, 0xFFFFFFFF);
would probably outperform your loop setting the background to white.
---- Edited after response ----
The key is in your original setup of the JFrame and inside the run rendering loop.
First you need to tell SWING to stop Rasterizing whenever it wants to; because, you'll be telling it when you're done drawing to the buffered image you want to swap out in full. Do this with JFrame's
setIgnoreRepaint(true);
Then you'll want to create a buffer strategy. Basically it specifies how many buffers you want to use
createBufferStrategy(2);
Now that you tried to create the buffer strategy, you need to grab the BufferStrategy object as you will need it later to switch buffers.
final BufferStrategy bufferStrategy = getBufferStrategy();
Inside your Thread modify the run() loop to contain:
...
move();
drawSqure(bi1);
Graphics g = bufferStrategy.getDrawGraphics();
g.drawImage(bi1, 0, 0, null);
g.dispose();
bufferStrategy.show();
...
The graphics grabbed from the bufferStrategy will be the off-screen Graphics object, when creating triple buffering, it will be the "next" off-screen Graphics object in a round-robin fashion.
The image and the Graphics context are not related in a containment scenario, and you told Swing you'd do the drawing yourself, so you have to draw the image manually. This is not always a bad thing, as you can specify the buffer flipping when the image is fully drawn (and not before).
Disposing of the graphics object is just a good idea as it helps in garbage collection. Showing the bufferStrategy will flip buffers.
While there might have been a misstep somewhere in the above code, this should get you 90% of the way there. Good luck!
---- Original post follows ----
It might seem silly to refer such a question to a javase tutorial, but have you looked into BufferStrategy and BufferCapatbilites?
The main issue I think you are encountering is that you are fooled by the name of the Image. A BufferedImage has nothing to do with double buffering, it has to do with "buffering the data (typically from disk) in memory." As such, you will need two BufferedImages if you wish to have a "double buffered image"; as it is unwise to alter pixels in image which is being shown (it might cause repainting issues).
In your rendering code, you grab the graphics object. If you set up double buffering according to the tutorial above, this means you will grab (by default) the off-screen Graphics object, and all drawing will be off-screen. Then you draw your image (the right one of course) to the off-screen object. Finally, you tell the strategy to show() the buffer, and it will do the replacement of the Graphics context for you.
Generally we use Canvas class which is suitable for animation in Java. Anyhoo, following is how you achieve double buffering:
class CustomCanvas extends Canvas {
private Image dbImage;
private Graphics dbg;
int x_pos, y_pos;
public CustomCanvas () {
}
public void update (Graphics g) {
// initialize buffer
if (dbImage == null) {
dbImage = createImage (this.getSize().width, this.getSize().height);
dbg = dbImage.getGraphics ();
}
// clear screen in background
dbg.setColor (getBackground ());
dbg.fillRect (0, 0, this.getSize().width, this.getSize().height);
// draw elements in background
dbg.setColor (getForeground());
paint (dbg);
// draw image on the screen
g.drawImage (dbImage, 0, 0, this);
}
public void paint (Graphics g)
{
g.setColor (Color.red);
g.fillOval (x_pos - radius, y_pos - radius, 2 * radius, 2 * radius);
}
}
Now you can update the x_pos and y_pos from a thread, followed by the 'repaint' call on the canvas object. The same technique should work on a JPanel as well.
What you want is basically impossible in windowed mode with Swing. There is no support for raster synchronization for window repaints, this is only available in fullscreen mode (and even then may not be supported by all platforms).
Swing components are double-buffered by default, that is they will do all the rendering to an intermediate buffer and that buffer is then finally copied to the screen, avoiding flicker from background clearing and then painting on top of it.
And thats the only strategy that is reasonable well supported on all underlying platforms. It avoids only repaint flickering, but not visual tearing from moving graphic elements.
A reasonably simple way of having access to the raw pixels of an area fully under you control would be to extend a custom component from JComponent and overwrite its paintComponent()-method to paint the area from a BufferedImage (from memory):
public class PixelBufferComponent extends JComponent {
private BufferedImage bufferImage;
public PixelBufferComponent(int width, int height) {
bufferImage = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
setPreferredSize(new Dimension(width, height));
}
public void paintComponent(Graphics g) {
g.drawImage(bufferImage, 0, 0, null);
}
}
You can then manipulate you buffered image whichever way you desire. To get your changes made visible on screen, simply call repaint() on it. If you do the pixel manipulation from a thread other than the EDT, you need TWO buffered images to cope with race conditions between the actual repaint and your manipulation thread.
Note that this skeleton will not paint the entire area of the component when used with a layout manager that stretches the component beyond its preferred size.
Note also, the buffered image approach mostly only makes sense if you do real low level pixel manipulation via setRGB(...) on the image or if you directly access the underlying DataBuffer directly. If you can do all the manipulations using Graphics2D's methods, you could do all the stuff in the paintComponent method using the provided graphics (which is actually a Graphics2D and can be simply casted).
Here's a variation in which all drawing takes place on the event dispatch thread.
Addendum:
Basically, I need to constantly modify a lot pixels in a BufferedImage…
This kinetic model illustrates several approaches to pixel animation.
import java.awt.Color;
import java.awt.Dimension;
import java.awt.EventQueue;
import java.awt.Graphics2D;
import java.awt.GridLayout;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import javax.swing.*;
import java.awt.image.BufferedImage;
/** #see http://stackoverflow.com/questions/4430356 */
public class DemosDoubleBuffering extends JPanel implements ActionListener {
private static final int W = 600;
private static final int H = 400;
private static final int r = 80;
private int xs = 3;
private int ys = xs;
private int x = 0;
private int y = 0;
private final BufferedImage bi;
private final JLabel jl = new JLabel();
private final Timer t = new Timer(10, this);
public static void main(final String[] args) {
EventQueue.invokeLater(new Runnable() {
#Override
public void run() {
JFrame frame = new JFrame();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
frame.add(new DemosDoubleBuffering());
frame.pack();
frame.setVisible(true);
}
});
}
public DemosDoubleBuffering() {
super(true);
this.setLayout(new GridLayout());
this.setPreferredSize(new Dimension(W, H));
bi = new BufferedImage(W, H, BufferedImage.TYPE_INT_ARGB);
jl.setIcon(new ImageIcon(bi));
this.add(jl);
t.start();
}
#Override
public void actionPerformed(ActionEvent e) {
move();
drawSquare(bi);
jl.repaint();
}
private void drawSquare(final BufferedImage bi) {
Graphics2D g = bi.createGraphics();
g.setColor(Color.white);
g.fillRect(0, 0, W, H);
g.setColor(Color.blue);
g.fillRect(x, y, r, r);
g.dispose();
}
private void move() {
if (!(x + xs >= 0 && x + xs + r < bi.getWidth())) {
xs = -xs;
}
if (!(y + ys >= 0 && y + ys + r < bi.getHeight())) {
ys = -ys;
}
x += xs;
y += ys;
}
}