Is Java a suitable alternative to C / C++ for realtime audio processing?
I am considering an app with ~100 (at max) tracks of audio with delay lines (30s # 48khz), filtering (512 point FIR?), and other DSP type operations occurring on each track simultaneously.
The operations would be converted and performed in floating point.
The system would probably be a quad core 3GHz with 4GB RAM, running Ubuntu.
I have seen articles about Java being much faster than it used to be, coming close to C / C++, and now having realtime extensions as well. Is this reality? Does it require hard core coding and tuning to achieve the %50-%100 performance of C some are spec'ing?
I am really looking for a sense if this is possible and a heads up for any gotchas.
For an audio application you often have only very small parts of code where most of the time is spent.
In Java, you can always use the JNI (Java Native interface) and move your computational heavy code into a C-module (or assembly using SSE if you really need the power). So I'd say use Java and get your code working. If it turns out that you don't meet your performance goal use JNI.
90% of the code will most likely be glue code and application stuff anyway. But keep in mind that you loose some of the cross platform features that way. If you can live with that JNI will always leave you the door open for native code performance.
Java is fine for many audio applications. Contrary to some of the other posters, I find Java audio a joy to work with. Compare the API and resources available to you to the horrendous, barely documented mindf*k that is CoreAudio and you'll be a believer. Java audio suffers from some latency issues, though for many apps this is irrelevant, and a lack of codecs. There are also plenty of people who've never bothered to take the time to write good audio playback engines(hint, never close a SourceDataLine, instead write zeros to it), and subsequently blame Java for their problems. From an API point of view, Java audio is very straightforward, very easy to use, and there is lots and lots of guidance over at jsresources.org.
Sure, why not?
The crucial questions (independent of language, this is from queueing theory) are:
what is the maximum throughput you need to handle (you've specified 100 x 48kHz, is that mono or stereo, how many bits equivalent at that frequency?)
can your Java routines keep up with this rate on the average?
what is the maximum permissible latency?
If your program can keep up with the throughput on the average, and you have enough room for latency, then you should be able to use queues for inputs and outputs, and the only parts of the program that are critical for timing are the pieces that put the data into the input queue and take it out of the output queue and send it to a DAC/speaker/whatever.
Delay lines have low computational load, you just need enough memory (+ memory bandwidth)... in fact you should probably just use the input/output queues for it, i.e. start putting data into the input queue immediately, and start taking data out of the output queue 30s later. If it's not there, your program is too slow...).
FIRs are more expensive, that's probably going to be the bottleneck (& what you'd want to optimize) unless you have some other ugly nasty operation in mind.
I think latency will be your major problem - it is quite hard to maintain latency already in C/C++ on modern OSes, and java surely adds to the problem (garbage collector). The general design for "real-time" audio processing is to have your processing threads running at real time scheduling (SCHED_FIFO on linux kernels, equivalent on other OSes), and those threads should never block. This means no system calls, no malloc, no IO of course, etc... Even paging is a problem (getting a page from disk to memory can easily take several ms), so you should lock some pages to be sure they are never swapped out.
You may be able to do those things in Java, but java makes it more complicated, not easier. I would look into a mixed design, where the core would be in C, and the rest (GUI, etc...) would be in java if you want.
One thing I didn't see in your question is whether you need to play out these processed samples or if you're doing something else with them (encoding them into a file, for example). I'd be more worried about the state of Java's sound engine than in how fast the JVM can crunch samples.
I pushed pretty hard on javax.sound.sampled a few years back and came away deeply unimpressed -- it doesn't compare with equivalent frameworks like OpenAL or Mac/iPhone's Core Audio (both of which I've used at a similar level of intensity). javax.sound.sampled requires you to push your samples into an opaque buffer of unknown duration, which makes synchronization nigh impossible. It's also poorly documented (very hard to find examples of streaming indeterminate-length audio over a Line as opposed to the trivial examples of in-memory Clips), has unimplemented methods (DataLine.getLevel()... whose non-implementation isn't even documented), and to top it off, I believe Sun laid off the last JavaSound engineer years ago.
If I had to use a Java engine for sound mixing and output, I'd probably try to use the JOAL bindings to OpenAL as a first choice, since I'd at least know the engine was currently supported and capable of very low-latency. Though I suspect in the long run that Nils is correct and you'll end up using JNI to call the native sound API.
Yes, Java is great for audio applications. You can use Java and access audio layers via Asio and have really low latency (64 samples latency which is next to nothing) on Windows platform. It means you will have lip-sync on video/movie. More latency on Mac as there is no Asio to "shortcut" the combination of OS X and "Java on top", but still OK. Linux also, but I am more ignorant. See soundpimp.com for a practical (and world first) example of Java and Asio working in perfect harmony. Also see the NRK Radio&tv Android app containing a sw mp3 decoder (from Java). You can do most audio things with Java, and then use a native layer if extra time critical.
Check out a library called Jsyn.
http://www.softsynth.com/jsyn/
Why not spend a day and write a simple java application that does minimal processing and validate whether the performance is adaquate.
From http://www.jsresources.org/faq_performance.html#java_slow
Let's collect some ethernal wisdom:
The earth is flat.
and, not to forget: Java is slow.
As several applications prove (see links section), Java is sufficient
to build audio editors, multitrack recording systems and MIDI
processing software. Try it out!
Related
I am trying to make a visual stimulus for an EEG study. The video is simply a flicker between a black frame and a white frame, and the alternation should occur at a range of rates: 12Hz, 24Hz, 48Hz, 72Hz.
Our monitors have a refresh rate of 144Hz and the computers are also fancy, and I am measuring the success of the videos with an oscilloscope to ensure accuracy. So, the hardware should not be an issue; theoretically, up to half the monitor's refresh rate should be possible. However, I have failed in both Java and MatLab.
I have tried using MatLab:
1) using imwrite() to make a gif
2) using the VideoWriter
3) using getframe() and movie2avi().
In all of these methods, the extra-high framerate is declared, and I can see in my command window that all the frames were inserted during the run. However, the final output file never exceeds 48Hz.
On top of that, the 48Hz as well as 24Hz and even 12Hz files also have serious timing issues with the frames.
I have also tried to make the files using Processing's MovieMaker: I set the framerate to 72Hz -- input a list of 72 .png files as frames -- and it should output a 1-second file that flickers at 72Hz.
However, the result only goes at 48Hz, and again the timing of the frames is not reliable.
I wouldn't be posting here if I hadn't exhausted my search; I'm really out of ideas. MatLab and Processing were both recommended ways of achieving this kind of high fps file, and both have big timing issues even with lower flicker frequencies. If anyone has any tips on increasing the temporal fidelity of the high-Hz flicker (graphics settings? codecs?), or of how to make it all the way to 72Hz, we'd greatly appreciate it!
As I said, I have only used Processing/Java and MatLab, so please feel free to recommend another platform.
THis is not an answer. It needs more than the comment-box, though, so please bear with me.
There are fundamental issues involved:
Simply drawing to whatever facility your OS/Graphics combo exposes, does not at all guarantee the drawed element to be present starting from the next frame (in all systems I know of).
This simply stems from the fact, that all these combos were explicitly NOT ment for an EEG Stimulus, but for consumption by visual understanding
Many combos offer lower-level facilities (e.g. OpenGL), that do carry such a promise, but come with other sets of problems, one of which is the less comfortable programming environment
With most OS/Harware combos, it might be less than trivial to sustain this stimulus - 144 Hz translates to less than 7ms - a time slot, that might be missed by a single bad scheduling decision of the OS or a need for a double-read even on a fast spinning disk. You would need to aim for some realtime-oriented OS dialect.
EDIT
After re-reading your question, I see you use Java. Forget it. a single GC break can easily be more than 7 ms.
There are a couple of free (as in beer and freedom) toolboxes for Matlab that wrap the low level openGL commands that you need in order to gain the type of control you want
MGL only runs on Mac but:
mgl is a set of matlab functions for displaying full screen visual stimuli from matlab. It is based on OpenGL functions, but abstracts these into more simple functions that can be used to code various kinds of visual stimuli.
Psychtoolbox runs on Mac, Windows, and Linux
The attraction of using computer displays for visual psychophysics is
that they allow software specification of the stimulus. Programs to
run experiments are often written in a low-level language (e.g. C or
Pascal) to achieve full control of the hardware for precise stimulus
display... The Psychophysics Toolbox is a software package that adds this capability to the Matlab and Octave application on Macintosh, Linux and Windows computers
It sounds like you are just starting out, in which case I would also suggest looking at the Python based PsychoPy
PsychoPy is an open-source package for running experiments in Python (a real and free alternative to Matlab). PsychoPy combines the graphical strengths of OpenGL with the easy
Python syntax to give scientists a free and simple stimulus
presentation and control package.
I plan to have ~100 synchronized audio tracks playing with live controllable volume per track. The audio comes from wav files.
Are there any libraries / best practices to achieve this in Java?
I'd recommend looking at Praxis and JAudioLibs. Neil Smith knows his stuff.
I've gotten maybe 30-50 wavs playing at once, and my library is certainly not the most efficient. And I wasn't changing volumes or pans for those wav files. If some of those sounds of yours are short enough to be stored in memory (e.g., a bank of sound effects) then you have a bit more chance of success in having so many tracks. I've gotten 100 short clips running at once (using a custom made clip, but again, no volume or panning added except the trigger values).
There's a good article on the real-time issues with Java. The issues are real but not insurmountable. Java is pretty low-level and fast, if you stick with the lowest level tools provided by javax.audio.sampled library, and don't mind a bit of latency.
http://quod.lib.umich.edu/cgi/p/pod/dod-idx?c=icmc;idno=bbp2372.2007.131
Controlling volume real-time is tough, as it requires either a very small buffer or a means of going in and doing per-frame calculations, or else risk getting effects like zippering or clicks. I had some success with real-time volume changing when I time-stamped mouse-motion-listener output and calculated the intermediate volumes. But I haven't tried applying that to more than one channel at a time. (It was a virtual Theremin project).
However, there might be a way to pre-build a selection of envelopes that would be quite efficient, and control volume via selecting from a menu of these envelopes. Hmmmm.
It's a challenging problem, and I don't know if even my DAW (Cakewalk Sonar, written in some flavor of C) would be able to handle this.
For the past few months, a developer and I have been working on a screensharing applet that streams to a media server like Wowza or Red5, but no matter what we do, we have about 5 seconds of latency, which is too long for a live application where people are interacting with each other. We've tried xuggle, different encoders, different players, different networks, different media servers, and even streaming locally, there's significant latency.
So, I'm beginning to wonder…
Is Java fast enough to do live screensharing?
I've seen lots of screen recording applets written in Java, but none of them are streaming live. Everything that's done live, such as GoToMeeting, seems to use C++. I'm thinking maybe there's a reason.
It's not a compression problem. Using ScreenVideo, we've compressed an hour-long stream down to about 100 MB, and we have plenty of bandwidth. The processor isn't overloaded doing the compression, either, but it seems to be taking too much time. We are getting the best results from some code pulled out of BigBlueButton, but still, the latency is terrible.
Streaming the WebCam, on the other hand, is nice and snappy. Almost no latency at all. So, the problem is the applet.
The only other idea I can think of is somehow emulating a WebCam with Java. Not sure if that would be faster or not.
Ideas? Or should I just give up on Java and do this in C++? I would hate to do that, because then I would have to create different versions for different platforms, but if it's the only way, it's the only way.
Many video streaming subsystems deliberately buffer so that a blip in connectivity doesn't impact the video, but that makes more sense in a recorded media scenario.
Make sure these systems have buffering turned off or turned down.
Also, while this isn't exactly scientific, you could run an app like wireshark on the outgoing and incoming computers, and try to see how long the traffic actually takes. If it's very fast, then I'd more seriously consider that buffering is the issue.
If you are on Windows, maybe just running Task Manager/Network tab would prove this or not (rather than installing something like wireshark, which isn't difficult... just trying to suggest a fast way to check)
I'm writing a large scale financial application that I use mostly Java for. Now, to get some data, I need to write a small script (<200 LOC) to download CSV files (over 20,000 of them) and store them to disk. I need this to be fast, but, a few minutes doesn't make a difference to me. I was planning to write it in Java which isn't very hard, but, I would be done a lot faster if I wrote it in Ruby, so I was wondering if there would be a large difference in speed between Ruby (or JRuby) and Java. The 20,000 files are all about 1/2 a megabyte, and the server I'm downloading from isn't keen to give away data (its completely legal, don't worry about that), so, my application has to randomly sleep in between, and, if the website denies a request, it has to sleep for 3 minutes.
Recommendations to any other easier-than-Java type of languages is welcome.
Use whatever makes you comfortable. Language implementation speed probably won't be an issue there, network speed and the sleeps you have to put in will be a bottleneck anyway.
Sounds like you app will be I/O bound, so the speed of the language is not terribly important
In a language like Ruby or Python, I would expect this to be more like 20 LOC or less. Especially since you have a limited request rate, there is no point using simultaneous connections to try to speed things up
If you have a bunch of machines with different ip addresses ( or one machine with several external addresses), you could split the job across those to speed things up since the rate limiting is usually by ip address
Where do your urls come from?
I'm developing a Java application that streams music via HTTP, and one problem I've come up against is that while the app is reading the audio file from disk and sending it to the client it usually maxes out the CPU at 90-100% (which can cause users problems running other apps).
Is it possible to control the thread doing this work to use less CPU, or does this need to be controlled by the OS? Are there any techniques for managing how intensive your application is at present?
I know you can start threads with a high/low priority, but this doesn't seem to have any effect for me in this scenario.
(I can't get my head past "I've asked the computer to do something, so it's obviously going to do it as fast as it can...")
Thanks!
rod.
That task (reading a file from the disk and sending it via HTTP) should not use any significant amount of CPU, especially at the bitrates required for music streaming (unless you're talking about multi-channel uncompressed PCM or something like that, but even then it should be I/O-bound and not use a lot of CPU).
You're probably doing the reading/writing in a very inefficient way. Do you read/write each byte separately or are you using some kind of buffer?
I would check how much buffering you are using. If you read/write one byte at a time you will consume a lot of CPU. However, if you are reading/writing blocks of say 4 kB it shouldn't use much CPU at all. If your network is the internet your CPU shouldn't be much over 10% of a single client.
One approximation for the buffer size is the bandwidth * delay. e.g. if you expect users to stream at 500 KB/s and there is a network latency of up to 0.1 sec, then the buffer size should be around 50 KB.
You can lower it's priority using methods in Thread (via Thread.currentThread() if necessary).
You can also put delays in it's processing loop (Thread.sleep()).
Other than that, let the O/S take care of it. If your program can use 100% CPU, and nothing else needs the CPU your app might as well use it rather than letting the O/S idle task have it.
It's also true that streaming data should be I/O bound, so you should definitely review what's being done between reading the data and sending it. Are you reading/sending byte by byte, unbuffered, for example?
EDIT: In response to marr75's comment, I am absolutely not advocating that you write poor, inefficient code which wastes CPU resources - There is an article on my web site which clearly conveys what I think about that mind-set. Rather, what I am saying is that if your code legitimately needs the CPU, and you've prioritized it to behave nicely if the user wants to do other things, then there is no point at all in artificially delaying the outcome just to avoid pegging the CPU - that only does the user the disservice of making them wait longer for the end result, which they presumably want as quickly as possible.
Do you have one or more of:
Software RAID
Compressed folder
Intrusive virus checker
Loopback file system
I don't think you can lower the priority without losing the functionality (stream music). Your program gets this much cpu from the OS, because it needs it. It's not like the OS is giving cpu-time away for no reason or because "it's in the mood for it".
If you think, you can do the task without using that much cpu-utilization, you can profile your app and find out, where this high cpu-utilization takes place and then try to improve your code.
I think you are doing the streaming in an inefficient way, but I say streaming CAN be a highly utilizing task.
I repeat, don't think about reducing the cpu-utilization by lowering the priority of the process or telling the OS "Don't give that much cpu-time to this process". That's the whole wrong intuition in my eyes. Reduce the cpu-utilization by improving the algorithms and code after profiling.
A good start in profiling java is this article: http://www.ibm.com/developerworks/edu/os-dw-os-ecl-tptp.html
In addition to the information given above: the JVM is free in how it uses OS threads. The Thread in your Java application might run in a seperate OS thread, or it might share that thread with other Threads. Check the documentation for the JVM you are using for additional information.
Ok, thanks for the advice guys! Looks like I'm just going to have to look into trying to improve the efficiency of the way my app is streaming (though not sure this is going to go far as I'm basically just reading the file from disk and writing it to the client...).
VisualVM is very easy to use to find out where your CPU time is being spent for Java applications, and it is included in the latest versions of the JDK (named jvisualvm.exe on Windows)
Follow up to my "well thought out buffers" comment, a good rule of thumb for TCP buffering,
buffer size = 2 * bandwidth * delay
So if you want to stream 214kbps music (around 27kB/s) and have, let's say 60ms of latency, you're looking at 3.24 kilobytes, and rounding off to a nice 4kB buffer will work very well for you on a wide range of systems.