I plan to have ~100 synchronized audio tracks playing with live controllable volume per track. The audio comes from wav files.
Are there any libraries / best practices to achieve this in Java?
I'd recommend looking at Praxis and JAudioLibs. Neil Smith knows his stuff.
I've gotten maybe 30-50 wavs playing at once, and my library is certainly not the most efficient. And I wasn't changing volumes or pans for those wav files. If some of those sounds of yours are short enough to be stored in memory (e.g., a bank of sound effects) then you have a bit more chance of success in having so many tracks. I've gotten 100 short clips running at once (using a custom made clip, but again, no volume or panning added except the trigger values).
There's a good article on the real-time issues with Java. The issues are real but not insurmountable. Java is pretty low-level and fast, if you stick with the lowest level tools provided by javax.audio.sampled library, and don't mind a bit of latency.
http://quod.lib.umich.edu/cgi/p/pod/dod-idx?c=icmc;idno=bbp2372.2007.131
Controlling volume real-time is tough, as it requires either a very small buffer or a means of going in and doing per-frame calculations, or else risk getting effects like zippering or clicks. I had some success with real-time volume changing when I time-stamped mouse-motion-listener output and calculated the intermediate volumes. But I haven't tried applying that to more than one channel at a time. (It was a virtual Theremin project).
However, there might be a way to pre-build a selection of envelopes that would be quite efficient, and control volume via selecting from a menu of these envelopes. Hmmmm.
It's a challenging problem, and I don't know if even my DAW (Cakewalk Sonar, written in some flavor of C) would be able to handle this.
Related
I am trying to make a visual stimulus for an EEG study. The video is simply a flicker between a black frame and a white frame, and the alternation should occur at a range of rates: 12Hz, 24Hz, 48Hz, 72Hz.
Our monitors have a refresh rate of 144Hz and the computers are also fancy, and I am measuring the success of the videos with an oscilloscope to ensure accuracy. So, the hardware should not be an issue; theoretically, up to half the monitor's refresh rate should be possible. However, I have failed in both Java and MatLab.
I have tried using MatLab:
1) using imwrite() to make a gif
2) using the VideoWriter
3) using getframe() and movie2avi().
In all of these methods, the extra-high framerate is declared, and I can see in my command window that all the frames were inserted during the run. However, the final output file never exceeds 48Hz.
On top of that, the 48Hz as well as 24Hz and even 12Hz files also have serious timing issues with the frames.
I have also tried to make the files using Processing's MovieMaker: I set the framerate to 72Hz -- input a list of 72 .png files as frames -- and it should output a 1-second file that flickers at 72Hz.
However, the result only goes at 48Hz, and again the timing of the frames is not reliable.
I wouldn't be posting here if I hadn't exhausted my search; I'm really out of ideas. MatLab and Processing were both recommended ways of achieving this kind of high fps file, and both have big timing issues even with lower flicker frequencies. If anyone has any tips on increasing the temporal fidelity of the high-Hz flicker (graphics settings? codecs?), or of how to make it all the way to 72Hz, we'd greatly appreciate it!
As I said, I have only used Processing/Java and MatLab, so please feel free to recommend another platform.
THis is not an answer. It needs more than the comment-box, though, so please bear with me.
There are fundamental issues involved:
Simply drawing to whatever facility your OS/Graphics combo exposes, does not at all guarantee the drawed element to be present starting from the next frame (in all systems I know of).
This simply stems from the fact, that all these combos were explicitly NOT ment for an EEG Stimulus, but for consumption by visual understanding
Many combos offer lower-level facilities (e.g. OpenGL), that do carry such a promise, but come with other sets of problems, one of which is the less comfortable programming environment
With most OS/Harware combos, it might be less than trivial to sustain this stimulus - 144 Hz translates to less than 7ms - a time slot, that might be missed by a single bad scheduling decision of the OS or a need for a double-read even on a fast spinning disk. You would need to aim for some realtime-oriented OS dialect.
EDIT
After re-reading your question, I see you use Java. Forget it. a single GC break can easily be more than 7 ms.
There are a couple of free (as in beer and freedom) toolboxes for Matlab that wrap the low level openGL commands that you need in order to gain the type of control you want
MGL only runs on Mac but:
mgl is a set of matlab functions for displaying full screen visual stimuli from matlab. It is based on OpenGL functions, but abstracts these into more simple functions that can be used to code various kinds of visual stimuli.
Psychtoolbox runs on Mac, Windows, and Linux
The attraction of using computer displays for visual psychophysics is
that they allow software specification of the stimulus. Programs to
run experiments are often written in a low-level language (e.g. C or
Pascal) to achieve full control of the hardware for precise stimulus
display... The Psychophysics Toolbox is a software package that adds this capability to the Matlab and Octave application on Macintosh, Linux and Windows computers
It sounds like you are just starting out, in which case I would also suggest looking at the Python based PsychoPy
PsychoPy is an open-source package for running experiments in Python (a real and free alternative to Matlab). PsychoPy combines the graphical strengths of OpenGL with the easy
Python syntax to give scientists a free and simple stimulus
presentation and control package.
For the past few months, a developer and I have been working on a screensharing applet that streams to a media server like Wowza or Red5, but no matter what we do, we have about 5 seconds of latency, which is too long for a live application where people are interacting with each other. We've tried xuggle, different encoders, different players, different networks, different media servers, and even streaming locally, there's significant latency.
So, I'm beginning to wonder…
Is Java fast enough to do live screensharing?
I've seen lots of screen recording applets written in Java, but none of them are streaming live. Everything that's done live, such as GoToMeeting, seems to use C++. I'm thinking maybe there's a reason.
It's not a compression problem. Using ScreenVideo, we've compressed an hour-long stream down to about 100 MB, and we have plenty of bandwidth. The processor isn't overloaded doing the compression, either, but it seems to be taking too much time. We are getting the best results from some code pulled out of BigBlueButton, but still, the latency is terrible.
Streaming the WebCam, on the other hand, is nice and snappy. Almost no latency at all. So, the problem is the applet.
The only other idea I can think of is somehow emulating a WebCam with Java. Not sure if that would be faster or not.
Ideas? Or should I just give up on Java and do this in C++? I would hate to do that, because then I would have to create different versions for different platforms, but if it's the only way, it's the only way.
Many video streaming subsystems deliberately buffer so that a blip in connectivity doesn't impact the video, but that makes more sense in a recorded media scenario.
Make sure these systems have buffering turned off or turned down.
Also, while this isn't exactly scientific, you could run an app like wireshark on the outgoing and incoming computers, and try to see how long the traffic actually takes. If it's very fast, then I'd more seriously consider that buffering is the issue.
If you are on Windows, maybe just running Task Manager/Network tab would prove this or not (rather than installing something like wireshark, which isn't difficult... just trying to suggest a fast way to check)
I'm developing a game for Android. It uses a surface view and uses the sort of standard 2D drawing APIs provided. When I first released the game, I was doing all sorts of daft things like re-drawing 9-patches on each frame and likewise with text. I have since optimised much of this by drawing to Bitmap objects and drawing them each frame, only re-drawing onto the Bitmap objects when required.
I've received complaints about battery drain before, and following my modifications I'd like to know (scientifically) if I've made any improvements. Unfortunately, I don't have any prior data to go by, so it would be most useful to compare the performance to some other game.
I've been running Traceview, and using the results of it mostly for the purposes of identifying CPU-time-consuming methods.
So -- what's the best way of determining my app's battery performance, and what's a good benchmark?
I know I can look at the %s of different apps through the settings, but this is again unscientific, as the figure I get from this also depends on what's happening in all of the other apps. I've looked through (most of) Google's documentation, and although the message is clear that you should be saving battery (and it gives the occasional tip as to how), there is little indication of how I can measure how well my app is performing. The last thing I want are more complaints of battery drain in the Android Market!
Thanks in advance.
EDIT
Thanks for all your helpful advice/suggestions. What I really want to know is how I can use the data coming from Traceview (ie: CPU time in ms spent on each frame of the game) to determine battery usage (if this is at all possible). Reading back on my original question, I can see I was a bit vague. Thanks again.
Here is my suggestion:
I watch power consumption while developing my apps (that sometimes poll the sensors at rates of <25ns) using PowerTutor. Check it out, it sounds like this maybe what you are looking for, the app tells you what you are using in mW, J, or relative to the rest of the system. Also, results are broken down by CPU, WiFi, Display, (other radios installed). The only catch is that it is written for a specific phone model, but I use it with great success on my EVO 4G, Galaxy S (Sprint Epic), and Hero.
Good luck,
-Steve
There is a possibility that your game is draining battery. I believe this depends on several reasons, which reads as follows:
Your application is a game. Games drains battery quickly.
You're iterating with help from a Thread. Have you limited the FPS to make the CPU skip unnecessary iterations? Since you're working with 2D I assume you're using the SurfaceView. 60 FPS will be enough for a real-time game.
You don't stop the Thread when your application terminates. Hence you reiterate code when your application isn't alive.
Have you an iterate lock that does wait(); during onPause?
The people commenting that your game is leaking battery probably aims when your application isn't in use. Otherwise, it would be wierd because every game on Android Market drains battery - more or less.
If you're trying to gauge the "improvement over your previous version", I don't think it makes sense to compare to another game! Unless those two games do the exact thing, this is as unscientific as it gets.
Instead, I would grab the previous version of your app from source control, run it, measure it, and then run it with the latest code and compare it again.
To compare, you could for example use the command line tool "top" (definitely available in busybox if your phone is rooted, not sure if it comes with a stock phone. Probably not). That shows you the CPU usage of your process in percent.
There is a battery profiler developed by Qualcomm called Trepn Profiler: https://developer.qualcomm.com/mobile-development/increase-app-performance/trepn-profiler
how I can use the data coming from Traceview (ie: CPU time in ms spent on each frame of the game) to determine battery usage (if this is at all possible)
In theory it would be possible to extrapolate the battery usage for your app by looking at the power consumption on a frame by frame basis. The best way to accomplish this would be to evaluate the power consumption of the CPU (only) for a given period (say two seconds) while your app is running the most CPU intensive operation, (additionally, GPU power usage could be gleaned this way also) while recording TraceView data (such as frames per second or flops per second) giving you the the traffic across the CPU/GPU for a given millisecond. Using this data you could accurately calculate the average peak power consumption for your app by running the above test a few times.
Here is why I say it is theory only: There are many variables to consider:
The number and nature of other processes running at the time of the above test (processor intensive)
Method of evaluating the power draw across the CPU/GPU (while tools such as PowerTutor are effective for evaluating power consumption, in this case the evaluation would not be as effective because of the need to collect time stamped power usage data. Additionally, just about any method of collecting power data would introduce an additional overhead (Schrödinger's cat) but that strictly depends on the level of accuracy you require/desire.)
The reason for the power consumption information - If you are looking to define the power consumption of your app for testing or BETA testing/evaluation purposes then it is a feasible task with some determination and the proper tools. If you are looking to gain usable information about power consumption "in the wild", on user's devices, then I would say it is plausible but not realistic. The vairables involved would make even the most determined and dedicated researcher faint. You would have to test on every possible combination of device/Android version in the wild. Additionally, the combinations of running processes/threads and installed apps is likely incalculable.
I hope this provides some insight to your question, although I may have gone deeper into it than needed.
-Steve
For anyone looking, one resource we've been using that is extremely helpful is a free app from AT&T called ARO.
Give it a look: ARO
It has helped me before and I don't see it mentioned very often so thought I'd drop it here for anyone looking.
"I know I can look at the %s of
different apps through the settings,
but this is again unscientific, as the
figure I get from this also depends on
what's happening in all of the other
apps."
The first thing I'd do is hunt for an app already out there that has a known, consistent battery usage, and then you can just use that as a reference to determine your app's usage.
If there is no such app, you will have to hope for an answer from someone else... and if you are successful making such an app, I would suggest selling your new "battery usage reference" app so that other programmers could use it. :)
I know this question is old and it's late, but for anyone who comes here looking for a solution I suggest you take a look at JouleUnit test:
http://dnlkntt.wordpress.com/2013/09/28/how-to-test-energy-consumption-on-android-devices/
It integrates into eclipse and gives you a great amount of detail about how much battery your app is consuming.
I know of three options that can help you for a having scientific measure:
Use a hardware specifically built for this. Monsoon HIGH VOLTAGE POWER MONITOR.
https://msoon.github.io/powermonitor/PowerTool/doc/Power%20Monitor%20Manual.pdf
Download and install Trepn Profiler (a tools from Qualcomm) on your phone. You wont need a computer for reporting. Reports are live and realtime on the phone. You can download Trepn Profiler from the following link: https://play.google.com/store/apps/details?id=com.quicinc.trepn&hl=en_US
Please take note that for recent phone (with android 6+) it works in estimation mode. If you need accurate numbers, you need to a list of select devices. Check the following link for the list:
https://developer.qualcomm.com/software/trepn-power-profiler/faq
You can profile apps separately, and the whole system.
Use Batterystats and Battery Historian from google.
https://developer.android.com/studio/profile/battery-historian
I'm writing a large scale financial application that I use mostly Java for. Now, to get some data, I need to write a small script (<200 LOC) to download CSV files (over 20,000 of them) and store them to disk. I need this to be fast, but, a few minutes doesn't make a difference to me. I was planning to write it in Java which isn't very hard, but, I would be done a lot faster if I wrote it in Ruby, so I was wondering if there would be a large difference in speed between Ruby (or JRuby) and Java. The 20,000 files are all about 1/2 a megabyte, and the server I'm downloading from isn't keen to give away data (its completely legal, don't worry about that), so, my application has to randomly sleep in between, and, if the website denies a request, it has to sleep for 3 minutes.
Recommendations to any other easier-than-Java type of languages is welcome.
Use whatever makes you comfortable. Language implementation speed probably won't be an issue there, network speed and the sleeps you have to put in will be a bottleneck anyway.
Sounds like you app will be I/O bound, so the speed of the language is not terribly important
In a language like Ruby or Python, I would expect this to be more like 20 LOC or less. Especially since you have a limited request rate, there is no point using simultaneous connections to try to speed things up
If you have a bunch of machines with different ip addresses ( or one machine with several external addresses), you could split the job across those to speed things up since the rate limiting is usually by ip address
Where do your urls come from?
Is Java a suitable alternative to C / C++ for realtime audio processing?
I am considering an app with ~100 (at max) tracks of audio with delay lines (30s # 48khz), filtering (512 point FIR?), and other DSP type operations occurring on each track simultaneously.
The operations would be converted and performed in floating point.
The system would probably be a quad core 3GHz with 4GB RAM, running Ubuntu.
I have seen articles about Java being much faster than it used to be, coming close to C / C++, and now having realtime extensions as well. Is this reality? Does it require hard core coding and tuning to achieve the %50-%100 performance of C some are spec'ing?
I am really looking for a sense if this is possible and a heads up for any gotchas.
For an audio application you often have only very small parts of code where most of the time is spent.
In Java, you can always use the JNI (Java Native interface) and move your computational heavy code into a C-module (or assembly using SSE if you really need the power). So I'd say use Java and get your code working. If it turns out that you don't meet your performance goal use JNI.
90% of the code will most likely be glue code and application stuff anyway. But keep in mind that you loose some of the cross platform features that way. If you can live with that JNI will always leave you the door open for native code performance.
Java is fine for many audio applications. Contrary to some of the other posters, I find Java audio a joy to work with. Compare the API and resources available to you to the horrendous, barely documented mindf*k that is CoreAudio and you'll be a believer. Java audio suffers from some latency issues, though for many apps this is irrelevant, and a lack of codecs. There are also plenty of people who've never bothered to take the time to write good audio playback engines(hint, never close a SourceDataLine, instead write zeros to it), and subsequently blame Java for their problems. From an API point of view, Java audio is very straightforward, very easy to use, and there is lots and lots of guidance over at jsresources.org.
Sure, why not?
The crucial questions (independent of language, this is from queueing theory) are:
what is the maximum throughput you need to handle (you've specified 100 x 48kHz, is that mono or stereo, how many bits equivalent at that frequency?)
can your Java routines keep up with this rate on the average?
what is the maximum permissible latency?
If your program can keep up with the throughput on the average, and you have enough room for latency, then you should be able to use queues for inputs and outputs, and the only parts of the program that are critical for timing are the pieces that put the data into the input queue and take it out of the output queue and send it to a DAC/speaker/whatever.
Delay lines have low computational load, you just need enough memory (+ memory bandwidth)... in fact you should probably just use the input/output queues for it, i.e. start putting data into the input queue immediately, and start taking data out of the output queue 30s later. If it's not there, your program is too slow...).
FIRs are more expensive, that's probably going to be the bottleneck (& what you'd want to optimize) unless you have some other ugly nasty operation in mind.
I think latency will be your major problem - it is quite hard to maintain latency already in C/C++ on modern OSes, and java surely adds to the problem (garbage collector). The general design for "real-time" audio processing is to have your processing threads running at real time scheduling (SCHED_FIFO on linux kernels, equivalent on other OSes), and those threads should never block. This means no system calls, no malloc, no IO of course, etc... Even paging is a problem (getting a page from disk to memory can easily take several ms), so you should lock some pages to be sure they are never swapped out.
You may be able to do those things in Java, but java makes it more complicated, not easier. I would look into a mixed design, where the core would be in C, and the rest (GUI, etc...) would be in java if you want.
One thing I didn't see in your question is whether you need to play out these processed samples or if you're doing something else with them (encoding them into a file, for example). I'd be more worried about the state of Java's sound engine than in how fast the JVM can crunch samples.
I pushed pretty hard on javax.sound.sampled a few years back and came away deeply unimpressed -- it doesn't compare with equivalent frameworks like OpenAL or Mac/iPhone's Core Audio (both of which I've used at a similar level of intensity). javax.sound.sampled requires you to push your samples into an opaque buffer of unknown duration, which makes synchronization nigh impossible. It's also poorly documented (very hard to find examples of streaming indeterminate-length audio over a Line as opposed to the trivial examples of in-memory Clips), has unimplemented methods (DataLine.getLevel()... whose non-implementation isn't even documented), and to top it off, I believe Sun laid off the last JavaSound engineer years ago.
If I had to use a Java engine for sound mixing and output, I'd probably try to use the JOAL bindings to OpenAL as a first choice, since I'd at least know the engine was currently supported and capable of very low-latency. Though I suspect in the long run that Nils is correct and you'll end up using JNI to call the native sound API.
Yes, Java is great for audio applications. You can use Java and access audio layers via Asio and have really low latency (64 samples latency which is next to nothing) on Windows platform. It means you will have lip-sync on video/movie. More latency on Mac as there is no Asio to "shortcut" the combination of OS X and "Java on top", but still OK. Linux also, but I am more ignorant. See soundpimp.com for a practical (and world first) example of Java and Asio working in perfect harmony. Also see the NRK Radio&tv Android app containing a sw mp3 decoder (from Java). You can do most audio things with Java, and then use a native layer if extra time critical.
Check out a library called Jsyn.
http://www.softsynth.com/jsyn/
Why not spend a day and write a simple java application that does minimal processing and validate whether the performance is adaquate.
From http://www.jsresources.org/faq_performance.html#java_slow
Let's collect some ethernal wisdom:
The earth is flat.
and, not to forget: Java is slow.
As several applications prove (see links section), Java is sufficient
to build audio editors, multitrack recording systems and MIDI
processing software. Try it out!