We Keep Coding

Audio Noise removal in java [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Need an algorithm (or a set of good ones to compare for various input data) that will reduce the noise level of a voice audio signal without distorting the signal appreciably using Java.
Input is an audio signal that includes a voice along with some background noises. The noise varies over the course of the recording. There definitely exists ways to remove noise like this, developed for voice recognition and movie making.
The desired output is a minimally distorted voice signal with the background distractions minimally audible to the human ear. The quantitative criteria are minimized
Signal to noise ratio and
Total harmonic distortion.

You are looking for adaptive noise removal and possibly a variety that adapts to changing noise conditions over time within the same stream or file.
Older approaches include:
Remove frequencies containing the majority of noise using bandpass and/or notch filters (which only works well if the desired signal and the noise band do not intersect)
Drop the noise level at points between words, notes, or other audio events (in the dead space) as does the Dolby noise reduction system scheme
Drop the noise floor across an entire file using a Hamming window or other window in conjunction with an FFT library
Hand edit sections of an audio track in programs like Cakewalk or its
competitors
These methods have been found less than desirable when trying to clean up a larger file or multiple files or in real time applications such as real time voice recognition or telephony.
One of the Java programs that I have not personally tried his here. Even though it has some level of automation, it is an LSE (least squares estimator) which works across a block of data but is not suitable for continuous operation or an audio file with changing unwanted noise conditions. (It's not as adaptive as one might hope.)
The solution that I found after much investigation and now use all the time is not written in Java. It is a MATLAB program that can also run in open source Octave with minimal modification. I started porting it to C++ but ran out of time to finish that.
The class of algorithm it implements is called MMSE (Noise Reduction Using Minimum Mean Square Estimators). The MATLAB version has been refined several times by Dr. Hendricks, concluding with the the 2010 version.
I've compared it with its competitors for both dialog and music and it equals or excels beyond the others in all cases I've tried. (I have no affiliation with Dr. Hendricks or MATLAB other than I like the results I've been getting from his implementation on that platform.)

Quickly exchanging data between two computers [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I have been experimenting with ways to use the processing power of two computers together as one (not by physically connecting them, but by splitting the task in half and each computer does a half, then the result from the "helper" computer is sent back to be combined with the result from the "main" computer via internet)
I've been using this method to compute fractal images and it works great. The left half and the right half of the image are computed on separate computers, then combined into one. The process of sending one half of the image to the other computer and combining them takes maybe a second, so the efficiency is great and cuts time down by about half.
The problem comes when you want to do this "multi computer processing" with something that needs data exchanged very frequently.
For example, I'd like to use this for something like an n-body simulation. You need the data exchange to happen multiple times per second, so if the exchange takes about a second it actually takes much longer to try and use two computers then it would with one.
So how do online video games do it? The players around you, what they are doing, what they are wearing, everything going on has to be exchanged between everyone playing many times per second.
I'm just looking for general ideas on how to send larger amounts of data and at fast speeds.
The way I have been doing it is with PHP on a free hosting site. The helper computer will compute its half of the data then sends it to the PHP file which saves that data somewhere. Then the main computer reads this and combines it with the data it computed already.
I have a feeling PHP isn't the way to go, but I don't know much about this sort of thing.

Your first step will be to move from using HTTP Requests to using Sockets directly - this will give you much more control over the communication, and give you improved performance by reducing the overhead of the HTTP protocol (this is potentially pretty significant). Plus, with sockets you can more easily have your programs communicate to each other directly, rather than through the PHP-based software.
There are a ton of guides online as to how you would do this sort of system, and I would recommend Googling things like "game networking" and "distributed computing".
Here is one series of articles that I have found useful in the past, that covers the sort of things that you will want to read about: http://gafferongames.com/networking-for-game-programmers/
(He doesn't use Java, but the ideas are universal)

What are the major differences and benefits of Porter and Lancaster Stemming algorithms? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I'm Working on document classification tasks in java.
Both algorithms came highly recommended, what are the benefits and disadvantages of each and which is more commonly used in the literature for Natural Language Processing tasks?

At the very basics of it, the major difference between the porter and lancaster stemming algorithms is that the lancaster stemmer is significantly more aggressive than the porter stemmer. The three major stemming algorithms in use today are Porter, Snowball(Porter2), and Lancaster (Paice-Husk), with the aggressiveness continuum basically following along those same lines. Porter is the least aggressive algorithm, with the specifics of each algorithm actually being fairly lengthy and technical. Here is a break down for you though:
Porter: Most commonly used stemmer without a doubt, also one of the most gentle stemmers. One of the few stemmers that actually has Java support which is a plus, though it is also the most computationally intensive of the algorithms(Granted not by a very significant margin). It is also the oldest stemming algorithm by a large margin.
Porter2: Nearly universally regarded as an improvement over porter, and for good reason. Porter himself in fact admits that it is better than his original algorithm. Slightly faster computation time than porter, with a fairly large community around it.
Lancaster: Very aggressive stemming algorithm, sometimes to a fault. With porter and snowball, the stemmed representations are usually fairly intuitive to a reader, not so with Lancaster, as many shorter words will become totally obfuscated. The fastest algorithm here, and will reduce your working set of words hugely, but if you want more distinction, not the tool you would want.
Honestly, I feel that Snowball is usually the way to go. There are certain circumstances in which Lancaster will hugely trim down your working set, which can be very useful, however the marginal speed increase over snowball in my opinion is not worth the lack of precision. Porter has the most implementations though and so is usually the default go-to algorithm, but if you can, use snowball.
Snowball - Additional info
Snowball is a small string processing language designed for creating
stemming algorithms for use in Information Retrieval.
The Snowball compiler translates a Snowball script into another
language - currently ISO C, C#, Go, Java, Javascript, Object Pascal,
Python and Rust are supported.
History of the name
Since it effectively provides a ‘suffix STRIPPER GRAMmar’, I had toyed
with the idea of calling it ‘strippergram’, but good sense has
prevailed, and so it is ‘Snowball’ named as a tribute to SNOBOL, the
excellent string handling language of Messrs Farber, Griswold, Poage
and Polonsky from the 1960s.
---Martin Porter
Stemmers implemented in the Snowball language are sometimes simply referred to as Snowball stemmers. For example, see the Natural Language Toolkit: nltk.stem.snowball.

Are there free realtime financial data feeds since the demise of OpenQuant? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
Now that the oligopole of market data providers successfully killed OpenQuant, does any alternative to proprietary and expensive subscriptions for realtime market data subsist?
Ideally I would like to be able to monitor tick by tick securities from the NYSE, NASDAQ and AMEX (about 6000 symbols).
Most vendors put a limit of 500 symbols watchable at the same time, this is unacceptable to me, even if one can imagine a rotation among the 500 symbols ie. making windows of 5 sec. of effective observation out of each minute for every symbol.
Currently I'm doing this by a Java thread pool calling Google Finance, but this is unsatisfactory for several reasons, one being that Google doesn't return the volume traded, but the main one being that Google promptly is killing bots attempting to take advantage of this service ;-)
Any hint much appreciated,
Cheers

I think you'll find all you need to know by looking at this question: source of historical stock data
I don't know of any free data feeds other than Yahoo!, but it doesn't offer tick-by-tick data, it only offers 1 minute intervals with a 15 minute delay. If you want to use an already existing tool to download the historical data, then I would recommend EclipseTrader. It only saves the Open, Close, High, Low, and Volume.
(source: divbyzero.com)
You can write your own data scraper with very little effort. I've written an article on downloading real-time data from yahoo on my blog, but it's in C#. If you're familiar with C# then you'll be able to translate the action in Java pretty quickly. If you write your own data scraper then you can get pretty much ANYTHING that Yahoo! shows on their web site: Bid, Ask, Dividend Share, Earnings Share, Day's High, Day's Low, etc, etc, etc.
If you don't know C# then don't worry, it's REALLY simple: Yahoo allows you to download CSV files with quotes just by modifying a URL. You can find out everything about the URL and the tags that are used on yahoo here: http://www.gummy-stuff.org/Yahoo-data.htm
Here are the basic steps you need to follow:
Construct a URL for the symbol or multiple symbols of your choice.
Add the tags which you're interested in downloading (Open, Close, Volume, Beta, 52 week high, etc, etc.).
Create a URLConnection with the URL you just constructed.
Use a BufferedReader to read the CSV file that is returned from the connection stream.
Your CSV will have the following format:
Each row is a different symbol.
Each column is a different tag.

Open a TDAmeritrade account and you will have free access to ThinkOrSwim real time trading and quotes platform. Live trading is real time and paper trading is delayed 15 minutes. I forget what the minimum required is to open a TDAmeritrade account but you can go to TDAMeritrade.com or thinkorswim.com to check them out.

Intrinio has a bunch of feeds with free and paid tiers. Essentially you only have to pay for what you need as opposed to the bigger data suppliers. Intrinio focuses on data quality and caters to developers as well, so I think it'd be a great option for you.
full disclosure - I work at Intrinio as a developer

There's a handy function in Google Sheets (ImportHTML) which I've been using for a while to reasonable effect.
For example -
=ImportHTML("http://www.bloomberg.com/markets/commodities/futures/metals/","table",1),5,3) returns the EUR Gold spot price.
It works with Yahoo too, so =Index(ImportHTML("http://finance.yahoo.com/q?s=DX-Y.NYB","table",0),2,2) returns the DXY.
The data updates with some small delay but it's usable.

Music transcription [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm working on a polyphonic music transcription project. I have read some papers and gone through articles which explains similar tasks. I'am very confused about many aspects of the problem domain. Hope someone will be able to help me.
So far I have obtained a stream of decoded audio data from a given mp3.
I have understood onset detection is the first step towards transcription. . Is there any java library available which can be used for detecting onsets.
Next, detecting the fundamental frequency is also done with the use of FFT as I have read.
I want to know the use of FFT in these tasks. (I'm not familiar with FFT much). Is it absolutely necessary to perform FFT for onset detection and F0 detection.
If not what are the other ways.
Can I perform the FFT on the audiostream I have, or is there some other operation that has to be done in order to manipulate these audio data.
Thanks a lot.

This field is known as machine listening.
Polyphonic transcription of digitally encoded music is one of the holy grails of machine listening. It is an unsolved problem, and an area of active research. The sub-fields include:
Onset detection
Beat extraction (detection of the metric structure, time sig, etc)
Pitch detection (possible using auto-correllation, and other methods, on monophonic signals, but an unsolved problem when applied to complex polyphonic music)
Key detection (key signature detection).
Depending on the nature of your project, you might find it useful to explore the SuperCollider programming environment. SC is a language designed for projects such as this, already has a large number of machine listening plugins (ugens), and a comprehensive framework for dealing with FFT, audio signals, and much more.

This question about note onset detection contains a lot of information which may be useful to you.
This sounds a huge but very interesting project, good luck to you.

Music transcription means creating music notation from sound (or audio data). While accomplished musicians and especially composers are able to do this, it's an extremely difficult task to do with a machine, and as far as i know, there has been little success so far - mostly academic experiments.
Basically, to recognize notes, you want to know where they start, where they end, and what is their pitch. Fourier transform is the most basic way to turn time domain (audio data) to frequency domain (pitches) - in principle. In practice, musical instruments generate lots of harmonics (overtones) and if we have polyphony (many F0s) added, it's a mess.
You could try feeding something like 50 millisecond sequential slices of the audio data to the FFT. This way you would get the spectrum of each slice, then detect the strongest peaks in each slice, and infer the rhythm from what happens between successive slices.
Sorry, I couldn't help much... But just wanted to point out that what you're trying to do is extremely difficult, seriously. Perhaps you should start from something simpler, like detecting one-note sine wave melodies. Good luck!

For detecting the fundamental frequency of the melody in polyphonic music you can try out the MELODIA vamp plug-in (non-commercial use only): http://mtg.upf.edu/technologies/melodia
If you want to implement a melody extraction algorithm yourself you're going to have to check out the current state-of-the-art in research, a good place to start might be the MIREX melody extraction annual evaluation campaign: http://www.music-ir.org/mirex/wiki/Audio_Melody_Extraction
That, or just google "melody extraction" ;)