How to reduce noise in signal overlap add in Java? - java

I have been working with a java program (developed by other people) for text-to-speech synthesis. The synthesis is done by concatenation of "di-phones". In the oroginal version, there was no signal processing. The diphones were just collected and concatenated together to produce the output. In order to improve the output, I tried to perform "phase matching" of the concatenating speech signals. The modification I've made is summarized here:
Audio data is collected from the AudioInputStream into a byte array.
Since the audio data is 16 bit, I converted the byte array to a short
array.
The "signal processing" is done on the short array.
To output the audio data, short array is again converted to byte array.
Here's the part of the code that I've changed in the existing program:
Audio Input
This segment is called for every diphone.
Original Version
audioInputStream = AudioSystem.getAudioInputStream(sound);
while ((cnt = audioInputStream.read(byteBuffer, 0, byteBuffer.length)) != -1) {
if (cnt > 0) {
byteArrayPlayStream.write(byteBuffer, 0, cnt);
}
}
My Version
// public varialbe declarations
byte byteSoundFile[]; // byteSoundFile will contain a whole word or the diphones of a whole word
short shortSoundFile[] = new short[5000000]; // sound contents are taken in a short[] array for signal processing
short shortBuffer[];
int pos = 0;
int previousPM = 0;
boolean isWord = false;
public static HashMap<String, Integer> peakMap1 = new HashMap<String, Integer>();
public static HashMap<String, Integer> peakMap2 = new HashMap<String, Integer>();
// code for receiving and processing audio data
if(pos == 0) {
// a new word is going to be processed.
// so reset the shortSoundFile array
Arrays.fill(shortSoundFile, (short)0);
}
audioInputStream = AudioSystem.getAudioInputStream(sound);
while ((cnt = audioInputStream.read(byteBuffer, 0, byteBuffer.length)) != -1) {
if (cnt > 0) {
byteArrayPlayStream.write(byteBuffer, 0, cnt);
}
}
byteSoundFile = byteArrayPlayStream.toByteArray();
int nSamples = byteSoundFile.length;
byteArrayPlayStream.reset();
if(nSamples > 80000) { // it is a word
pos = nSamples;
isWord = true;
}
else { // it is a diphone
// audio data is converted from byte to short, so nSamples is halved
nSamples /= 2;
// transfer byteSoundFile contents to shortBuffer using byte-to-short conversion
shortBuffer = new short[nSamples];
for(int i=0; i<nSamples; i++) {
shortBuffer[i] = (short)((short)(byteSoundFile[i<<1]) << 8 | (short)byteSoundFile[(i<<1)+1]);
}
/************************************/
/**** phase-matching starts here ****/
/************************************/
int pm1 = 0;
int pm2 = 0;
String soundStr = sound.toString();
if(soundStr.contains("\\") && soundStr.contains(".")) {
soundStr = soundStr.substring(soundStr.indexOf("\\")+1, soundStr.indexOf("."));
}
if(peakMap1.containsKey(soundStr)) {
// perform overlap and add
System.out.println("we are here");
pm1 = peakMap1.get(soundStr);
pm2 = peakMap2.get(soundStr);
/*
Idea:
If pm1 is located after more than one third of the samples,
then threre will be too much overlapping.
If pm2 is located before the two third of the samples,
then where will also be extra overlapping for the next diphone.
In both of the cases, we will not perform the peak-matching operation.
*/
int idx1 = (previousPM == 0) ? pos : previousPM - pm1;
if((idx1 < 0) || (pm1 > (nSamples/3))) {
idx1 = pos;
}
int idx2 = idx1 + nSamples - 1;
for(int i=idx1, j=0; i<=idx2; i++, j++) {
if(i < pos) {
shortSoundFile[i] = (short) ((shortSoundFile[i] >> 1) + (shortBuffer[j] >> 1));
}
else {
shortSoundFile[i] = shortBuffer[j];
}
}
previousPM = (pm2 < (nSamples/3)*2) ? 0 : idx1 + pm2;
pos = idx2 + 1;
}
else {
// no peak found. simply concatenate the audio data
for(int i=0; i<nSamples; i++) {
shortSoundFile[pos++] = shortBuffer[i];
}
previousPM = 0;
}
Audio Output
After collecting all the diphones of a word, this segment is called to play the audio output.
Original Version
byte audioData[] = byteArrayPlayStream.toByteArray();
... code for writing audioData to output steam
My Version
byte audioData[];
if(isWord) {
audioData = Arrays.copyOf(byteSoundFile, pos);
isWord = false;
}
else {
audioData = new byte[pos*2];
for(int i=0; i<pos; i++) {
audioData[(i<<1)] = (byte) (shortSoundFile[i] >>> 8);
audioData[(i<<1)+1] = (byte) (shortSoundFile[i]);
}
}
pos = 0;
... code for writing audioData to output steam
But after the modification has done, the output has become worse. There is a lot of noise in the output.
Here is a sample audio with modification: modified output
Here is a sample audio from the original version: original output
Now I'd appreciate it if anyone can point out the reason that generates the noise and how to remove it. Am I doing anything wrong in the code? I have tested my algorithm in Mablab and it worked fine.

The problem has been solved temporarily. It turns out that the conversion between byte array and short array is not necessary. The required signal processing operations can be performed directly on byte arrays.
I'd like to keep this question open in case someone finds the bug(s) in the given code.

Related

Find the start of a cache line for a Java byte array

For a high performance blocked bloom filter, I would like to align data to cache lines. (I know it's easier to do such tricks in C, but I would like to use Java.)
I do have a solution, but I'm not sure if it's correct, or if there is a better way. My solution tries to find the start of the cache line using the following algorithm:
for each possible offset o (0..63; I assume cache line length of 64)
start a thread that reads from data[o] and writes that to data[o + 8]
in the main thread, write '1' to data[o], and wait until that ends up in data[o + 8] (so wait for the other thread)
repeat that
Then, measure how fast this was, basically how many increments for a loop of 1 million (in each thread). My logic is, it is slower if the data is in a different cache line.
Here my code:
public static void main(String... args) {
for(int i=0; i<20; i++) {
int size = (int) (1000 + Math.random() * 1000);
byte[] data = new byte[size];
int cacheLineOffset = getCacheLineOffset(data);
System.out.println("offset: " + cacheLineOffset);
}
}
private static int getCacheLineOffset(byte[] data) {
for (int i = 0; i < 10; i++) {
int x = tryGetCacheLineOffset(data, i + 3);
if (x != -1) {
return x;
}
}
System.out.println("Cache line start not found");
return 0;
}
private static int tryGetCacheLineOffset(byte[] data, int testCount) {
// assume synchronization between two threads is faster(?)
// if each thread works on the same cache line
int[] counters = new int[64];
int testOffset = 8;
for (int test = 0; test < testCount; test++) {
for (int offset = 0; offset < 64; offset++) {
final int o = offset;
final Semaphore sema = new Semaphore(0);
Thread t = new Thread() {
public void run() {
try {
sema.acquire();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
for (int i = 0; i < 1000000; i++) {
data[o + testOffset] = data[o];
}
}
};
t.start();
sema.release();
data[o] = 1;
int counter = 0;
byte waitfor = 1;
for (int i = 0; i < 1000000; i++) {
byte x = data[o + testOffset];
if (x == waitfor) {
data[o]++;
counter++;
waitfor++;
}
}
try {
t.join();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
counters[offset] += counter;
}
}
Arrays.fill(data, 0, testOffset + 64, (byte) 0);
int low = Integer.MAX_VALUE, high = Integer.MIN_VALUE;
for (int i = 0; i < 64; i++) {
// average of 3
int avg3 = (counters[(i - 1 + 64) % 64] + counters[i] + counters[(i + 1) % 64]) / 3;
low = Math.min(low, avg3);
high = Math.max(high, avg3);
}
if (low * 1.1 > high) {
// no significant difference between low and high
return -1;
}
int lowCount = 0;
boolean[] isLow = new boolean[64];
for (int i = 0; i < 64; i++) {
if (counters[i] < (low + high) / 2) {
isLow[i] = true;
lowCount++;
}
}
if (lowCount != 8) {
// unclear
return -1;
}
for (int i = 0; i < 64; i++) {
if (isLow[(i - 1 + 64) % 64] && !isLow[i]) {
return i;
}
}
return -1;
}
It prints (example):
offset: 16
offset: 24
offset: 0
offset: 40
offset: 40
offset: 8
offset: 24
offset: 40
...
So arrays in Java seems to be aligned to 8 bytes.
You know that the GC can move objects... so your perfectly aligned array may get misaligned later.
I'd try ByteBuffer; I guess, a direct one gets aligned a lot (to a page boundary).
Unsafe can give you the address and with JNI, you can get an array pinned.
First things first - everything in java is 8 bytes aligned, not arrays only. There's a tool for that Java Object Layout, that you can play with. Small-ish thing here (unrelated, but related) - in java-9 String(s) internally are stored as byte[] to shrink their space for LATIN-1 ones, because everything is 8-bytes aligned there was an addition of a field coder (byte) without making any instance of the string bigger - there was a gap big enough to fit that byte.
Your entire idea that objects that are aligned will be faster to access is right. That is much much more visible when multiple threads try to access that data, also known as false-sharing (but I bet you knew that). Btw here, there are methods in Unsafe that will show you object addresses, but since GC can move these around, this becomes useless for your requirement.
You would not be the first one that tries to overcome this. Unfortunately if you read that blog entry - you will see that even very experienced developers (which I admire) fail at this. A VM is notoriously smart to remove checks and code that you might think is needed somewhere, especially when JIT C2 kicks in.
What you are really looking for is:
jdk.internal.vm.annotation.Contended
annotation. This is the only way that will guarantee cache line alignment. If you really want to read about all other "tricks" that could be done, than Alekesy Shipilev's examples are the ones you are looking for.

Android - Detect differences between original .wav file and filtered .wav file

I have an original .wav file of frequency of 18kHz and 19kHz on the left and right channel.
I have another filtered .wav file that is filtering 18kHz - 20kHz using IIR Filter with bandpass filter.
How do I detect the differences between the two? As in, how do I check that the the audio being implemented with the filter is being filtered successfully?
I am using the library that I found https://github.com/ddf/Minim/blob/master/src/ddf/minim/effects/BandPass.java https://github.com/DASAR/Minim-Android/blob/master/src/ddf/minim/effects/IIRFilter.java
The following is the code relevant to the filtering.
float[][] deinterleaveData(float[] samples, int numChannels) {
// assert(samples.length() % numChannels == 0);
int numFrames = samples.length / numChannels;
float[][] result = new float[numChannels][];
for (int ch = 0; ch < numChannels; ch++) {
result[ch] = new float[numFrames];
for (int i = 0; i < numFrames; i++) {
result[ch][i] = samples[numChannels * i + ch];
}
}
return result;
}
float[] interleaveData(float[][] data) {
int numChannels = data.length;
int numFrames = data[0].length;
float[] result = new float[numFrames*numChannels];
for (int i = 0; i < numFrames; i++) {
for (int ch = 0; ch < numChannels; ch++) {
result[numChannels * i + ch] = data[ch][i];
}
}
return result;
}
/**
* Convert byte[] raw audio to 16 bit int format.
* #param rawdata
*/
private int[] byteToShort(byte[] rawdata) {
int[] converted = new int[rawdata.length / 2];
for (int i = 0; i < converted.length; i++) {
// Wave file data are stored in little-endian order
int lo = rawdata[2*i];
int hi = rawdata[2*i+1];
converted[i] = ((hi&0xFF)<<8) | (lo&0xFF);
}
return converted;
}
private float[] byteToFloat(byte[] audio) {
return shortToFloat(byteToShort(audio));
}
/**
* Convert int[] audio to 32 bit float format.
* From [-32768,32768] to [-1,1]
* #param audio
*/
private float[] shortToFloat(int[] audio) {
float[] converted = new float[audio.length];
for (int i = 0; i < converted.length; i++) {
// [-32768,32768] -> [-1,1]
converted[i] = audio[i] / 32768f; /* default range for Android PCM audio buffers) */
}
return converted;
}
private void writeAudioDataToFile() throws IOException {
int read = 0;
byte data[] = new byte[bufferSize];
String filename = getTempFilename();
FileOutputStream os = null;
FileOutputStream rs = null;
try {
os = new FileOutputStream(filename);
rs = new FileOutputStream(getFilename().split(".wav")[0] + ".txt");
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if (null != os) {
BandPass bandpass = new BandPass(19000,2000,44100);
while (isRecording) {
// decode and deinterleave stereo 16-bit per sample data
float[][] signals = deinterleaveData(byteToFloat(data), 2);
// filter data samples, updating the buffers with the filtered samples.
bandpass.process(signals[0], signals[1]);
// recombine signals for playback
audioTrack.write(interleaveData(signals), 0, count, WRITE_NON_BLOCKING);
// audioTrack.write(data, 0, count);
read = recorder.read(data, 0, bufferSize);
if (AudioRecord.ERROR_INVALID_OPERATION != read) {
try {
os.write(data);
rs.write(data);
} catch (IOException e) {
e.printStackTrace();
}
}
}
try {
os.close();
rs.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Full code is here if necessary: http://pastebin.com/23aS2A2w
Do I find the peaks and valleys of the original .wav file and the filtered .wav file to detect the difference? If not, how would I detect?
Thanks for all responses and help. Appreciated it!
There are a couple of techniques you could use for measuring the frequency response of a filter.
The first would be to run the filter with pure sine waves at various frequencies and then measure the amplitude of the output. For example, generate a tone at 1kHz with a peak amplitude of 1.0 and run the filter. Then look at the output. If the output is half the amplitude then you can say the filter has 6dB of attenuation at 1kHz. If you do this for enough frequencies you can make a plot of the frequency response with frequency on the x axis and output level on the y axis. This is a pretty crude approach though and requires a lot of points to get fine detail. Also, after changing frequencies you would need to skip some of the output to avoid looking at transients.
Another approach is to measure the impulse response of the filter. You do this by inputing a signal which has a single 1.0 followed by the rest zeros. Taking an FFT of the impulse response will give you the frequency response. You need to be careful though to supply enough samples to the filter to allow the impulse to die out though. You can figure this out by looking at the output samples and determining when the samples decay down to zero.

Reading and writing huge files in java

My idea is to make a little software that reads a file (which can't be read "naturally", but it contains some images), turns its data into hex, looks for the PNG chunks (a kind of marks that are at the beginning and end of a .png file), and saves the resulting data in different files (after getting it back from hex). I am doing this in Java, using a code like this:
// out is where to show the result and file is the source
public static void hexDump(PrintStream out, File file) throws IOException {
InputStream is = new FileInputStream(file);
StringBuffer Buffer = new StringBuffer();
while (is.available() > 0) {
StringBuilder sb1 = new StringBuilder();
for (int j = 0; j < 16; j++) {
if (is.available() > 0) {
int value = (int) is.read();
// transform the current data into hex
sb1.append(String.format("%02X ", value));
}
}
Buffer.append(sb1);
// Should I look for the PNG here? I'm not sure
}
is.close();
// Print the result in out (that may be the console or a file)
out.print(Buffer);
}
I'm sure there are another ways to do this using less "machine-resources" while opening huge files. If you have any idea, please tell me. Thanks!
This is the first time I post, so if there is any error, please help me to correct it.
As Erwin Bolwidt says in the comments, first thing is don't convert to hex. If for some reason you must convert to hex, quit appending the content to two buffers, and always use StringBuilder, not StringBuffer. StringBuilder can be as much as 3x faster than StringBuffer.
Also, buffer your file reads with BufferedReader. Reading one character at a time with FileInputStream.read() is very slow.
A very simple way to do this, which is probably quite fast, is to read the entire file into memory (as binary data, not as a hex dump) and then search for the markers.
This has two limitations:
it only handles files up to 2 GiB in length (max size of Java arrays)
it requires large chunks of memory - it is possible to optimize this by reader smaller chunks but that makes the algorithm more complex
The basic code to do that is like this:
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
public class Png {
static final String PNG_MARKER_HEX = "abcdef0123456789"; // TODO: replace with real marker
static final byte[] PNG_MARKER = hexStringToByteArray(PNG_MARKER_HEX);
public void splitPngChunks(File file) throws IOException {
byte[] bytes = Files.readAllBytes(file.toPath());
int offset = KMPMatch.indexOf(bytes, 0, PNG_MARKER);
while (offset >= 0) {
int nextOffset = KMPMatch.indexOf(bytes, 0, PNG_MARKER);
if (nextOffset < 0) {
writePngChunk(bytes, offset, bytes.length - offset);
} else {
writePngChunk(bytes, offset, nextOffset - offset);
}
offset = nextOffset;
}
}
public void writePngChunk(byte[] bytes, int offset, int length) {
// TODO: implement - where do you want to write the chunks?
}
}
I'm not sure how these PNG chunk markers work exactly, I'm assuming above that they start the section of the data that you're interested in, and that the next marker starts the next section of the data.
There are two things missing in standard Java: code to convert a hex string to a byte array and code to search for a byte array inside another byte array.
Both can be found in various apache-commons libraries but I'll include that answers the people posted to earlier questions on StackOverflow. You can copy these verbatim into the Png class to make the above code work.
Convert a string representation of a hex dump to a byte array using Java?
public static byte[] hexStringToByteArray(String s) {
int len = s.length();
byte[] data = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
data[i / 2] = (byte) ((Character.digit(s.charAt(i), 16) << 4) + Character.digit(s.charAt(i + 1), 16));
}
return data;
}
Searching for a sequence of Bytes in a Binary File with Java
/**
* Knuth-Morris-Pratt Algorithm for Pattern Matching
*/
static class KMPMatch {
/**
* Finds the first occurrence of the pattern in the text.
*/
public static int indexOf(byte[] data, int offset, byte[] pattern) {
int[] failure = computeFailure(pattern);
int j = 0;
if (data.length - offset <= 0)
return -1;
for (int i = offset; i < data.length; i++) {
while (j > 0 && pattern[j] != data[i]) {
j = failure[j - 1];
}
if (pattern[j] == data[i]) {
j++;
}
if (j == pattern.length) {
return i - pattern.length + 1;
}
}
return -1;
}
/**
* Computes the failure function using a boot-strapping process, where the pattern is matched against itself.
*/
private static int[] computeFailure(byte[] pattern) {
int[] failure = new int[pattern.length];
int j = 0;
for (int i = 1; i < pattern.length; i++) {
while (j > 0 && pattern[j] != pattern[i]) {
j = failure[j - 1];
}
if (pattern[j] == pattern[i]) {
j++;
}
failure[i] = j;
}
return failure;
}
}
I modified this last piece of code to make it possible to start the search at an offset other than zero.
Reading the file a byte at a time would be taking substantial time here. You can improve that by orders of magnitude. You should be using a DataInputStream around a BufferedInputStream around the FileInputStream, and reading 16 bytes at a time with readFully.
And then processing them, without conversion to and from hex, which is quite unnecessary here, and writing them to the output(s) as you go, via a BufferedOutputStream around the FileOutputStream, rather than concatenating the entire file into memory and having to write it all out in one go. Of course that takes time, but that's because it does, not because you have to do it that way.

Calculate FFT from audio file

Before, I asked question about Get frequency wav audio using FFT and Complex class ,
There, I need to calculate FFT value from AudioRecord input --> from microphone , I somehow managed to get the FFT value...
Now I need to calculate FFT value from *.wav audio file that I saved before,
I saved the audio in 'raw' folder inside 'res' folder from my project
I still using the same FFT Class: http://www.cs.princeton.edu/introcs/97data/FFT.java
The complex class to go with it: http://introcs.cs.princeton.edu/java/97data/Complex.java.html
I use this method to read audio file from my raw foldern, then I call method calculateFFT to go with it
private static final int RECORDER_BPP = 16;
private static final int RECORDER_SAMPLERATE = 44100;
private static final int RECORDER_CHANNELS = AudioFormat.CHANNEL_IN_STEREO;
private static final int RECORDER_AUDIO_ENCODING = AudioFormat.ENCODING_PCM_16BIT;
private void asli(){
int counter = 0;
int data;
InputStream inputStream = getResources().openRawResource(R.raw.b1);
DataInputStream dataInputStream = new DataInputStream(inputStream);
List<Integer> content = new ArrayList<Integer>();
try {
while ((data = dataInputStream.read()) != -1) {
content.add(data);
counter++; }
} catch (IOException e) {
e.printStackTrace();}
int[] b = new int[content.size()];
int cont = 0;
byte[] audio = convertArray(b);
}
Method to convert to byte
public byte[] convertArray(int[] array) {
int minBufferSize = AudioTrack.getMinBufferSize(RECORDER_SAMPLERATE,RECORDER_CHANNELS,RECORDER_AUDIO_ENCODING);
AudioTrack audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,RECORDER_SAMPLERATE,RECORDER_CHANNELS,RECORDER_AUDIO_ENCODING,minBufferSize, AudioTrack.MODE_STREAM);
byte[] newarray = new byte[array.length];
for (int i = 0; i < array.length; i++) {
newarray[i] = (byte) ((array[i]) & 0xFF); }
absNormalizedSignal = calculateFFT(newarray);
return newarray;
}
And this is the CalculateFFT method
public double[] calculateFFT(byte[] signal)
{
final int mNumberOfFFTPoints =1024;
double mMaxFFTSample;
double temp;
Complex[] y;
Complex[] complexSignal = new Complex[mNumberOfFFTPoints];
double[] absSignal = new double[mNumberOfFFTPoints/2];
for(int i = 0; i < mNumberOfFFTPoints; i++){
temp = (double)((signal[2*i] & 0xFF) | (signal[2*i+1] << 8)) / 32768.0F;
complexSignal[i] = new Complex(temp,0.0);
}
y = FFT.fft(complexSignal);
mMaxFFTSample = 0.0;
mPeakPos = 0;
for(int i = 0; i < (mNumberOfFFTPoints/2); i++)
{
absSignal[i] = Math.sqrt(Math.pow(y[i].re(), 2) + Math.pow(y[i].im(), 2));
if(absSignal[i] > mMaxFFTSample)
{
mMaxFFTSample = absSignal[i];
mPeakPos = i;
}
}
return absSignal;
}
I used this CalculateFFT method too to process audio from AudioRecorder --> that one with microphone input before... I managed to get value from the AudioRecorder, but I failed to get value from my audio file... I'm not planning to play the audio.. I just need to process it with FFT.
Is there any wrong with my code ?? :o Seems like I fail at getting value from method Asli(); But I dont know which part is wrong..
Any help would be appreciated... :)
Thanks
I spent a better part of the morning coding a solution for this using bits and pieces of FFT java snippets I was finding... but then I stumbled upon this amazingly wondeful google code project that has a bunch of util classes for doing signal processing tasks on WAV and MP3 files alike.
https://github.com/Uriopass/audio-analysis
Formerly SVN export was on Google code here: https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/audio-analysis/source-archive.zip
It now becomes INSANELY easy:
WaveDecoder decoder = new WaveDecoder(new FileInputStream(wavFile));
FFT fft = new FFT(1024, wavFileObj.getSampleRate());
Now you can use the fft object to do various calculations. They have a bunch of great examples, such as generating a List containing the spectral flux:
float[] samples = new float[1024];
float[] spectrum = new float[1024 / 2 + 1];
float[] lastSpectrum = new float[1024 / 2 + 1];
List<Float> spectralFlux = new ArrayList<Float>();
while (decoder.readSamples(samples) > 0) {
fft.forward(samples);
System.arraycopy(spectrum, 0, lastSpectrum, 0, spectrum.length);
System.arraycopy(fft.getSpectrum(), 0, spectrum, 0, spectrum.length);
float flux = 0;
for (int i = 0; i < spectrum.length; i++)
flux += (spectrum[i] - lastSpectrum[i]);
spectralFlux.add(flux);
}
My company needed a way for me to analyze some audio to see if some expected hold music had happened. So first I generated a WAV file for an example that did have the hold music. Then I captured some audio of one of thee examples that did not have the hold music. Now all that is left is to average up the spectral flux of the wav and I'm set.
Note: I could not have simply taken amplitudes... but the fourier transformation has frequencies that I could correctly use to make my comparison.
I love math.

Interleaved parallel file read slower than sequential read?

I have implemented a small IO class, which can read from multiple and same files on different disks (e.g two hard disks containing the same file). In sequential case, both disks read 60MB/s in average over the file, but when I do an interleaved (e.g. 4k disk 1, 4k disk 2 then combine), the effective read speed is reduced to 40MB/s instead of increasing?
Context: Win 7 + JDK 7b70, 2GB RAM, 2.2GB test file. Basically, I try to mimic Win7's ReadyBoost and RAID x in a poor man's fashion.
In the heart, when a read() is issued to the class, it creates two runnables with instructions to read a pre-opened RandomAccessFile from a certain position and length. Using an executor service and Future.get() calls, when both finish, the data read gets copied into a common buffer and returned to the caller.
Is there a conceptional error in my approach? (For example, the OS caching mechanism will always counteract?)
protected <T> List<T> waitForAll(List<Future<T>> futures)
throws MultiIOException {
MultiIOException mex = null;
int i = 0;
List<T> result = new ArrayList<T>(futures.size());
for (Future<T> f : futures) {
try {
result.add(f.get());
} catch (InterruptedException ex) {
if (mex == null) {
mex = new MultiIOException();
}
mex.exceptions.add(new ExceptionPair(metrics[i].file, ex));
} catch (ExecutionException ex) {
if (mex == null) {
mex = new MultiIOException();
}
mex.exceptions.add(new ExceptionPair(metrics[i].file, ex));
}
i++;
}
if (mex != null) {
throw mex;
}
return result;
}
public int read(long position, byte[] output, int start, int length)
throws IOException {
if (start < 0 || start + length > output.length) {
throw new IndexOutOfBoundsException(
String.format("start=%d, length=%d, output=%d",
start, length, output.length));
}
// compute the fragment sizes and positions
int result = 0;
final long[] positions = new long[metrics.length];
final int[] lengths = new int[metrics.length];
double speedSum = 0.0;
double maxValue = 0.0;
int maxIndex = 0;
for (int i = 0; i < metrics.length; i++) {
speedSum += metrics[i].readSpeed;
if (metrics[i].readSpeed > maxValue) {
maxValue = metrics[i].readSpeed;
maxIndex = i;
}
}
// adjust read lengths
int lengthSum = length;
for (int i = 0; i < metrics.length; i++) {
int len = (int)Math.ceil(length * metrics[i].readSpeed / speedSum);
lengths[i] = (len > lengthSum) ? lengthSum : len;
lengthSum -= lengths[i];
}
if (lengthSum > 0) {
lengths[maxIndex] += lengthSum;
}
// adjust read positions
long positionDelta = position;
for (int i = 0; i < metrics.length; i++) {
positions[i] = positionDelta;
positionDelta += (long)lengths[i];
}
List<Future<byte[]>> futures = new LinkedList<Future<byte[]>>();
// read in parallel
for (int i = 0; i < metrics.length; i++) {
final int j = i;
futures.add(exec.submit(new Callable<byte[]>() {
#Override
public byte[] call() throws Exception {
byte[] buffer = new byte[lengths[j]];
long t = System.nanoTime();
long t0 = t;
long currPos = metrics[j].handle.getFilePointer();
metrics[j].handle.seek(positions[j]);
t = System.nanoTime() - t;
metrics[j].seekTime = t * 1024.0 * 1024.0 /
Math.abs(currPos - positions[j]) / 1E9 ;
int c = metrics[j].handle.read(buffer);
t0 = System.nanoTime() - t0;
// adjust the read speed if we read something
if (c > 0) {
metrics[j].readSpeed = (alpha * c * 1E9 / t0 / 1024 / 1024
+ (1 - alpha) * metrics[j].readSpeed) ;
}
if (c < 0) {
return null;
} else
if (c == 0) {
return EMPTY_BYTE_ARRAY;
} else
if (c < buffer.length) {
return Arrays.copyOf(buffer, c);
}
return buffer;
}
}));
}
List<byte[]> data = waitForAll(futures);
boolean eof = true;
for (byte[] b : data) {
if (b != null && b.length > 0) {
System.arraycopy(b, 0, output, start + result, b.length);
result += b.length;
eof = false;
} else {
break; // the rest probably reached EOF
}
}
// if there was no data at all, we reached the end of file
if (eof) {
return -1;
}
sequentialPosition = position + (long)result;
// evaluate the fastest file to read
double maxSpeed = 0;
maxIndex = 0;
for (int i = 0; i < metrics.length; i++) {
if (metrics[i].readSpeed > maxSpeed) {
maxSpeed = metrics[i].readSpeed;
maxIndex = i;
}
}
fastest = metrics[maxIndex];
return result;
}
(FileMetrics in metrics array contain measurements of read speed to adaptively determine the buffer sizes of various input channels - in my test with alpha = 0 and readSpeed = 1 results equal distribution)
Edit
I ran an non-entangled test (e.g read the two files independently in separate threads.) and I've got a combined effective speed of 110MB/s.
Edit2
I guess I know why is this happening.
When I read in parallel and in sequence, it is not a sequential read for the disks, but rather read-skip-read-skip pattern due the interleaving (and possibly riddled with allocation table lookups). This basically reduces the effective read speed per disk to half or worse.
As you said, a sequential read on a disk is much faster than a read-skip-read-skip pattern. Hard disks are capable of high bandwidth when reading sequentially, but the seek time (latency) is expensive.
Instead of storing a copy of the file in each disk, try storing block i of the file on disk i (mod 2). This way you can read from both disks sequentially and recombine the result in memory.
If you want to do a parallel read, break the read into two sequential reads. Find the halfway point and read the first half from the first file and the second half from the second file.
If you are sure that you performing no more than one read per disk (otherwise you will have many disk misses), you still create contention on other parts in the computer - the bus, the raid controller (if exists) and so on.
Maybe http://stxxl.sourceforge.net/ might be of any interest for you, too.

Categories