We Keep Coding

Sending large data over TCP/IP socket

I have a small project running a server in C# and a client in Java. The server sends images to the client.
Some images are quite big (up to 10MiB sometimes), so I split the image bytes and send it in chunks of 32768 bytes each.
My C# Server code is as follows:
using (var stream = new MemoryStream(ImageData))
{
for (int j = 1; j <= dataSplitParameters.NumberOfChunks; j++)
{
byte[] chunk;
if (j == dataSplitParameters.NumberOfChunks)
chunk = new byte[dataSplitParameters.FinalChunkSize];
else
chunk = new byte[dataSplitParameters.ChunkSize];
int result = stream.Read(chunk, 0, chunk.Length);
string line = DateTime.Now + ", Status OK, " + ImageName+ ", ImageChunk, " + j + ", " + dataSplitParameters.NumberOfChunks + ", " + chunk.Length;
//write read params
streamWriter.WriteLine(line);
streamWriter.Flush();
//write the data
binaryWriter.Write(chunk);
binaryWriter.Flush();
Console.WriteLine(line);
string deliveryReport = streamReader.ReadLine();
Console.WriteLine(deliveryReport);
}
}
And my Java Client code is as follows:
long dataRead = 0;
for (int j = 1; j <= numberOfChunks; j++) {
String line = bufferedReader.readLine();
tokens = line.split(", ");
System.out.println(line);
int toRead = Integer.parseInt(tokens[tokens.length - 1]);
byte[] chunk = new byte[toRead];
int read = inputStream.read(chunk, 0, toRead);
//do something with the data
dataRead += read;
String progressReport = pageLabel + ", progress: " + dataRead + "/" + dataLength + " bytes.";
bufferedOutputStream.write((progressReport + "\n").getBytes());
bufferedOutputStream.flush();
System.out.println(progressReport);
}
The problem is when I run the code, either the client crashes with an error saying it is reading bogus data, or both the client and the server hang. This is the error:
Document Page 1, progress: 49153/226604 bytes.
�9��%>�YI!��F�����h�
Exception in thread "main" java.lang.NumberFormatException: For input string: .....
What am I doing wrong?

The basic problem.
Once you wrap an inputstream into a bufferedreader you must stop accessing the inputstream. That bufferedreader is buffered, it will read as much data as it wants to, it is NOT limited to reading exactly up to the next newline symbol(s) and stopping there.
The BufferedReader on the java side has read a lot more than that, so it's consumed a whole bunch of image data already, and there's no way out from here. By making that BufferedReader, you've made the job impossible, so you can't do that.
The underlying problem.
You have a single TCP/IP connection. On this, you send some irrelevant text (the page, the progress, etc), and then you send an unknown amount of image data, and then you send another irrelevant progress update.
That's fundamentally broken. How can an image parser possibly know that halfway through sending an image, you get a status update line? Text is just binary data too, there is no magic identifier that lets a client know: This byte is part of the image data, but this byte is some text sent in-between with progress info.
The simple fix.
You'd think the simple fix is.. well, stop doing that then! Why are you sending this progress? The client is perfectly capable of knowing how many bytes it read, there is no point sending that. Just.. take your binary data. open the outputstream. send all that data. And on the client side, open the inputstream, read all that data. Don't involve strings. Don't use anything that smacks of 'works with characters' (so, BufferedReader? No. BufferedInputStream is fine).
... but now the client doesn't know the title, nor the total size!
So make a wire protocol. It can be near trivial.
This is your wire protocol:
4 bytes, big endian: SizeOfName
SizeOfName number of bytes. UTF-8 encoded document title.
4 bytes, big endian: SizeOfData
SizeOfData number of bytes. The image data.
And that's if you actually want the client to be able to render a progress bar and to know the title. If that's not needed, don't do any of that, just straight up send the bytes, and signal that the file has been completely sent by.. closing the connection.
Here's some sample java code:
try (InputStream in = ....) {
int nameSize = readInt(in);
byte[] nameBytes = in.readNBytes(nameSize);
String name = new String(nameBytes, StandardCharsets.UTF_8);
int dataSize = readInt(in);
try (OutputStream out =
Files.newOutputStream(Paths.get("/Users/TriSky/image.png")) {
byte[] buffer = new byte[65536];
while (dataSize > 0) {
int r = in.read(buffer);
if (r == -1) throw new IOException("Early end-of-stream");
out.write(buffer, 0, r);
dataSize -= r;
}
}
}
public int readInt(InputStream in) throws IOException {
byte[] b = in.readNBytes(4);
return ByteBuffer.wrap(b).getInt();
}
Closing notes
Another bug in your app is that you're using the wrong method. Java's 'read(bytes)' method will NOT (neccessarily) fully fill that byte array. All read(byte[]) will do is read at least 1 byte (unless the stream is closed, then it reads none, and returns -1. The idea is: read will read the optimal number of bytes: Exactly as many as are ready to give you right now. How many is that? Who knows - if you ignore the returned value of in.read(bytes), your code is neccessarily broken, and you're doing just that. What you really want is for example readNBytes which guarantees that it fully fills that byte array (or until stream ends, whichever happens first).
Note that in the transfer code above, I also use the basic read, but here I don't ignore the return value.

Your Java code seems to be using a BufferedReader. It reads data into a buffer of its own, meaning it is no longer available in the underlying socket input stream - that's your first problem. You have a second problem with how inputStream.read is used - it's not guaranteed to read all the bytes you ask for, you would have to put a loop around it.
This is not a particularly easy problem to solve. When you mix binary and text data in the same stream, it is difficult to read it back. In Java, there is a class called DataInputStream that can help a little - it has a readLine method to read a line of text, and also methods to read binary data:
DataInputStream dataInput = new DataInputStream(inputStream);
for (int j = 1; j <= numberOfChunks; j++) {
String line = dataInput.readLine();
...
byte[] chunk = new byte[toRead];
int read = dataInput.readFully(chunk);
...
}
DataInputStream has limitations: the readLine method is deprecated because it assumes the text is encoded in latin-1, and does not let you use a different text encoding. If you want to go further down this road you'll want to create a class of your own to read your stream format.
Some images are quite big (up to 10MiB sometimes), so I split the image bytes and send it in chunks of 32768 bytes each.
You know this is totally unnecessary right? There is absolutely no problem sending multiple megabytes of data into a TCP socket, and streaming all of the data in on the receiving side.

When you try to send image, you have to open the image as a normal file then substring the image into some chunks and every chunk change it into "base64encode" when you send and the client decode it because the image data is not normal data, so base64encode change this symbols to normal chars like AfHM65Hkgf7MM

How do I convert data read from a WAV file to an array of signed 16-bit raw audio data in java?

I have no idea how to do this. I have read the answers to several similar questions and some websites that probably had the answer somewhere, but either I could not understand them or they were not what I am trying to do. It is also possible that some did have the answer, but I could not focus well enough to interpret it. I want a method that converts the data from a WAV file signed 16-bit raw audio data and puts this into a short[]. I would prefer short minimalistic easy to understand answers because I would have less difficulty focusing on those.
Edit: Some have said this might be a duplicate of stackoverflow.com/questions/5210147/reading-wav-file-in-java. I do not understand that question or its answers well enough to even say whether it is different or why or how to change my question so it is not confused for that one.
Another edit: I have attempted using Phil Freihofner's answer, but when testing this by attempting to pay back the audio, I just heard a lot of clicks. I am not sure if I implemented it correctly. Here is the method that reads the file:
static void loadAudioDataTest(String filepath){
int totalFramesRead = 0;
File fileIn = new File(filepath);
try {
AudioInputStream audioInputStream =
AudioSystem.getAudioInputStream(fileIn);
int bytesPerFrame =
audioInputStream.getFormat().getFrameSize();
if (bytesPerFrame == AudioSystem.NOT_SPECIFIED) {
bytesPerFrame = 1;
}
int numBytes = 1024 * bytesPerFrame;
byte[] audioBytes = new byte[numBytes];
audioArray=new short[numBytes/2];
try{
int numBytesRead = 0;
int numFramesRead = 0;
while ((numBytesRead =
audioInputStream.read(audioBytes)) != -1) {
numFramesRead = numBytesRead / bytesPerFrame;
totalFramesRead += numFramesRead;
}for(int a=0;a<audioArray.length;a++){
audioArray[acc]=(short)((audioBytes[a*2]&0xff)|(audioBytes[acc*2+1]<<8));
}
} catch (Exception ex) {
// Handle the error...
}
} catch (Exception e) {
// Handle the error...
}
}
This bit plays the sound and is inside an actionPerformed(ActionEvent) void that is repeatedly activated by a timer, in case the issue is there
byte[]buf=new byte[2];
AudioFormat af=new AudioFormat(44100,16,1,true,false);
SourceDataLine sdl;
try{
sdl=AudioSystem.getSourceDataLine(af);
sdl.open();
sdl.start();
buf[1]=(byte) (audioArray[t%audioArray.length]&0xFF);
buf[0]=(byte) (audioArray[t%audioArray.length]>>8);
sdl.write(buf,0,2);
sdl.drain();
sdl.stop();
}catch(LineUnavailableException e1){
e1.printStackTrace();
}t++;

The current core java class commonly used for loading data into a byte array is AudioInputStream (javax.sound.sampled.AudioInputStream). An example of its use, with explanation, can be found in the Oracle tutorial Using Files and Format Converters. The sample code is in the section titled "Reading Sound Files". Note the point in the innermost while loop with the following line: // Here, do something useful with the audio data. At that point, you would load the data into your array.
Taking two bytes and converting them to a short has been answered several times but I don't have the links handy. It's easier to just post some code I have used.
audioArray[i] = ( buffer[bufferIdx] & 0xff )
| ( buffer[bufferIdx + 1] << 8 ) ;
... where audioArray could be a short[]. (In my code I use float[] and do another step to scale the values to range from -1 to 1.)
This is a slightly modified snipped from the library AudioCue on github, quoting from lines 391-393.

EntityMetadata packet issue

I am attempting to make a player appear like they are sneaking (crouching) on Minecraft 1.8.8 running Spigot, based on http://wiki.vg/Entities#Entity_Metadata_Format I have done the following:
Created a data watcher and mapped appropriate value for crouched from the wiki:
DataWatcher dw = new DataWatcher(null);
dw.a(0, (byte) 0x02);
Created the packet, where target is a Player object of the player that needs to appear sneaking:
PacketPlayOutEntityMetadata metadataPacket = new PacketPlayOutEntityMetadata(target.getEntityId(), dw, false);
Sent the packet to everyone online:
for (Player p : Bukkit.getOnlinePlayers()) {
((CraftPlayer) p).getHandle().playerConnection.sendPacket(metadataPacket);
}
This does not appear to be working though, how would be the appropriate way to go about this?
I attempted to use ProtocolLib too, though ideally I am looking for a solution that works using packets.

The problem is that you use the wrong method for updating. There is a internal boolean in the datawatcher that checks for updates. There are 2 ways solving this problem.
Using DataWatcher#watch:
Player target = Bukkit.getPlayer("RandomGuy");
DataWatcher dw = ((CraftPlayer) target).getHandle().getDataWatcher();
dw.watch(0, (byte) 2);
PacketPlayOutEntityMetadata metadataPacket = new PacketPlayOutEntityMetadata(target.getEntityId(), dw, false);
//sending packet...
Skipping the internal boolean (not recommended):
Player target = Bukkit.getPlayer("RandomGuy");
DataWatcher dw = ((CraftPlayer) target).getHandle().getDataWatcher();
dw.a(0, (byte) 2);
PacketPlayOutEntityMetadata metadataPacket = new PacketPlayOutEntityMetadata(target.getEntityId(), dw, true);
//sending packet...
P.S. If that is a fake entity, I'd recommend instantiating a reference of an EntityPlayer for better packet control.

Create OBD2 MUT scanner by using Java FTDI lib - jd2xx - openport1.3 usb cable

I am trying to create a Java obd2 scanner app for my mitsubishi lancer mx 1997 which use MUTII protocol over OBD2. Can anybody help me to read MUT request codes using jd2xx library.
I have tried the below program, but it didn read engine RPM.
package lancerscan;
import jd2xx.JD2XX;
public class Test2 {
public static void main(String[] args) throws Exception {
JD2XX jd = new JD2XX();
jd.open(0);
jd.setBaudRate(38400);
jd.setDataCharacteristics(
8, JD2XX.STOP_BITS_1, JD2XX.PARITY_NONE);
jd.setFlowControl(
JD2XX.FLOW_NONE, 0, 0);
jd.setTimeouts(1000, 1000);
String msg = "21";
int ret = jd.write(msg.getBytes());
System.out.println(ret + " bytes sent.");
int rd = jd.read();
System.out.println(">>>" + rd);
int status = jd.getQueueStatus();
byte[] data = new byte[(int) status];
long lngBytesReturned = jd.read(data, 0, data.length);
System.out.println("======= " + lngBytesReturned);
}
}
MUT request code for Engine RPM is 0x21
more MUT request codes can be found here
similar C programs which works fine is here; main prjct files are here
Thanks,
harsha

First your using a different baud rate to that in the example. The example uses 15625 baud but you are using 38400 baud.
Secondly you are missing some of the setup commands. I am not sure if this will make a difference but its something that is different between your code and the example.
Mitsubishi require you to set the car ECU into diagnostic mode by sending 0x00 at a rate of 5 baud on one of the pins. On the OpenPort 1.3D cable this translates to setting the break to on for 1800 ms and then turning it off. You can see this is done with the ftdimut_init() command from the libftdimut.c file.
printf("Sending 0x00 at 5 baud\n");
printf("Break on......\n");
ftStatus = FT_SetBreakOn(ftdimut_ftHandle);
if(ftStatus != FT_OK) return ftStatus;
ftdimut_msleep(1800);
printf("Break off......\n");
ftStatus = FT_SetBreakOff(ftdimut_ftHandle);
if(ftStatus != FT_OK) return ftStatus;
The car ECU will then send you 4 bytes containing the ECU ID. This can then be used to check the ECU correctly entered diagnostic mode. You can see this in libftdimut.c.
ftStatus = FT_Read(ftdimut_ftHandle, buf, 4, &bytesRead);
if(ftStatus != FT_OK) return ftStatus;
if(bytesRead == 4) {
return FT_OK;
}
Now assuming that you got the 4 bytes back you can start to send the diagnostic codes such as 0x17 or 0x21.

I just saw your post on my blog, niallm answer is correct, you need to do a 5 baud init first which involves driving the KLine, you can use something like the 5 baud init posted in that answer, more info about the protocol:
http://evoecu.logic.net/wiki/MUT_Protocol
After getting a 4 byte response you can start sending requests at 15625 baud (I'm communicating with a 94 3000GT so the CEL light stops blinking), also in my case I send the converted values (0x21 = 33 decimal) as a byte array.

Java - reading, manipulating and writing WAV files

In a Java program, what is the best way to read an audio file (WAV file) to an array of numbers (float[], short[], ...), and to write a WAV file from an array of numbers?

I read WAV files via an AudioInputStream. The following snippet from the Java Sound Tutorials works well.
int totalFramesRead = 0;
File fileIn = new File(somePathName);
// somePathName is a pre-existing string whose value was
// based on a user selection.
try {
AudioInputStream audioInputStream =
AudioSystem.getAudioInputStream(fileIn);
int bytesPerFrame =
audioInputStream.getFormat().getFrameSize();
if (bytesPerFrame == AudioSystem.NOT_SPECIFIED) {
// some audio formats may have unspecified frame size
// in that case we may read any amount of bytes
bytesPerFrame = 1;
}
// Set an arbitrary buffer size of 1024 frames.
int numBytes = 1024 * bytesPerFrame;
byte[] audioBytes = new byte[numBytes];
try {
int numBytesRead = 0;
int numFramesRead = 0;
// Try to read numBytes bytes from the file.
while ((numBytesRead =
audioInputStream.read(audioBytes)) != -1) {
// Calculate the number of frames actually read.
numFramesRead = numBytesRead / bytesPerFrame;
totalFramesRead += numFramesRead;
// Here, do something useful with the audio data that's
// now in the audioBytes array...
}
} catch (Exception ex) {
// Handle the error...
}
} catch (Exception e) {
// Handle the error...
}
To write a WAV, I found that quite tricky. On the surface it seems like a circular problem, the command that writes relies on an AudioInputStream as a parameter.
But how do you write bytes to an AudioInputStream? Shouldn't there be an AudioOutputStream?
What I found was that one can define an object that has access to the raw audio byte data to implement TargetDataLine.
This requires a lot of methods be implemented, but most can stay in dummy form as they are not required for writing data to a file. The key method to implement is read(byte[] buffer, int bufferoffset, int numberofbytestoread).
As this method will probably be called multiple times, there should also be an instance variable that indicates how far through the data one has progressed, and update that as part of the above read method.
When you have implemented this method, then your object can be used in to create a new AudioInputStream which in turn can be used with:
AudioSystem.write(yourAudioInputStream, AudioFileFormat.WAV, yourFileDestination)
As a reminder, an AudioInputStream can be created with a TargetDataLine as a source.
As to the direct manipulating the data, I have had good success acting on the data in the buffer in the innermost loop of the snippet example above, audioBytes.
While you are in that inner loop, you can convert the bytes to integers or floats and multiply a volume value (ranging from 0.0 to 1.0) and then convert them back to little endian bytes.
I believe since you have access to a series of samples in that buffer you can also engage various forms of DSP filtering algorithms at that stage. In my experience I have found that it is better to do volume changes directly on data in this buffer because then you can make the smallest possible increment: one delta per sample, minimizing the chance of clicks due to volume-induced discontinuities.
I find the "control lines" for volume provided by Java tend to situations where the jumps in volume will cause clicks, and I believe this is because the deltas are only implemented at the granularity of a single buffer read (often in the range of one change per 1024 samples) rather than dividing the change into smaller pieces and adding them one per sample. But I'm not privy to how the Volume Controls were implemented, so please take that conjecture with a grain of salt.
All and all, Java.Sound has been a real headache to figure out. I fault the Tutorial for not including an explicit example of writing a file directly from bytes. I fault the Tutorial for burying the best example of Play a File coding in the "How to Convert..." section. However, there's a LOT of valuable FREE info in that tutorial.
EDIT: 12/13/17
I've since used the following code to write audio from a PCM file in my own projects. Instead of implementing TargetDataLine one can extend InputStream and use that as a parameter to the AudioSystem.write method.
public class StereoPcmInputStream extends InputStream
{
private float[] dataFrames;
private int framesCounter;
private int cursor;
private int[] pcmOut = new int[2];
private int[] frameBytes = new int[4];
private int idx;
private int framesToRead;
public void setDataFrames(float[] dataFrames)
{
this.dataFrames = dataFrames;
framesToRead = dataFrames.length / 2;
}
#Override
public int read() throws IOException
{
while(available() > 0)
{
idx &= 3;
if (idx == 0) // set up next frame's worth of data
{
framesCounter++; // count elapsing frames
// scale to 16 bits
pcmOut[0] = (int)(dataFrames[cursor++] * Short.MAX_VALUE);
pcmOut[1] = (int)(dataFrames[cursor++] * Short.MAX_VALUE);
// output as unsigned bytes, in range [0..255]
frameBytes[0] = (char)pcmOut[0];
frameBytes[1] = (char)(pcmOut[0] >> 8);
frameBytes[2] = (char)pcmOut[1];
frameBytes[3] = (char)(pcmOut[1] >> 8);
}
return frameBytes[idx++];
}
return -1;
}
#Override
public int available()
{
// NOTE: not concurrency safe.
// 1st half of sum: there are 4 reads available per frame to be read
// 2nd half of sum: the # of bytes of the current frame that remain to be read
return 4 * ((framesToRead - 1) - framesCounter)
+ (4 - (idx % 4));
}
#Override
public void reset()
{
cursor = 0;
framesCounter = 0;
idx = 0;
}
#Override
public void close()
{
System.out.println(
"StereoPcmInputStream stopped after reading frames:"
+ framesCounter);
}
}
The source data to be exported here is in the form of stereo floats ranging from -1 to 1. The format of the resulting stream is 16-bit, stereo, little-endian.
I omitted skip and markSupported methods for my particular application. But it shouldn't be difficult to add them if they are needed.

This is the source code to write directly to a wav file.
You just need to know the mathematics and sound engineering to produce the sound you want.
In this example the equation calculates a binaural beat.
import javax.sound.sampled.AudioFileFormat;
import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
public class Program {
public static void main(String[] args) throws IOException {
final double sampleRate = 44100.0;
final double frequency = 440;
final double frequency2 = 90;
final double amplitude = 1.0;
final double seconds = 2.0;
final double twoPiF = 2 * Math.PI * frequency;
final double piF = Math.PI * frequency2;
float[] buffer = new float[(int)(seconds * sampleRate)];
for (int sample = 0; sample < buffer.length; sample++) {
double time = sample / sampleRate;
buffer[sample] = (float)(amplitude * Math.cos(piF * time) * Math.sin(twoPiF * time));
}
final byte[] byteBuffer = new byte[buffer.length * 2];
int bufferIndex = 0;
for (int i = 0; i < byteBuffer.length; i++) {
final int x = (int)(buffer[bufferIndex++] * 32767.0);
byteBuffer[i++] = (byte)x;
byteBuffer[i] = (byte)(x >>> 8);
}
File out = new File("out10.wav");
final boolean bigEndian = false;
final boolean signed = true;
final int bits = 16;
final int channels = 1;
AudioFormat format = new AudioFormat((float)sampleRate, bits, channels, signed, bigEndian);
ByteArrayInputStream bais = new ByteArrayInputStream(byteBuffer);
AudioInputStream audioInputStream = new AudioInputStream(bais, format, buffer.length);
AudioSystem.write(audioInputStream, AudioFileFormat.Type.WAVE, out);
audioInputStream.close();
}
}

Some more detail on what you'd like to achieve would be helpful. If raw WAV data is okay for you, simply use a FileInputStream and probably a Scanner to turn it into numbers. But let me try to give you some meaningful sample code to get you started:
There is a class called com.sun.media.sound.WaveFileWriter for this purpose.
InputStream in = ...;
OutputStream out = ...;
AudioInputStream in = AudioSystem.getAudioInputStream(in);
WaveFileWriter writer = new WaveFileWriter();
writer.write(in, AudioFileFormat.Type.WAVE, outStream);
You could implement your own AudioInputStream that does whatever voodoo to turn your number arrays into audio data.
writer.write(new VoodooAudioInputStream(numbers), AudioFileFormat.Type.WAVE, outStream);
As #stacker mentioned, you should get yourself familiar with the API of course.

The javax.sound.sample package is not suitable for processing WAV files if you need to have access to the actual sample values. The package lets you change volume, sample rate, etc., but if you want other effects (say, adding an echo), you are on your own. (The Java tutorial hints that it should be possible to process the sample values directly, but the tech writer overpromised.)
This site has a simple class for processing WAV files: http://www.labbookpages.co.uk/audio/javaWavFiles.html

WAV File Specification
https://ccrma.stanford.edu/courses/422/projects/WaveFormat/
There is an API for your purpose
http://code.google.com/p/musicg/

First of all, you may need to know the headers and data positions of a WAVE structure, you can find the spec here.
Be aware that the data are little endian.
There's an API which may helps you to achieve your goal.

Wave files are supported by the javax.sound.sample package
Since isn't a trivial API you should read an article / tutorial which introduces the API like
Java Sound, An Introduction

If anyone still can find it required, there is an audio framework I'm working on that aimed to solve that and similar issues. Though it's on Kotlin. You can find it on GitHub: https://github.com/WaveBeans/wavebeans
It would look like this:
wave("file:///path/to/file.wav")
.map { it.asInt() } // here it as Sample type, need to convert it to desired type
.asSequence(44100.0f) // framework processes everything as sequence/stream
.toList() // read fully
.toTypedArray() // convert to array
And it's not dependent on Java Audio.

I use FileInputStream with some magic:
byte[] byteInput = new byte[(int)file.length() - 44];
short[] input = new short[(int)(byteInput.length / 2f)];
try{
FileInputStream fis = new FileInputStream(file);
fis.read(byteInput, 44, byteInput.length - 45);
ByteBuffer.wrap(byteInput).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(input);
}catch(Exception e ){
e.printStackTrace();
}
Your sample values are in short[] input!