We Keep Coding

Microsoft Asynchronous Conversation Transcription problem: SLF4J problem + Connection Issue Error 11001

I tried to run the sample code from:
https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-async-conversation-transcription
implementing helper class from:
https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/master/samples/java/jre/console/src/com/microsoft/cognitiveservices/speech/samples/console/WavStream.java
with some slight modifications so that it can read in wav files not limited to 16kHz 16bit single channel, and when I run it, this comes up:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Conversation transcriber canceled:SessionId:b2496d2c13424b3ba3138f2c8ce0893f ResultId:258796dbc69d491786f3ccdd8ec708d6 CancellationReason:Error CancellationErrorCode:ConnectionFailure Error details:<Connection failed (no connection to the remote host). Internal error: 1. Error details: 11001. Please check network connection, firewall setting, and the region name used to create speech factory. SessionId: b2496d2c13424b3ba3138f2c8ce0893f
Conversation transcriber stopped:SessionId: b2496d2c13424b3ba3138f2c8ce0893f.
I'm pretty sure that the API key and the region setup are correct and running and the Internet configuration isn't having any problem.
Here are the codes:
Main.java:
package speechsdk.quickstart;
import com.azure.core.util.polling.PollerFlux;
import com.azure.core.util.polling.SyncPoller;
import com.microsoft.cognitiveservices.speech.*;
import com.microsoft.cognitiveservices.speech.audio.*;
import com.microsoft.cognitiveservices.speech.remoteconversation.RemoteConversationTranscriptionClient;
import com.microsoft.cognitiveservices.speech.remoteconversation.RemoteConversationTranscriptionOperation;
import com.microsoft.cognitiveservices.speech.remoteconversation.RemoteConversationTranscriptionResult;
import com.microsoft.cognitiveservices.speech.transcription.Conversation;
import com.microsoft.cognitiveservices.speech.transcription.ConversationTranscriber;
import com.microsoft.cognitiveservices.speech.transcription.ConversationTranscriptionResult;
import javax.sound.sampled.AudioFileFormat;
import java.io.FileInputStream;
import java.util.UUID;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
/**
* Quickstart: recognize speech using the Speech SDK for Java.
*/
public class Main {
/**
* #param args Arguments are ignored in this sample.
*/
public static void main(String[] args) {
try {
// Create the speech config object
SpeechConfig speechConfig = SpeechConfig.fromSubscription("APIKEY", "SERVICEREGION");
speechConfig.setProperty("ConversationTranscriptionInRoomAndOnline", "true");
// Set the property for asynchronous transcription
speechConfig.setServiceProperty("transcriptionMode", "Async", ServicePropertyChannel.UriQueryParameter);
// Set the property for real-time plus asynchronous transcription
//speechConfig.setServiceProperty("transcriptionMode", "RealTimeAndAsync", ServicePropertyChannel.UriQueryParameter);
// pick a conversation Id that is a GUID.
String conversationId = UUID.randomUUID().toString();
// Create a Conversation
Conversation conversation = new Conversation(speechConfig, conversationId);
// Create an audio stream from a wav file or from the default microphone if you want to stream live audio from the supported devices
// Replace with your own audio file name and Helper class which implements AudioConfig using PullAudioInputStreamCallback
WavStream wavStream = new WavStream(new FileInputStream("sample.wav"));
PullAudioInputStreamCallback wavfilePullStreamCallback = wavStream;
// Create an audio stream format assuming the file used above is 16Khz, 16 bits and 8 channel pcm wav file
//AudioStreamFormat audioStreamFormat = AudioStreamFormat.getWaveFormatPCM((long)16000, (short)16,(short)8);
AudioStreamFormat audioStreamFormat = AudioStreamFormat.getWaveFormatPCM(wavStream.getSamplespersecond(), (short) wavStream.getBitspersample(), (short) wavStream.getChannel());
// Create an input stream
AudioInputStream audioStream = AudioInputStream.createPullStream(wavfilePullStreamCallback, audioStreamFormat);
// Create a conversation transcriber
ConversationTranscriber transcriber = new ConversationTranscriber(AudioConfig.fromStreamInput(audioStream));
// join a conversation
transcriber.joinConversationAsync(conversation);
// Add the event listener for the realtime events
transcriber.transcribed.addEventListener((o, e) -> {
System.out.println("Conversation transcriber Recognized:" + e.toString());
});
transcriber.canceled.addEventListener((o, e) -> {
System.out.println("Conversation transcriber canceled:" + e.toString());
try {
transcriber.stopTranscribingAsync().get();
} catch (InterruptedException ex) {
ex.printStackTrace();
} catch (ExecutionException ex) {
ex.printStackTrace();
}
});
transcriber.sessionStopped.addEventListener((o, e) -> {
System.out.println("Conversation transcriber stopped:" + e.toString());
try {
transcriber.stopTranscribingAsync().get();
} catch (InterruptedException ex) {
ex.printStackTrace();
} catch (ExecutionException ex) {
ex.printStackTrace();
}
});
// start the transcription.
Future<?> future = transcriber.startTranscribingAsync();
// Create a remote Conversation Transcription client
RemoteConversationTranscriptionClient client = new RemoteConversationTranscriptionClient(speechConfig);
// Get the PollerFlux for the remote operation
PollerFlux<RemoteConversationTranscriptionOperation, RemoteConversationTranscriptionResult> remoteTranscriptionOperation = client.getTranscriptionOperation(conversationId);
// Subscribe to PollerFlux to get the remote operation status
remoteTranscriptionOperation.subscribe(
pollResponse -> {
System.out.println("Poll response status : " + pollResponse.getStatus());
System.out.println("Poll response status : " + pollResponse.getValue().getServiceStatus());
}
);
// Obtain the blocking operation using getSyncPoller
SyncPoller<RemoteConversationTranscriptionOperation, RemoteConversationTranscriptionResult> blockingOperation = remoteTranscriptionOperation.getSyncPoller();
// Wait for the operation to finish
blockingOperation.waitForCompletion();
// Get the final result response
RemoteConversationTranscriptionResult resultResponse = blockingOperation.getFinalResult();
// Print the result
if(resultResponse != null) {
if(resultResponse.getConversationTranscriptionResults() != null) {
for (int i = 0; i < resultResponse.getConversationTranscriptionResults().size(); i++) {
ConversationTranscriptionResult result = resultResponse.getConversationTranscriptionResults().get(i);
System.out.println(result.getProperties().getProperty(PropertyId.SpeechServiceResponse_JsonResult.name()));
System.out.println(result.getProperties().getProperty(PropertyId.SpeechServiceResponse_JsonResult));
System.out.println(result.getOffset());
System.out.println(result.getDuration());
System.out.println(result.getUserId());
System.out.println(result.getReason());
System.out.println(result.getResultId());
System.out.println(result.getText());
System.out.println(result.toString());
}
}
}
System.out.println("Operation finished");
} catch (Exception ex) {
//System.out.println("Unexpected exception: " + ex.getMessage());
ex.printStackTrace();
assert(false);
System.exit(1);
}
}
}
Helper.java:
package speechsdk.quickstart;
import com.microsoft.cognitiveservices.speech.audio.PullAudioInputStreamCallback;
import com.microsoft.cognitiveservices.speech.internal.AudioConfig;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import org.slf4j.*;
class WavStream extends PullAudioInputStreamCallback {
private final InputStream stream;
private long samplespersecond;
private int bitspersample;
private int channel;
public long getSamplespersecond()
{
return samplespersecond;
}
public int getBitspersample()
{
return bitspersample;
}
public int getChannel()
{
return channel;
}
public WavStream(InputStream wavStream) {
try {
this.stream = parseWavHeader(wavStream);
} catch (Exception ex) {
throw new IllegalArgumentException(ex.getMessage());
}
}
#Override
public int read(byte[] dataBuffer) {
long ret = 0;
try {
ret = this.stream.read(dataBuffer, 0, dataBuffer.length);
} catch (Exception ex) {
System.out.println("Read " + ex);
}
return (int)Math.max(0, ret);
}
#Override
public void close() {
try {
this.stream.close();
} catch (IOException ex) {
// ignored
}
}
// endregion
// region Wav File helper functions
private int ReadInt32(InputStream inputStream) throws IOException {
int n = 0;
for (int i = 0; i < 4; i++) {
n |= inputStream.read() << (i * 8);
}
return n;
}
private long ReadUInt32(InputStream inputStream) throws IOException {
long n = 0;
for (int i = 0; i < 4; i++) {
n |= inputStream.read() << (i * 8);
}
return n;
}
private int ReadUInt16(InputStream inputStream) throws IOException {
int n = 0;
for (int i = 0; i < 2; i++) {
n |= inputStream.read() << (i * 8);
}
return n;
}
public InputStream parseWavHeader(InputStream reader) throws IOException {
// Note: assumption about order of chunks
// Tag "RIFF"
byte data[] = new byte[4];
int numRead = reader.read(data, 0, 4);
ThrowIfFalse((numRead == 4) && (data[0] == 'R') && (data[1] == 'I') && (data[2] == 'F') && (data[3] == 'F'), "RIFF");
// Chunk size
/* int fileLength = */ReadInt32(reader);
// Subchunk, Wave Header
// Subchunk, Format
// Tag: "WAVE"
numRead = reader.read(data, 0, 4);
ThrowIfFalse((numRead == 4) && (data[0] == 'W') && (data[1] == 'A') && (data[2] == 'V') && (data[3] == 'E'), "WAVE");
// Tag: "fmt"
numRead = reader.read(data, 0, 4);
ThrowIfFalse((numRead == 4) && (data[0] == 'f') && (data[1] == 'm') && (data[2] == 't') && (data[3] == ' '), "fmt ");
// chunk format size
long formatSize = ReadInt32(reader);
ThrowIfFalse(formatSize >= 16, "formatSize");
int formatTag = ReadUInt16(reader);
int channels = ReadUInt16(reader);
int samplesPerSec = (int) ReadUInt32(reader);
int avgBytesPerSec = (int) ReadUInt32(reader);
int blockAlign = ReadUInt16(reader);
int bitsPerSample = ReadUInt16(reader);
ThrowIfFalse(formatTag == 1, "PCM"); // PCM
//ThrowIfFalse(channels == 1, "single channel");
channel = channels;
//ThrowIfFalse(samplesPerSec == 16000, "samples per second");
samplespersecond = samplesPerSec;
//ThrowIfFalse(bitsPerSample == 16, "bits per sample");
bitspersample = bitsPerSample;
// Until now we have read 16 bytes in format, the rest is cbSize and is ignored
// for now.
if (formatSize > 16) {
numRead = reader.read(new byte[(int) (formatSize - 16)]);
ThrowIfFalse(numRead == (int)(formatSize - 16), "could not skip extended format");
}
// Second Chunk, data
// tag: data.
numRead = reader.read(data, 0, 4);
//for (byte i : data) System.out.print((char) i);
//System.out.println();
//ThrowIfFalse((numRead == 4) && (data[0] == 'd') && (data[1] == 'a') && (data[2] == 't') && (data[3] == 'a'), "data");
// data chunk size
// Note: assumption is that only a single data chunk
/* int dataLength = */ReadInt32(reader);
numRead = reader.read(data, 0, 4);
while (!((numRead == 4) && (data[0] == 'd') && (data[1] == 'a') && (data[2] == 't') && (data[3] == 'a')))
{
numRead = reader.read(data, 0, 4);
//for (byte i : data) System.out.print((char) i);
//System.out.println();
ReadInt32(reader);
}
//for (byte i : data) System.out.println((char) i);
return reader;
}
private static void ThrowIfFalse(Boolean condition, String message) {
if (!condition) {
throw new IllegalArgumentException(message);
}
}
// endregion
}

Using azure document, if you can please configure the following and provide us the log.
config.setProperty(PropertyId.Speech_LogFilename, "LogfilePathAndName");
By the way, what is the region you are using to do the transcription?

For conversation transcription, the input wave file should be 8 channel, 16 bits and 16 Khz wave file. Try putting a sleep after the following code till you get the session stopped called.
// start the transcription.
Future<?> future = transcriber.startTranscribingAsync();
Also if possible change the following code to
PollerFlux<RemoteConversationTranscriptionOperation, RemoteConversationTranscriptionResult> remoteTranscriptionOperation = client.getTranscriptionOperation(conversationId,5);
Do let us know if it works.

Read and Write to USB Device on Raspberry Pi 3 using Java

I'm using Raspbian on a Raspberry Pi 3. I'm learning how to code in Java (SE runtime version 1.8.0_65), and I need to communicate raw data with a USB connected Bill Acceptor. According to the manufacturer's documentation, the USB unit mimics a serial-type interface. When I plug the device in, it appears in /dev/serial/by-id. I wrote C code 20+ years ago using modems on SCO Unix setups. If memory serves, I basically treated the modems /dev/ttyxx as a file for reading and writing. I wrote a C++ program 10+ years ago on Windows XP using a similar Bill Acceptor unit (same manufacturer and model) connected to a serial port instead of USB. The documentation shows both USB and serial units use the same data strings for input and output, so I know the SendString data should be correct for triggering a valid response. However, I'm not getting a reply from the unit, so I'm guessing I'm not connecting properly to it.
Here is the code...
public static void main(String[] args) {
try
{
System.out.println("*****************************");
System.out.println("***** Starting Program *****");
System.out.println("*****************************");
String strUsbDeviceDir = "/dev/serial/by-id";
File myUsbDeviceDir = new File(strUsbDeviceDir);
if(myUsbDeviceDir.exists())
{
String[] myUsbDevices = myUsbDeviceDir.list();
for(int i=0; i<myUsbDevices.length; i++)
{
if(myUsbDevices[i].contains("EBDS_over_USB"))
{
System.out.println("Connecting to " + myUsbDevices[i]);
funcBillAcceptor(strUsbDeviceDir + "/" + myUsbDevices[i]);
}
else
{
System.out.println("Not using " + myUsbDevices[i]);
}
}
}
}
catch (Exception ex)
{
System.err.println(ex.toString());
}
}
public static void funcBillAcceptor(String strBillAcceptor)
{
boolean bOddCount = false;
byte[] SendString = new byte[8];
byte[] RecvString = new byte[10];
byte CheckDigit;
int iSendStringCount, iRecvStringCount, iRecvEmptyCount;
try
{
File fBillAcceptor = new File(strBillAcceptor);
if(!fBillAcceptor.canRead())
{
System.out.println("No Read Permission for " + strBillAcceptor);
return;
}
if(!fBillAcceptor.canWrite())
{
System.out.println("No Write Permission for " + strBillAcceptor);
return;
}
RandomAccessFile rafBillAcceptor = new RandomAccessFile(strBillAcceptor, "rwd");
if(rafBillAcceptor != null)
{
System.out.println("Successfully opened " + strBillAcceptor);
}
while(fBillAcceptor.exists())
{
SendString[0] = (byte) 0x02; //STX
SendString[1] = (byte) 0x08;
if(bOddCount)
{
SendString[2] = (byte) 0x10;
bOddCount = false;
}
else
{
SendString[2] = (byte) 0x11;
bOddCount = true;
}
SendString[3] = (byte) 0x1F;
SendString[4] = (byte) 0x0C;
SendString[5] = (byte) 0x00;
SendString[6] = (byte) 0x03; //ETX
//CheckDigit skips STX (byte 0) with byte 1 as seed/initial value
//To calculate the check digit, start with next byte (2)
CheckDigit = SendString[1];
iSendStringCount = 2;
while(SendString[iSendStringCount] != 0x03)
{
CheckDigit = (byte) (SendString[iSendStringCount]^CheckDigit); //XOR current CheckDigit value with next byte
iSendStringCount++;
}
iSendStringCount++; //advance one more so we don't overwrite ETX
SendString[iSendStringCount] = (byte) CheckDigit;
try
{
rafBillAcceptor.write(SendString);
System.out.println("Sent: " + DatatypeConverter.printHexBinary(SendString));
}
catch (Exception ex)
{
System.err.println("Write exception: " + ex.toString());
}
System.out.println("Reading...");
iRecvStringCount = iRecvEmptyCount = 0;
try
{
do
{
iRecvStringCount = rafBillAcceptor.read(RecvString);
System.out.println("Read " + iRecvStringCount + " bytes.");
if(iRecvStringCount < 0)
{
iRecvEmptyCount++;
Thread.sleep(5);
}
} while (iRecvStringCount < 0 && iRecvEmptyCount < 100);
if(iRecvStringCount > 0)
{
System.out.println("Received: " + DatatypeConverter.printHexBinary(RecvString));
}
}
catch (Exception ex)
{
System.err.println("Read exception: " + ex.toString());
}
}
}
catch (Exception ex)
{
System.err.println(ex.toString());
}
}
Here is the output I'm getting...
*****************************
***** Starting Program *****
*****************************
Connecting to usb-Silicon_Labs_Series_2000_Bill_Acceptor__EBDS_over_USB__46580120748-if00-port0
Successfully opened /dev/serial/by-id/usb-Silicon_Labs_Series_2000_Bill_Acceptor__EBDS_over_USB__46580120748-if00-port0
Sent: 0208111F0C00030A
Reading...
Am I missing something obvious, or just going about this all wrong? Thanks for any suggestions!

I was able to successfully communicate with the Bill Acceptor from a Raspberry Pi 3 (running Raspbian) using the RXTXComm library. When communicating with the device from an old Windows XP computer using a 9-pin serial/RS232 harness, I had to use 9600/7/E/1. If I don't use those values for setSerialPortParams (in my sample code), I'm not able to correctly send/receive data. So, you need to know what the device is expecting in order to use this approach with other "serial" hardware connected via USB. Below is code with the basic functionality. I started with one of the sample programs that come with the RXTXComm library and built out from there. Since this is a sample program, I haven't yet added logic to verify the checksum digit for data coming from the device.
You may notice at the bottom of the code that I'm able to determine the /dev/ttyUSBx using the canonical name of the device that I pull from the /dev/serial/by-id directory/folder. It finds the correct device name regardless of which USB port the unit is plugged in to or what order the USB devices get initialized by the system. I have a card dispenser that is also a USB serial device, so I was able to test and confirm this functionality.
public TwoWaySerialComm()
{
super();
}
static boolean bReadyToSend = false;
void connect ( String portName ) throws Exception
{
CommPortIdentifier portIdentifier = CommPortIdentifier.getPortIdentifier(portName);
if ( portIdentifier.isCurrentlyOwned() )
{
System.out.println("Error: " + portName + " is currently in use.");
}
else
{
CommPort commPort = portIdentifier.open(this.getClass().getName(), 2000);
if ( commPort instanceof SerialPort )
{
SerialPort serialPort = (SerialPort) commPort;
serialPort.setSerialPortParams(9600, SerialPort.DATABITS_7, SerialPort.STOPBITS_1, SerialPort.PARITY_EVEN);
InputStream in = serialPort.getInputStream();
OutputStream out = serialPort.getOutputStream();
// Thread handling logic
// https://www.javaspecialists.eu/archive/Issue056.html
Thread readThread = new Thread(new SerialReader(in));
readThread.start();
Thread writeThread = new Thread(new SerialWriter(out));
writeThread.start();
// Running threads for 10 seconds then stopping to ensure threads are shutting down
long threadCurrTime = System.currentTimeMillis();
long threadStartTime = System.currentTimeMillis();
while( (threadCurrTime - threadStartTime) < 10000)
{
try
{
Thread.sleep(100);
}
catch (InterruptedException ex)
{
Logger.getLogger(TwoWaySerialComm.class.getName()).log(Level.SEVERE, null, ex);
}
threadCurrTime = System.currentTimeMillis();
}
if(writeThread.isAlive())
{
//System.out.println("Sending interrupt to SerialWriter thread...");
writeThread.interrupt();
writeThread.join();
//System.out.println("SerialWriter thread is shut down.");
}
//else
//{
// System.out.println("SerialWriter thread is already shut down.");
//}
if(readThread.isAlive())
{
//System.out.println("Sending interrupt to SerialReader thread...");
readThread.interrupt();
readThread.join();
//System.out.println("SerialReader thread is shut down.");
}
//else
//{
// System.out.println("SerialReader thread is already shut down.");
//}
commPort.close();
}
else
{
System.out.println("Error: " + portName + " is not recognized as a valid serial device.");
}
}
}
/* SerialReader thread logic */
public static class SerialReader implements Runnable
{
InputStream in;
boolean bShuttingDown = false;
public SerialReader ( InputStream in )
{
this.in = in;
}
public void run ()
{
byte[] RecvString = new byte[12];
String strResponse;
int len = -1;
try
{
while (!bShuttingDown)
{
len = this.in.read(RecvString);
if( len > -1 )
{
strResponse = "";
if(RecvString[0] == 0x02 && RecvString[9] == 0x03)
{
if(RecvString[3] == 0x00 && RecvString[4] == 0x00 && RecvString[5] == 0x00 && RecvString[6] == 0x00)
{
strResponse = "Device not ready.";
}
else
{
//-- RecvString[3]------------------
if(RecvString[3] == 0x01)
strResponse = " - Idling";
else
if(RecvString[3] == 0x02)
strResponse = " - Accepting";
else
if(RecvString[3] == 0x04)
{
strResponse = " - Escrowed";
if(RecvString[5] == 0x08)
strResponse += " $1";
else
if(RecvString[5] == 0x10)
strResponse += " $2";
else
if(RecvString[5] == 0x18)
strResponse += " $5";
else
if(RecvString[5] == 0x20)
strResponse += " $10";
else
if(RecvString[5] == 0x28)
strResponse += " $20";
else
strResponse += " unrecognized bill inserted";
}
else
if(RecvString[3] == 0x08)
strResponse = " - Stacking";
else
if(RecvString[3] == 0x10)
strResponse = " - Stacked";
else
if(RecvString[3] == 0x11)
strResponse = " - Returning";
else
if(RecvString[3] == 0x12)
strResponse = " - Returned";
//-- RecvString[4]------------------
if(RecvString[4] == 0x01)
strResponse += " - Cheated";
else
if(RecvString[4] == 0x02)
strResponse += " - Rejected";
else
if(RecvString[4] == 0x04)
strResponse += " - Jammed";
else
if(RecvString[4] == 0x08)
strResponse += " - Bill Stacker Full";
else
if(RecvString[4] == 0x10)
strResponse += " - Removable Cassette Installed";
else
if(RecvString[4] == 0x11)
strResponse += " - Reserved";
else
if(RecvString[4] == 0x12)
strResponse += " - Calibration mode";
//-- RecvString[5]------------------
if(RecvString[5] == 0x01)
strResponse += " - Power Up Reset";
else
if(RecvString[5] == 0x02)
strResponse += " - Invalid Command";
else
if(RecvString[5] == 0x04)
strResponse += " - Non-recoverable fault";
}
if(!strResponse.contains("Idling"))
System.out.println("Recv: " + DatatypeConverter.printHexBinary(RecvString) + strResponse);
}
else
{
System.out.println("Recv (invalid): " + DatatypeConverter.printHexBinary(RecvString));
}
try
{
Thread.sleep(100); // need this delay before we send next polling message, otherwise the data doesn't come in correctly
bReadyToSend = true;
}
catch (InterruptedException ex)
{
Thread.currentThread().interrupt();
//System.out.println("Shut down SerialReader thread.");
bShuttingDown = true;
break;
}
}
}
}
catch ( IOException ex )
{
System.out.println("Recv exception: " + ex.toString());
}
}
}
/* SerialWriter thread logic */
public static class SerialWriter implements Runnable
{
OutputStream out;
long lastSendTime = System.currentTimeMillis() - 1001;
long currSendTime;
byte[] SendString = new byte[8];
byte CheckDigit;
int iSendStringCount;
boolean bOddCount = true;
boolean bShuttingDown = false;
public SerialWriter ( OutputStream out )
{
this.out = out;
}
public void run ()
{
while(!bShuttingDown)
{
currSendTime = System.currentTimeMillis();
if(currSendTime - lastSendTime > 1000) // if it's been more than a second, send query string
bReadyToSend = true;
if(bReadyToSend)
{
SendString[0] = (byte) 0x02; //STX
SendString[1] = (byte) 0x08;
if(bOddCount)
{
SendString[2] = (byte) 0x10;
bOddCount = false;
}
else
{
SendString[2] = (byte) 0x11;
bOddCount = true;
}
SendString[3] = (byte) 0x7F;
SendString[4] = (byte) 0x1C;
SendString[5] = (byte) 0x00;
SendString[6] = (byte) 0x03; //ETX
//CheckDigit skips STX (byte 0) with byte 1 as seed/initial value
//To calculate the check digit, start with next byte (2)
CheckDigit = SendString[1];
iSendStringCount = 2;
while(SendString[iSendStringCount] != 0x03)
{
CheckDigit = (byte) (SendString[iSendStringCount]^CheckDigit); //XOR current CheckDigit value with next byte
iSendStringCount++;
}
iSendStringCount++; //advance one more so we don't overwrite ETX
SendString[iSendStringCount] = (byte) CheckDigit;
try
{
lastSendTime = System.currentTimeMillis();
this.out.write(SendString);
//System.out.println("Sent: " + DatatypeConverter.printHexBinary(SendString));
}
catch ( IOException ex )
{
System.out.println("Send exception: " + ex.toString());
}
try
{
Thread.sleep(1); // this is hear simply to catch an external interrupt
}
catch (InterruptedException ex)
{
Thread.currentThread().interrupt();
//System.out.println("Shut down SerialWriter thread.");
bShuttingDown = true;
break;
}
bReadyToSend = false;
}
}
}
}
public static void main ( String[] args )
{
try
{
System.out.println("*****************************");
System.out.println("***** Starting Program *****");
System.out.println("*****************************");
String strUsbDeviceDir = "/dev/serial/by-id";
File myUsbDeviceDir = new File(strUsbDeviceDir);
if(myUsbDeviceDir.exists())
{
String[] myUsbDevices = myUsbDeviceDir.list();
for(int i=0; i<myUsbDevices.length; i++)
{
if(myUsbDevices[i].contains("EBDS_over_USB"))
{
File tempFile = new File(strUsbDeviceDir + "/" + myUsbDevices[i]);
String usbCanonicalName = tempFile.getCanonicalFile().toString(); //gives me /dev/ttyUSBx where 'x' is the USB device number
System.out.println("Connecting to " + usbCanonicalName + " (" + myUsbDevices[i] + ")");
(new TwoWaySerialComm()).connect(usbCanonicalName);
}
else
{
System.out.println("Not using " + myUsbDevices[i]);
}
}
}
}
catch ( Exception ex )
{
System.out.println("Connect exception: " + ex.toString());
}
System.out.println("*****************************");
System.out.println("***** Program Finished ******");
System.out.println("*****************************");
}
Output when I put a $1 bill in (I'm developing/compiling from NetBeans IDE 8.2 on Windows 10 Pro and running using remote debugging on the RPi3. I'm guessing that's the source of the RXTX Version mismatch warning):
*****************************
***** Starting Program *****
*****************************
Connecting to /dev/ttyUSB0 (usb-Silicon_Labs_Series_2000_Bill_Acceptor__EBDS_over_USB__46580120748-if00-port0)
Stable Library
=========================================
Native lib Version = RXTX-2.2pre2
Java lib Version = RXTX-2.1-7
WARNING: RXTX Version mismatch
Jar version = RXTX-2.1-7
native lib Version = RXTX-2.2pre2
Recv (invalid): 020000000000000000000000
Recv (invalid): 0B2001100800503603540000
Recv: 020B20041008005036035100 - Escrowed $1 - Removable Cassette Installed
*****************************
***** Program Finished ******
*****************************
I hope this description and sample code can help someone else.

Using Kafka low-level API, should I commit the offset when finished fetching data?

public void run() {
// find the meta data about the topic and partition we are interested in
PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition);
if (metadata == null) {
System.out.println("Can't find metadata for Topic and Partition. Exiting");
return;
}
if (metadata.leader() == null) {
System.out.println("Can't find Leader for Topic and Partition. Exiting");
return;
}
String leadBroker = metadata.leader().host();
String clientName = "Client_" + a_topic + "_" + a_partition;
SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
long readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), clientName);
//long readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.LatestTime(), clientName);
int numErrors = 0;
while (a_maxReads > 0) {
if (consumer == null) {
consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
}
FetchRequest req = new FetchRequestBuilder()
.clientId(clientName)
.addFetch(a_topic, a_partition, readOffset, 100000) // Note: this fetchSize of 100000 might need to be increased if large batches are written to Kafka
.build();
FetchResponse fetchResponse = consumer.fetch(req);
if (fetchResponse.hasError()) {
numErrors++;
// Something went wrong!
short code = fetchResponse.errorCode(a_topic, a_partition);
System.out.println("Error fetching data from the Broker:" + leadBroker + " Reason: " + code);
if (numErrors > 5) break;
if (code == ErrorMapping.OffsetOutOfRangeCode()) {
// We asked for an invalid offset. For simple case ask for the last element to reset
readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.LatestTime(), clientName);
continue;
}
consumer.close();
consumer = null;
try {
leadBroker = findNewLeader(leadBroker, a_topic, a_partition, a_port);
} catch (Exception e) {
e.printStackTrace();
}
continue;
}
numErrors = 0;
long numRead = 0;
for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(a_topic, a_partition)) {
long currentOffset = messageAndOffset.offset();
if (currentOffset < readOffset) {
System.out.println("Found an old offset: " + currentOffset + " Expecting: " + readOffset);
continue;
}
readOffset = messageAndOffset.nextOffset();
ByteBuffer payload = messageAndOffset.message().payload();
byte[] bytes = new byte[payload.limit()];
payload.get(bytes);
try {
dataPoints.add(simpleAPIConsumer.parse(simpleAPIConsumer.deserializing(bytes)));//add data to List
} catch (Exception e) {
e.printStackTrace();
}
numRead++;
a_maxReads--;
}
if (numRead == 0) {
try {
Thread.sleep(1000);
} catch (InterruptedException ie) {
}
}
}
simpleAPIConsumer.dataHandle(dataPoints);//Handel Data
if (consumer != null) consumer.close();
}
I found this method in Kafka source. Should I use it?
/**
* Commit offsets for a topic to Zookeeper
* #param request a [[kafka.javaapi.OffsetCommitRequest]] object.
* #return a [[kafka.javaapi.OffsetCommitResponse]] object.
*/
def commitOffsets(request: kafka.javaapi.OffsetCommitRequest):kafka.javaapi.OffsetCommitResponse = {
import kafka.javaapi.Implicits._
underlying.commitOffsets(request.underlying)
}

The purpose of committing an offset after every fetch is to achieve exactly-once message processing.
You need to make sure that you commit offset once you processed the message (where "process" means whatever you do with a message after you pull it out from Kafka). Like you're wrapping message processing and offset commit into a transaction, where either both succeed or fail.
This way, if your client crashes you'll be able to start from the correct offset after you restart.

Mime4j: DefaultMessageBuilder fails to parse mbox content

I've downloaded mime4j 0.8.0 snapshot from subversion and built it with maven.
The relevant jars I generated can be found here.
Now I try to parse a toy mbox file from mime4j test.
I use this sample code. Briefly:
final File mbox = new File("c:\\mbox.rlug");
int count = 0;
for (CharBufferWrapper message : MboxIterator.fromFile(mbox).charset(ENCODER.charset()).build()) {
System.out.println(messageSummary(message.asInputStream(ENCODER.charset())));
count++;
}
System.out.println("Found " + count + " messages");
+
private static String messageSummary(InputStream messageBytes) throws IOException, MimeException {
MessageBuilder builder = new DefaultMessageBuilder();
Message message = builder.parseMessage(messageBytes);
return String.format("\nMessage %s \n" +
"Sent by:\t%s\n" +
"To:\t%s\n",
message.getSubject(),
message.getSender(),
message.getTo());
}
The output is:
Message null Sent by: null To: null
Message null Sent by: null To: null
Message null Sent by: null To: null
Message null Sent by: null To: null
Message null Sent by: null To: null
Found 5 messages
There are indeed 5 messages, but why are all fields null?

Based on #zvisofer answer, I found the guilty piece of code in BufferedLineReaderInputStream:
#Override
public int readLine(final ByteArrayBuffer dst)
throws MaxLineLimitException, IOException {
if (dst == null) {
throw new IllegalArgumentException("Buffer may not be null");
}
if (!readAllowed()) return -1;
int total = 0;
boolean found = false;
int bytesRead = 0;
while (!found) {
if (!hasBufferedData()) {
bytesRead = fillBuffer();
if (bytesRead == -1) {
break;
}
}
int i = indexOf((byte)'\n');
int chunk;
if (i != -1) {
found = true;
chunk = i + 1 - pos();
} else {
chunk = length();
}
if (chunk > 0) {
dst.append(buf(), pos(), chunk);
skip(chunk);
total += chunk;
}
if (this.maxLineLen > 0 && dst.length() >= this.maxLineLen) {
throw new MaxLineLimitException("Maximum line length limit exceeded");
}
}
if (total == 0 && bytesRead == -1) {
return -1;
} else {
return total;
}
}
The best thing to do would be to report the bug but here is a fix, a little dirty but it's working fine
Create the class org.apache.james.mime4j.io.BufferedLineReaderInputStream in your project
Replace the method public int readLine(final ByteArrayBuffer dst) by this one :
#Override
public int readLine(final ByteArrayBuffer dst)
throws MaxLineLimitException, IOException {
if (dst == null) {
throw new IllegalArgumentException("Buffer may not be null");
}
if (!readAllowed()) return -1;
int total = 0;
boolean found = false;
int bytesRead = 0;
while (!found) {
if (!hasBufferedData()) {
bytesRead = fillBuffer();
if (bytesRead == -1) {
break;
}
}
int chunk;
int i = indexOf((byte)'\r');
if (i != -1) {
found = true;
chunk = i + 2 - pos();
} else {
i = indexOf((byte)'\n');
if (i != -1) {
found = true;
chunk = i + 1 - pos();
} else {
chunk = length();
}
}
if (chunk > 0) {
dst.append(buf(), pos(), chunk);
skip(chunk);
total += chunk;
}
if (this.maxLineLen > 0 && dst.length() >= this.maxLineLen) {
throw new MaxLineLimitException("Maximum line length limit exceeded");
}
}
if (total == 0 && bytesRead == -1) {
return -1;
} else {
return total;
}
}
Enjoy both unix and dos files :)

I found the problem.
DefaultMessageBuilder fails to parse mbox files that have windows line separator \r\n.
When replacing them with UNIX line separator \n the parsing works.
This is a critical issue, since the mbox files downloaded from Gmail use \r\n.

I downloaded your jar files, the sample code that you pointed to, and the sample mbox file that you pointed to, compiled the sample (with no changes) and ran it against the sample mbox file.
It worked as expected (fields contained the expected data, not nulls).
This was on a Mac with Java 1.6_0_65, and also with 1.8.0_11
Output was as follows:
$ java -cp .:apache-mime4j-core-0.8.0-SNAPSHOT.jar:apache-mime4j-dom-0.8.0-SNAPSHOT.jar:apache-mime4j-mbox-iterator-0.8.0-SNAPSHOT.jar IterateOverMbox mbox.rlug.txt
Message Din windows ma pot, din LINUX NU ma pot conecta (la ZAPP)
Sent by: rlug-bounce#lug.ro To: [rlug#lug.ro]
Message Re: RH 8.0 boot floppy Sent by: rlug-bounce#lug.ro
To: [rlug#lug.ro]
Message Qmail mysql virtualusers +ssl + smtp auth +pop3 Sent
by: rlug-bounce#lug.ro To: [rlug#lug.ro]
Message Re: Din windows ma pot, din LINUX NU ma pot conecta (la ZAPP)
Sent by: rlug-bounce#lug.ro To: [rlug#lug.ro]
Message LSTP problem - solved Sent by: rlug-bounce#lug.ro
To: [rlug#lug.ro]
Found 5 messages Done in: 108 milis

java websockets onMessage return InputStream to frontend javascript

I have come across a few posts, none of which has really answered my question, I have also been in the process of raising the question but putting it off only to research more dead ends...
So here is the issue, using j2ssh libraries to make ssh connections :
#OnMessage
public String handleOpen(Session usersession) {
System.out.println("Client is now connected.")
properties = new SshConnectionProperties();
properties.setHost(host)
properties.setPort(sshPort)
ssh.connect(properties, new IgnoreHostKeyVerification())
int result=0
PasswordAuthenticationClient pwd = new PasswordAuthenticationClient()
pwd.setUsername(username)
pwd.setPassword(password)
result = ssh.authenticate(pwd)
if(result == 4) {
isAuthenticated=true
}
// Evaluate the result
if (isAuthenticated) {
session = ssh.openSessionChannel()
SessionOutputReader sor = new SessionOutputReader(session)
if (session.requestPseudoTerminal("gogrid",80,24, 0 , 0, "")) {
if (session.startShell()) {
ChannelOutputStream out = session.getOutputStream()
session.getOutputStream().write("${usercommand} \n".getBytes())
println "--------------------------------authenticated user"
handleMessage(session.getInputStream(),usersession)
}
}
}
}
#OnMessage
public void handleMessage(InputStream stream, Session usersession) {
try {
usersession.getBasicRemote().sendText('SOMETHING')
usersession.getBasicRemote().sendObject(stream)
//return stream
} catch (IOException e) {
}
}
What I would like to do is for handleMessage to return the entire inputStream
Then within the javascript on the client's webpage to recieve that input stream :
...
webSocket.onmessage=function(message) {processMessage(message);};
var THRESHOLD = 10240;
...
function processMessage(message) {
//webSocket.send(textMessage.value);
//textMessage.value="";
messagesTextarea.value +=" woot woot woot\n";
if(message.data instanceof ArrayBuffer) {
var wordarray = new Uint16Array(message.data);
for (var i = 0; i < wordarray.length; i++)
{
console.log(wordarray[i]);
wordarray[i]=wordarray[i]+1;
}
messagesTextarea.value +=""+wordarray.buffer+"\n"
}else{
messagesTextarea.value +=""+message.data+"\n"
}
/*
setInterval( function() {
//messagesTextarea.value +=" Receive from Server ===> "+ message.data +"\n";
if (webSocket.bufferedAmount < THRESHOLD) {
if ((message.data != null)&&(message.data !="")) {
messagesTextarea.value +=" got from Server ===> ("+message.data+")";
}
} }, 5000);
*/
}
The objective is to launch ssh connection (no input really required - although with input would be of benefit for further client calls...
Once connected to run
handleMessage(session.getInputStream(),usersession)
This then returns the stream from the connection back to the javascript frontend.
At the moment I can see a connection to handleMessage but nothing is being returned to javascript frontend...
So I guess its related to how do you stream inputstreams via a javascript frontend.....

Ive actually got it working - up for lots of testing at the momemnt:
Java Endpoint:
#OnMessage
public void handleMessage(String message, Session usersession) {
InputStream input=session.getInputStream()
byte[] buffer=new byte[255]
int read;
int i=0
def pattern = ~/^\s+$/
while((read = input.read(buffer)) > 0) {
String out1 = new String(buffer, 0, read)
def m=pattern.matcher(out1).matches()
if (m==false) {
usersession.getBasicRemote().sendText(out1)
}
}
}
Front end Java script (note the call is not hitting array if call )
function processMessage(message) {
if(message.data instanceof ArrayBuffer) {
var wordarray = new Uint16Array(message.data);
for (var i = 0; i < wordarray.length; i++) {
console.log(wordarray[i]);
wordarray[i]=wordarray[i]+1;
}
messagesTextarea.value +=""+wordarray.buffer+"\n"
}else{
messagesTextarea.value +=""+message.data+"\n"
}
}