We Keep Coding

h264 raw data stream and rendering in Android

We have an application for Android that work with network camera.
Our main problem is that video displayed with artifacts. Most of the screen in green squares. When you start to move your hand in front of the camera, the squares disappear but video still with artifacts. We have checked buffer length, packets size and many parameters…. Now we have no idea what is wrong.
I will describe the whole process:
Camera work with SIP protocol. According to SIP we collect SDP data and establish connection. We have discovered that video translate as H264 base profile in RTP packets.We receive UDP packets. Extract RTP. Look to the headers of RTP.
We received packets with type 7 and 8. These two packets we use to configure MediaCodec.
private void initMedia(ByteBuffer header_sps, ByteBuffer header_pps) {
try {
mMediaCodec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
//mMediaCodec = MediaCodec.createByCodecName("OMX.google.h264.decoder");
MediaFormat mediaFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 640, 480);
mediaFormat.setByteBuffer("csd-0", header_sps);
mediaFormat.setByteBuffer("csd-1", header_pps);
mMediaCodec.configure(mediaFormat, videoView.getHolder().getSurface(), null, 0);
mMediaCodec.start();
mConfigured = true;
startMs = System.currentTimeMillis();
show.start();
} catch (IOException e) {
e.printStackTrace();
}
}
Also we receive packets 28 it mean that it is parts and we should reconstruct it.
public ByteBuffer writeRawH264toByteBuffer() throws IOException, NotImplementedException {
ByteBuffer res = null;
switch (nal.getType()){
case NAL.FU_A: //FU-A, 5.8. Fragmentation Units (FUs)/rfc6184
FUHeader fu = getFUHeader();
if(fu.isFirst()){
//if(debug) System.out.println("first");
res = ByteBuffer.allocate(5+getH264PayloadLength());
res.put(H264RTP.NON_IDR_PICTURE);
res.put(getReconstructedNal());
res.put(rtp.getBuffer(), getH264PayloadStart(), getH264PayloadLength());
} else {
//if(debug) System.out.println("end");
res = ByteBuffer.allocate(getH264PayloadLength());
res.put(rtp.getBuffer(), getH264PayloadStart(), getH264PayloadLength());
}
break;
case NAL.SPS: //Sequence parameter set
case NAL.PPS: //Picture parameter set
case NAL.NAL_UNIT:
res = ByteBuffer.allocate(4+getH264PayloadLength());
//System.out.println("sps or pps write");
res.put(H264RTP.NON_IDR_PICTURE);
res.put(rtp.getBuffer(), rtp.getPayloadStart(), rtp.getPayloadLength());
break;
default:
throw new NotImplementedException("NAL type " + getNAL().getType() + " not implemented");
}
return res;
}
NON_IDR_PICTURE is byte array {0x00, 0x00, 0x00, 0x01}
We use VideoView for translating video on Android device
This one write packets:
if (mConfigured) {
int index = mMediaCodec.dequeueInputBuffer(mTimeoutUsDegueueInput);
if (index >= 0) {
ByteBuffer buffer = mMediaCodec.getInputBuffer(index);
//buffer.clear();
int capacity = wrapper.getByPayload().writeRawH264toByteBuffer(buffer);
mMediaCodec.queueInputBuffer(index, 0, capacity, wrapper.getSequence(), 0);
}
}
and this one renew VideoView (in the separate thread)
while(true)
if (mConfigured) {
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
int index = mMediaCodec.dequeueOutputBuffer(info, mTimeoutUsDegueueOutput);
if (index >= 0) {
mMediaCodec.releaseOutputBuffer(index, info.size > 0);
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
break;
}
}
} else {
try {
Thread.sleep(10);
} catch (InterruptedException ignore) {
}
}
Now i have no idea why video crashed with artifacts and what to debug.
Example of video:
screen of video

The problem was with a FU_A reconstruction.
Problem was in this string
int capacity = wrapper.getByPayload().writeRawH264toByteBuffer(buffer);
Packet FU_A should be reconstruct to full packet and only after it put in to the decoder

loop to readline() breaks without exceptions

I'm currently trying to create an App which enables me to control an Arduino board. For some reason, whenever it gets to the stage where the server waits for a message and gets to this loop, the loop breaks without any exceptions thrown when there is no action taken.
What I mean is that if I don't use any buttons to control the board, the loop breaks.
for(;;) {
if(message.equals("END")) {
break;
}
message = "";
switch (arduinoState) {
case 1: sendMessage("Server[Cmd]: fwd");
arduinoState = 0;
break;
case 2: sendMessage("Server[Cmd]: right");
arduinoState = 0;
break;
case 3: sendMessage("Server[Cmd]: bwd");
arduinoState = 0;
break;
case 4: sendMessage("Server[Cmd]: left");
arduinoState = 0;
break;
case 5: sendMessage("Server[Cmd]: kante");
arduinoState = 0;
break;
}
message = input.readLine();
updateStatusHandler.post(new updateStatusThread("Arduino: " + message, false));
}
EDIT: As requested once, here is the Server class.

So I've found a solution. The problem was that message = input.readLine(); keeps reading lines till the stream ends. As arduinoState = 0, no message will be sent to the client as there's no action to do thus no message from the client will be returned. This ends to a continously loop of reading. This means that even if I would change arduinoState, nothing will happen.
Solution: What I did was creating a class which is treated like a thread. Message will now on be sent independently from the waitForConnection() method.
sendMessageThread are used in setOnClickListeners() for buttons I use in the app.
class sendMessageThread implements Runnable {
private String s = "";
public sendMessageThread(String s) {
this.s = s;
}
public void run() {
try {
mWriter.write(s + '\n');
mWriter.flush();
mHandler.post(new updateStatusThread("Befehl gesendet: \"" + s + "\"", false));
} catch (IOException e) {
e.printStackTrace();
}
}
}
But I had to do a workaround lowering StrictMode. Why? Although I'm using a thread to take network actions, the exception android.os.NetworkOnMainThreadException is thrown. I guess it's because I'm using mWriter (BufferedWriter) as a global reference. I fetch the BufferedWriter in whileConnected() in the Server class.
StrictMode workaround:
//Remember to set it in the onCreate() method,
//right after super.onCreate() and setContentView()
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);
whileConnected() method
private void whileConnected() {
try {
mWriter = this.output;
do {
input.ready();
String line;
while((line = input.readLine()) != null) {
if(line.length() > 0) {
mHandler.post(new updateStatusThread("Arduino: " + line, false));
break;
}
}
} while (!message.equals("END"));
} catch (Exception e) {
e.printStackTrace();
}
}
If you still have any questions or tips and tricks to improve myself, feel free to leave an answer or comment down below.

Maybe the issue is infinite looping on UI thread, it's just a guess.
Also, have you tried while loop?

IOIO UART readback problems

I'm extending the BaseIOIOLooper to open up a UART device and send messages. I'm testing with a readback, where I send a packet over a line and receive that packet on another line and print it out. Because I don't want the InputStream.read() method to block, I am handling packet formation and input in a different thread. I have narrowed my problem down to the InputStream.read() method, which returns -1 (no bytes read, but no exception).
Here is what it looks like in the Looper thread:
#Override
protected void setup() throws ConnectionLostException, InterruptedException {
log_.write_log_line(log_header_ + "Beginning IOIO setup.");
// Initialize IOIO UART pins
// Input at pin 1, output at pin 2
try {
inQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
outQueue_ = MinMaxPriorityQueue.orderedBy(new ComparePackets())
.maximumSize(QUEUESIZE).create();
ioio_.waitForConnect();
uart_ = ioio_.openUart(1, 2, 38400, Uart.Parity.NONE, Uart.StopBits.ONE);
// Start InputHandler. Takes packets from ELKA on inQueue_
in_= new InputHandler(inQueue_, uart_.getInputStream());
in_.start();
// Start OutputHandler. Takes packets from subprocesses on outQueue_
out_= new OutputHandler(outQueue_);
out_.start();
// Get output stream
os_=uart_.getOutputStream();
// Set default target state
setTargetState(State.TRANSFERRING);
currInPacket_[0]=1; //Initial value to start transferring
log_.write_log_line(log_header_ + "IOIO setup complete.\n\t" +
"Input pin set to 1\n\tOutput pin set to 2\n\tBaud rate set to 38400\n\t" +
"Parity set to even\n\tStop bits set to 1");
} catch (IncompatibilityException e) {
log_.write_log_line(log_header_+e.toString());
} catch (ConnectionLostException e) {
log_.write_log_line(log_header_+e.toString());
} catch (Exception e) {
log_.write_log_line(log_header_+"mystery exception: "+e.toString());
}
}
And in the InputHandler thread:
#Override
public void run() {
boolean notRead;
byte i;
log_.write_log_line(log_header_+"Beginning InputHandler thread");
while (!stop) {
i = 0;
notRead = true;
nextInPacket = new byte[BUFFERSIZE];
readBytes = -1;
//StringBuilder s=new StringBuilder();
//TODO re-implement this with signals
while (i < READATTEMPTS && notRead) {
try {
// Make sure to adjust packet size. Done manually here for speed.
readBytes = is_.read(nextInPacket, 0, BUFFERSIZE);
/* Debugging
for (int j=0;j<nextInPacket.length;j++)
s.append(Byte.toString(nextInPacket[j]));
log_.write_log_line(log_header_+s.toString());
*/
if (readBytes != -1) {
notRead = false;
nextInPacket= new byte[]{1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0};
synchronized (q_) {
q_.add(nextInPacket);
}
//log_.write_log_line(log_header_ + "Incoming packet contains valid data.");
} else i++;
} catch (IOException e) {
log_.write_log_line(log_header_ + "mystery exception:\n\t" + e.toString());
}
}
if (i>=READATTEMPTS)
log_.write_log_line(log_header_+"Too many read attempts from input stream.");
/*
try {
sleep(100);
} catch (InterruptedException e) {
log_.write_log_line(log_header_+"fuck");
}
*/
}
}
On an oscilloscope, pins 1 and 2 both read an oscillating voltage, albeit at a very high amplitude, which is of some concern. Point is nothing is available to be read from the InputStream in the InputHandler class. Any ideas?

-1 returned from read() should only happen whenever the UART is closed. The closure can happen as result of explicitly calling close() on the Uart object or calling softReset() on the IOIO object.
The Android log might give you some clues about what's going on.
The reading you're seeing on the oscilloscope is suspicious: how high is "very high amplitude"? You should only ever see 0V or 3.3V on those pins, or floating in case the pins where not opened (or closed) for some reason.

JAVA SSLENGINE: Unsupported record version Unknown-0.0 while trying to unwrap bytebuffer record with SSLEngine

I am using an SSLEngine over Java NIO unblocking server sockets to handle connections. I am able to successfully handshake the clients and pass small record sets to the server. However when I try to transfer a file to the server text/binary I am getting the following error:
javax.net.ssl.SSLException: Unsupported record version Unknown-0.0
at sun.security.ssl.InputRecord.checkRecordVersion(InputRecord.java:552)
at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:113)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:862)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:775)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at ncp.io.network.tls.TLSWrapper.unwrap(TLSWrapper.java:170)
at ncp.io.network.tls.TLSIO.decodeData(TLSIO.java:110)
at ncp.io.network.tls.TLSIO.handleRead(TLSIO.java:71)
at ncp.io.network.SocketThread.run(SocketThread.java:137)
I am however unable to find out the reason for this error.
Below is my code snippet
#Override
public int handleRead(ByteBuffer temp) {
int read = opsManager.handleRead(temp);
if (read > 0) {
try {
tlsDecodeBuffer = decodeData(temp);
try {
temp.clear();
temp.put(tlsDecodeBuffer);
}catch (BufferOverflowException e){
temp = ByteBuffer.allocateDirect(tlsDecodeBuffer.remaining());
temp.put(tlsDecodeBuffer);
}
temp.flip();
temp.rewind();
if(tlsDecodeBuffer.hasRemaining())
tlsDecodeBuffer.compact();
else
tlsDecodeBuffer.clear();
}catch (SSLException e){
// Error occurs here:
e.printStackTrace();
log.warning("Insecure connection attempted/ SSL failure for:" + e.getMessage());
opsManager.close();
}
catch (IOException e) {
e.printStackTrace();
}
return read;
} else {
return -1;
}
}
private ByteBuffer decodeData(ByteBuffer input) throws IOException {
ncp.io.network.tls.TLSStatus stat = null;
boolean continueLoop = true;
do {
tlsDecodeBuffer = wrapper.unwrap(input, tlsDecodeBuffer);
switch (wrapper.getStatus()) {
case NEED_WRITE:
writeBuff(ByteBuffer.allocate(0));
break;
case UNDERFLOW:
if (tlsDecodeBuffer.capacity() == tlsDecodeBuffer.remaining()) {
throw new BufferUnderflowException();
} else {
input.compact();
continueLoop = false;
}
break;
case CLOSED:
if (log.isLoggable(Level.FINER)) {
log.finer("TLS Socket closed..." + toString());
}
throw new EOFException("Socket has been closed.");
default:
break;
}
stat = wrapper.getStatus();
} while (continueLoop && ((stat == TLSStatus.NEED_READ) || (stat == TLSStatus.OK))
&& input.hasRemaining());
if (continueLoop) {
if (input.hasRemaining()) {
input.rewind();
} else {
input.clear();
}
}
tlsDecodeBuffer.flip();
return tlsDecodeBuffer;
}

Your code doesn't make sense.
When you get a BufferOverflowException in unwrap(), you need to empty the target buffer via flip()/get()/compact().
When you get a BufferOverflowException in wrap(), you need to empty the target buffer via flip()/write()/compact(), where the write() goes to the network.
In neither case does it make any sense to allocate a new buffer etc.
rewind() after flip() doesn't begin to make sense in any context.
There are numerous posts and answers here about how to use the SSLEngine properly. I suggest you read them thoroughly.

Java SerialPortEvent issue: DATA_AVAILABLE calls faster then incoming data

I'm posting for the first time so I hope that I write everything according to the format.
Currently I'm working on a project using Serial Communication with RXTX between a java application and a measuring device. This works great, but now I want to catch the data that the device sends with events.
The code below works but has the following issue:
DATA_AVAILABLE will be called 4 times before all the data will be send.
I catch this in a string called vBuffer and I'm able to catch the data to get the full string.
Now I want to return this data (the full string), but cannot find a SerialPortEvent that will wait till all data is send to return this string.
In the example below I use OUTPUT_BUFFER_EMPTY but this is called at the beginning of sending a command. This means that when sending the command for the 2nd time, the OUTPUT_BUFFER_EMPTY event will return vBuffer with data from the first command, and immediately after starts the 2nd command. At 3rd time OUTPUT_BUFFER_EMPTY sends the data from the 2nd command and starts the 3rd etc.
Is there a way in DATA_AVAILABLE to wait till all data is send, or is there another event that will be called after all data is send?
Extra info: A command is send with a Stringbuilder of chars to make sure the right format for the device is send. The lay-out of a command is as follows: <STX><COMMAND><RTX><CR><LF>.
Might I be able to catch the end by looking at when the command ends? If so, how?
Update: This is the code how I send a function:
StringBuilder message = new StringBuilder();
message.append(new Character((char) 2)); // STX (Start of Text)
message.append("M"); // Command character
message.append(new Character((char) 3)); // ETX (End of Text
message.append(new Character((char) 13)); // CR (Carriage Return)
message.append(new Character((char) 10)); // LF (Line Feed)
outputStream.write(message.toString().getBytes());
After this the DATA_AVAILABLE will kick in. But doesn't wait till all received data is done.
Edit: To bump this up, still not further with the problem.
serialEvent Method :
public void serialEvent(SerialPortEvent event)
{
switch (event.getEventType())
{
case SerialPortEvent.BI:
case SerialPortEvent.OE:
case SerialPortEvent.FE:
case SerialPortEvent.PE:
case SerialPortEvent.CD:
case SerialPortEvent.CTS:
case SerialPortEvent.DSR:
case SerialPortEvent.RI:
case SerialPortEvent.OUTPUT_BUFFER_EMPTY:
if (vBuffer != "")
{
System.out.println(vBuffer);
}
break;
case SerialPortEvent.DATA_AVAILABLE:
byte[] readBuffer = new byte[40];
try
{
while (inputStream.available() > 0)
{
int numBytes = inputStream.read(readBuffer);
}
vBuffer += new String(readBuffer);
System.out.print(new String(readBuffer));
}
catch (IOException e)
{
System.out.println(e);
}
break;

I've found a way to check the command set STX and ETX too see if the message is complete (ETX is in the end of the message). If this checks true, then I've got a complete message.
Issue solved!
public void serialEvent(SerialPortEvent event)
{
if (event.getEventType() == SerialPortEvent.DATA_AVAILABLE)
{
System.out.println("Data available event received");
try
{
int available = inputStream.available();
byte[] readBuffer = new byte[available];
if (available > 0)
{
inputStream.read(readBuffer);
}
messageBuffer = messageBuffer + new String(readBuffer);
try
{
int start = messageBuffer.indexOf(new Character((char) 2));
int end = messageBuffer.indexOf(new Character((char) 10));
if (start >= 0 && end >= 0)
{
System.out.println("We found 1 complete message!!");
System.out.println(messageBuffer.substring(start, end));
_fireBufferEvent();
messageBuffer = "";
}
}
catch (IndexOutOfBoundsException ex)
{
System.out.println("IndexOutOfBoundsException, message not complete yet. Waiting for more data.");
}
}
catch (IOException ex)
{
System.out.println("IOException while reading data:");
System.out.println(ex);
}
}