We Keep Coding

VoIP RTP Streaming from/to server (in Java) to/from android

My target is to have a push-to-talk chat app in GSM/UMTS/LTE networks; initially I wanted use multicast addresses and peer-to-peer without overload the server; unfortunatly, after deep investigation, I discovered that multicast is not allowed in GSM/UMTS/LTE networks, therefore I have to use the server in order to bounce the VoIP packets. I' don't like very much this solution because I have to overload the server, but I didn't find any better solution. If you have an alternative solution is very much apprieciated...
Therefore I have to send VoIP from an android client to a server (PC), and viceversa. The server is in Java, it has to receive the VoIP packets and then to send the VoIP packets to other N clients; the server is a bouncer of VoIP packets.
I developped the code, but it doesn't work; I don't have any error, simply I had very bad VoIP service: I loose a lot of pieces and what I hear is very much noised... Where is the error? I suppose It should be in the server code; the server simply get the packet and resend them, without know that they are VoIP on RTP.
Please find below
the code I use to send the VoIP packets to server. It works because I don't have problems when I use it for individual call sending the VoIP packets directly from Android to Android; the code to receive in android the packets from server is very similar, so I don't recopy it. As you can see I use android.net.rtp .
the code I use on the Java server to bounce the VoIP packets
Thank you in advance, Fausto
// ANDROID CODE USED TO SEND VOIP TO SERVER
//Attribute definition
private static final AudioCodec myAudioCodec_COSTANTE = AudioCodec.PCMU ;
private static final int myAudioGroupTX_COSTANTE = AudioGroup.MODE_NORMAL ;
private static final int myAudioGroupRX_COSTANTE = AudioGroup.MODE_NORMAL ;
private static final int myRtpStreamTX_COSTANTE = RtpStream.MODE_SEND_ONLY ;
private static final int myRtpStreamRX_COSTANTE = RtpStream.MODE_RECEIVE_ONLY ;
private static final int myAudioManagerTX_COSTANTE = AudioManager.MODE_IN_COMMUNICATION;
private static final int myAudioManagerRX_COSTANTE = AudioManager.MODE_IN_COMMUNICATION;
//Method called for VoIP trasmission
myAudioStream = new AudioStream(localClientIP);
myAudioGroup = new AudioGroup();
myAudioManager = (AudioManager) myContext.getSystemService(Context.AUDIO_SERVICE);
myAudioGroup.setMode(myAudioGroupTX_COSTANTE);
myAudioStream.join(null);
myAudioStream.setCodec(myAudioCodec_COSTANTE);
myAudioStream.setMode(myRtpStreamTX_COSTANTE);
myAudioStream.associate(ipaddress_Server, port_Server)
myAudioStream.join(myAudioGroup);
myAudioManager.setMode(myAudioManagerTX_COSTANTE);
myAudioManager.setSpeakerphoneOn(false);
myAudioManager.setMicrophoneMute(false);
//JAVA SERVER CODE USED TO RECEIVE VOIP FROM ANDROID AND TO RESEND IT TO SERVER
DatagramSocket datagramSocket_RX_VoIP= new DatagramSocket();
DatagramSocket datagramSocket_TX_VoIP= new DatagramSocket();
int unicast_port_TX_VoIP = 5000 ;
String unicast_ip_TX_VoIP = "192.168.0.3";
Thread t = new Thread(new Runnable() {
public void run() {
try {
DatagramPacket myPacket;
while (true) {
myPacket = ManagePacket.initializePacket(); //Function to prepare the packe ; the problem is not here!!!
datagramSocket_RX_VoIP.receive(myPacket);
InetAddress ppp = InetAddress.getByName(unicast_ip_TX_VoIP);
myPacket.setAddress(ppp);
myPacket.setPort( unicast_port_TX_VoIP ) ;
datagramSocket_TX_VoIP.send(myPacket);
}
} catch (Exception ex) {
log.debug("Exception: " + ex.getMessage(), ex);
}
}
});
t.start();

You don't give enough detail about your application. With any UDP streaming application you need to address the following issues:
Network Jitter and Buffering: When a packet arrives, you cannot play the audio immediately after you receive it because the next packet might be later than expected and you will have a gap in your audio playback. The variance in the arrival rate is called network jitter. You need to buffer some amount of data before you try to play back. Usually you use some sort of ring buffer.
Packet loss: There will be packet losses with UDP. You need to "deal" with this. If you send 10 packets and packet #4 is missing, you can't play packet #3 then packet #5. It will sound bad. Ways to deal with this:
Loss Concealment: Try to minimize the bad effect off a packet lost. You can play silence (although this doesn't sound the best unless you fade it to silence). You can "estimate" the lost audio by generating a missing audio by examining the surrounding packets.
Forward Error Correction: Send packets more than once. There are numerous ways and schemes. The tradeoff is higher latency and more network utilization
Out of order arrivals: Packets may arrive out of order. Use the RTP sequence numbers to deal with this.
Audio streaming is not a trivial task. You can't just open a socket, send data, and play it on the other end and expect it to work. Your biggest concern with UDP streaming is network jitter and packet loss. If you don't want to deal with losses and you can have some extra latency, use TCP but be sure to buffer up enough audio before you start playing.

How can socket Server catch data from client just in the moment when client send it?

The program consist that I have to send a byte from by Android app to a Wifi Access Point, then this byte is interpreted by a hardware device.
I can send byte to client and it receives the byte correctly (and more bytes, don't know why, maybe because protocol). Hardware filter protocol bytes and catch only the correct one.
There's is how i send it (byte es created previously in another method, but it's correct):
public static void sendByte (Byte data) throws IOException {
DataOutputStream output;
Socket client;
client = new Socket("1.2.3.4", 2000);
output = new DataOutputStream(client.getOutputStream());
output.write(data);
output.close();
client.close();
Log.w("INFO","Data sended");
}
When I send the byte, hardware part change the color of a light, and it happens successfully.
I putted this 3 lines too:
StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
StrictMode.setThreadPolicy(policy);
Until here there's no problem.
Then I want to receive from that hardware bytes too. Imagine that someone change the color of that light, I want to know it. So I created a receiving method:
public static String readByte() throws IOException{
InputStream input;
DataInputStream iData;
String data = null;
try {
byte[] bytes = {0, 0, 0, 0, 0, 0, 0, 0};
Socket server = new Socket(2000);
Socket client = server.accept();
input = client.getInputStream();
iData = new DataInputStream(input);
Scanner scanner = new Scanner(iData);
iData.skip(0);
iData.read(bytes, 0, 8);
data = bytesToHex(bytes); //A simple method that change bytes to hex, this method it's correct
Log.w("READ", "" + data);
input.close();
client.close();
server.close();
}catch(IOException e){
Log.w("EROR","No es pot conectar");
}
return data;
}
Here I create a server that connects to client to get data (I don't know if it's necessary to create a server). The problem is that I always receive 7 bytes, and they are always the same, I used skip(7) to skip the protocol bytes, but then I don't receive anymore bytes.
I know that the hardware send bytes through the wifi where I'm connected, but I can't catch them.
_________TO SUM UP_________
I think the problem is that I can't catch the bytes because the hardware part simply send me, and my Android app can't store it. I would like to read the byte just when hardware sends me it, or something like this. I searched methods everywhere and object attributes and I can't find a solution to this :(
Thanks for your attention.
Wanted to know to if ServerSocket is strictly necessary or not.

When java program sends data to hardware, at that time, java program is the client and hardware is the server.
When the hardware sends data to java program, the hardware is the client and your java program is the server.
So, in your java program, you need to have two threads, one client thread (with a Socket to send bytes) and a server thread (with a ServerSocket, lets say for example, on port 8999) to receive bytes.
For the hardware to send you the bytes, it needs the following details.
1. IP Address or host name of the device on which your java program runs.
2. The port in which the ServerSocket is listening (8999 in our case).
If your hardware manual says it can send data to you, then, I am sure you can configure these details somewhere. (I am not sure what your hardware is. So, look at your hardware manual about how to configure these details).
Sometimes, the device can send response on the same Socket you opened. (Again, I am at a loss how it is done in your hardware. Refer your hardware manual). In such case, keep your socket open and read from it in a separate thread.

Java high speed udp traffic packet receiving for bandwidth testing

I am trying to build a bandwidth testing tool, kind of like IPerf but in java, I seem to be getting more packet loss than expected however at slightly higher bandwidths (starts at about 30-40Mb/s) and I was hoping someone could possibly point out some optimization or something that I am doing wrong that would cause me to be missing packets.
this is the receiving code, which hands off queues of size 2000 to another class which gathers metrics, it only passes relevant information from the packet. using NIO
while (data.isRunning())
{
if(channel.receive(buf) != null)
{
int j = buf.array().length;
//add the packets important information to the queue
packet_info.add(new PacketInfoContainer(buf.getLong(j-12), System.nanoTime(), buf.getInt(j-4)));
// if we have 2000 packets worth of information, time to handle it!
if((packet_info.size() == 2000))
{
Runnable r1;
//if this is running on the client side, do it this way so that we can calculate progress
if(client_side)
{
if(data_con.isUserRequestStop())
{
System.out.println("suposed to quit");
data.stopTest();
break;
}
if(packets_expected > 0)
{
total_packets_received+=1000;
setChanged();
notifyObservers("update_progress" + Integer.toString( (int) (((double)total_packets_received/(double)packets_expected) * 1000) ) );
}
r1 = new PacketHandler(packet_info, results, buffer_size, client);
}
//server side, no nonsense
else
{
r1 = new PacketHandler(packet_info, results, buffer_size);
}
pool.submit(r1);
packet_info = new LinkedList<PacketInfoContainer>();
}
}
buf.clear();
}

UDP is not very well... may be you can use TCP & check tcp stats of S.O. to see retransmissions...
netstat -s
you can use CharacterGenerator, change BufferedOutputStream to 64KB and removing os.flush(); to speedup and test...

It won't allow me to comment yet, here I go.
You shouldn't be seeing dropped packets until the wire limit is hit. I suggest isolating the problem of dropped packets and using tools to figure out if you have a hardware / environment problem before spending lots of time looking at your code.
https://serverfault.com/questions/561107/how-to-find-out-the-reasons-why-the-network-interface-is-dropping-packets
Have you tried to run iperf in UDP mode? then check the dropped packet statistics? https://iperf.fr/iperf-doc.php
bmon is a neat tool that will show you the carrier error, dropped, fifo error stats.

How can I keep TCP packets from being dropped?

I'm creating a program on my Android phone to send the output of the camera to a server on the same network. Here is my Java code:
camera.setPreviewCallbackWithBuffer(new Camera.PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera cam) {
try {
socket = new Socket("XXX.XXX.XXX.XXX", 3000);
out = socket.getOutputStream();
out.write(data);
socket.close();
} catch (Exception e) {
e.printStackTrace();
}
camera.addCallbackBuffer(data);
}
The server is a NodeJS server:
time = 0
video_server.on 'connection', (socket) ->
buffer = []
socket.on 'data', (data) ->
buffer.push data
socket.on 'end', ->
new_time = (new Date()).getTime()
fps = Math.round(1000/(new_time - time)*100)/100
console.log fps
time = new_time
stream = fs.createWriteStream 'image.jpg'
stream.on 'close', ->
console.log 'Image saved.', fps
stream.write data for data in buffer
stream.end()
My terminal is showing about 1.5 fps (5 Mbps). I know very little about network programming, but I do know there should definitely be enough bandwidth. Each image is 640x480x1.5 at 18 fps, which is about 63 Mbps. The local network should easily be able to handle this, but my debugger in Android is giving me a lot of "Connection refused" messages.
Any help on fixing my bad network practices would be great. (I'll get to image compression in a little bit -- but right now I need to optimize this step).

You've designed the system so that it has to do many times more work than it should have to do. You're requiring a connection to be built up and torn down for each frame transferred. That is not only killing your throughput, but it can also run you out of resources.
With a sane design, all that would be required to transfer a frame is to send and receive the frame data. With your design, for each frame, a TCP connection has to be built up (3 steps), the frame data has to be sent and received, and the TCP connection has to be torn down. Worse, the receiver cannot know it has received all of the frame data until the connection shutdown occurs. So this cannot be hidden in the background.
Design a sane protocol and the problems will go away.

Is this working at all? I do not see where you are binding to port 3000 on the server.
In any case, if this is a video stream, you should probably be using UDP instead of TCP. In UDP, packets may be dropped, but for a video stream this will probably not be noticeable. UDP communication requires much less overhead than TCP due to the number of messages exchanged. TCP contains a lot of "acking" to make sure each piece of data reaches its destination; UDP doesn't care, and thus sends less packets. In my experience, UDP based code is generally less complex than TCP based code.
_ryan

My ISP is forcing me to buffer tcp data before sending it

I have a Java TCP game server, I use java.net.ServerSocket and everything runs just fine, but recently my ISP did a some kind of an upgrade, where, if you send two packets very fast for the same TCP connexion, they close it by force.
This is why a lot of my players are disconnected randomly when there's a lot of traffic in game (when there is a lot of chance that the server will send 2 packets at same time for the same person)
Here is an example of what I mean:
If I do something like this, my ISP will close the connexion for no reason to both client and server side:
tcpOut.print("Hello.");
tcpOut.flush();
tcpOut.print("How are you?");
tcpOut.flush();
But it will work just fine if i do something like this:
tcpOut.print("Hello.");
tcpOut.flush();
Thread.sleep(200);
tcpOut.print("How are you?");
tcpOut.flush();
Or this:
tcpOut.print("Hello.");
tcpOut.print("How are you?");
tcpOut.flush();
This only started a couple of weeks ago when they (the ISP) did some changes to the service and the network. I noticed using Wireshark that you have to have at least ~150ms time between two packets for same TCP connexion or else it will close.
1)Do you guys know what is this called ? does is it even have a name ? Is it legal ?
Now I have to re-write my game server knowing that I use a method called: send(PrintWriter out, String packetData);
2)Is there any easy solution to ask java to buffer the data before it sends it to clients ? Or wait 150ms before each sending without having to rewrite the whole thing ? I did some googling but I can't find anything that deals with this problem. Any tips or information to help about this would be really appreciated, btw speed optimisation is very crucial. Thank you.

If your ISP imposes such quality of service policies and you have no way to negotiate them with it, I propose you enforce that rules on your side too with TCP/IP stack QoS configuration.
A flush marks your TCP packet as urgent (URG flag) so that it is sent whatever the buffer/TCP window state is. Now you have to tell your operating system or any network equipment on the line to either
ignore (or simply reset) the urgent flag when the previous packet has been sent in the last 150 ms and do some buffering if necessary
delay the delivery of consecutive urgent packets to honor the 150 ms constraint.
Probably an expensive software for Windows exists to do so. Personally, I think putting a Linux box as router between your Windows workstations and modem with the appropriate QoS settings in iptables and qdisc will do the trick.

You may create a Writer wrapper implementation to keep track of last flush call timestamp. A quick implementation is to add a wait call to honor the 150 ms delay between two consecutive flushes.
public class ControlledFlushWriter extends Writer {
private long enforcedDelay = 150;
private long lastFlush = 0;
private Writer delegated;
public ControlledFlushWriter(Writer writer, long flushDelay) {
this.delegated = writer:
this.enforcedDelay = flushDelay;
}
/* simple delegation for other abstract methods... */
public void flush() {
long now = System.currentTimeMillis();
if (now < lastFlush + enforcedDelay) {
try {
Thread.sleep(lastFlush + enforcedDelay - now);
} catch (InterruptedException e) {
// probably prefer to give up flushing
// instead of risking a connection reset !
return;
}
}
lastFlush = System.currentTimeMillis();
this.delegated.flush();
}
}
It now should be enough to wrap your existing PrintWriter with this ControlledFlushWriter to work-around your ISP QoS without re-writing all your application.
After all, it sounds reasonable to prevent a connection to flag any of its packet as urgent... In such a condition, it is difficult to implement a fair QoS link sharing.