We Keep Coding

(Netty) Python client to Java server, ASCII to binary or hex

I need to use python client to connect Netty server(base on JAVA).
Client:
UDP_IP = "127.0.0.1"
UDP_PORT = 9999
MESSAGE = "16,01,12,03,1b,14,30,23,8000,03E8,0000,0000"
s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
s.sendto(MESSAGE, (UDP_IP, UDP_PORT))
But the Netty server shows :
31 36 2c 30 31 2c 31 32 2c 30 33 2c 31 62 2c 31
34 2c 33 30 2c 32 33 2c 38 30 30 30 2c 30 33 45
38 2c 30 30 30 30 2c 30 30 30 30
It looks like "ASCII-hex"...
What should I do to make me get "16,01,12,03,1b,14,30,23,8000,03E8,0000,0000"
?
Thanks!!

You will need to convert the ByteBuf to a String. That said you will need to account for fragmentation as well as TCP does not guarantee that all will be received in one packet. For this you most likely want to prefix your message with some field which will tell you how long the message is.
For this you can use LengthFieldBasedFrameDecoder in netty and then have it followed by a StringDecoder.

Set up AS3 NetConnection with custom Java server

The question is: How to correctly set up a connection between the client, written on ActionScript, and a server, written on Java, using the NetConnection on the client side for live-streaming?
When using sockets (Socket, XMLSocket, DatagramSocket) in ActionScript, we need to provide security policy checking/answering at the server side, this process is absolutely clear and works fine.
But in case when we need to stream live media, we have to use NetConnection class instead of sockets. And this is the place where the problems begin.
It looks like the NetConnection works in different way, providing a special mechanism to connect with it at the server side. The standard security policy checking, at the server side, shows that there is different data incoming from the NetConnection's instance from the client.
Here is the example of what we receive from the simple socket :
60 112 111 108 105 99 121 45 102 105 108 101 45 114 101 113 117 101 115 116 47 62
And below is what we receive from NetConnection instance (5 tries) :
3 0 17 63 -43 -128 0 7 2 -111 101 -96 -14 -116 31 -60 56 -68 -59 33 -65 -89
3 0 17 -70 -73 -128 0 7 2 53 -16 15 -100 38 81 -84 15 -46 -53 35 112 -83
3 0 18 11 -36 -128 0 7 2 117 -99 -103 83 25 29 -68 86 25 16 86 36 -28
3 0 23 -60 -67 -128 0 7 2 21 -73 80 -12 80 -83 -52 68 15 37 -72 -47 11
3 0 24 36 4 -128 0 7 2 -25 42 -12 81 121 20 -52 57 -89 115 -112 76 -89
I know, NetConnection works fine with Flash Media Server, as well as Red5 and other Java-implemented web servers. So it's possible to implement this connection for sure but...
How can it be done? Any of your ideas, suggestions, examples etc. will be helpful! Thanks.

h.264 udp stream decoding

I receive h.264 data throught UDP on Android. It looks, like a receive NAL units.
What I want to do, is to extract the
raw mpeg packets, that I can then push to my player. I use ijkplayer(build on ffmpeg) and it does not work if I push exactly the same data as I receive.
My question is: how can I extract raw h.264 packets. Is this stream wrapped in another protocol? Or is there possibility, that the stream is broken and some data is missing?
Data - shortened ... for brevity
0 0 0 1 97 0 -96 -101 -7 55 -117 1 82 -100 9 -107 -77 14 -20 -120 65 ...
0 0 0 1 97 0 80 38 -2 77 -1 57 -39 19 -20 86 -31 -47 11 122 -51 -82 ...
0 0 0 1 97 0 120 38 -2 77 -1 -118 7 -97 -80 -29 -2 -79 126 -25 47 3...
0 0 0 1 103 100 0 40 -84 -76 5 1 -19 -128 -86 64 0 0 3 0 -128 0 ...
0 0 0 1 104 -18 60 -80
0 0 0 1 97 0 -96 -102 25 51 -1 11 93 79 -78 94 -70 23 120 ...
0 0 0 1 97 0 80 38 -122 76 -1 16 80 -100 -84 -18 116 64 ...
0 0 0 1 97 0 120 38 -122 76 -1 26 -19 80 -103 -74 -31 44 ...
0 0 0 1 97 -102 57 51 -1 0 20 110 -37 37 -77 52 113 ...

0 0 0 1 is the h.264 start code. That is as "raw" as it gets. (there is no such think as an "mpeg packet")

I found on another forum that I need to wait for "0 0 0 1 9" which marks the beginning of Access unit and then I can feed the data to MediaCodec.

Clustering lists having maximum overlap with size restriction

I have the following group of numbers:
group1: 12 56 57 58 59 60 61 62 63 64 75 89 91 100 105 107 108 Group Size: 40
group2: 56 57 60 71 72 73 74 91 92 93 94 100 105 107 108 110 111 Group Size: 30
group3: 57 58 91 107 108 110 112 114 117 118 120 127 129 139 184 Group Size: 15
group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47 Group Size: 40
group5: 57 58 91 201 205 207 210 212 214 216 217 218 219 220 221 225 Group Size: 30
.
groupN: 50 51 52 53 54 210 214 216 219 225 700 701 702 705 706 708 Group Size: 40
Now I want to cluster together groups having maximum overlap such that after clustering, maximum size within a cluster does not exceed 90. For example here, the clusters are: (group1,group2,group3),(group5,groupN) and group4. The overlapping elements in the 3 groups are shown below:
Cluster1: (group1,group2,group3): 57 91 107 108 Cluster Size: (Group1_size+group2_size+group3_size =85 <90)
Cluster2: group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47 Cluster Size: (group4_size < 40)
Cluster3: (group5,groupN): 201 214 216 219 225 Cluster Size: (group5_size + groupN_size 70 <90)
If I include group5 in cluster1 then its size will be 85+30=115 and I want to return a size<90, therefore I can not include group4 in cluster1.
The elements in the respective clusters after removing the duplicate overlapping elements are:
Cluster1: (group1, group2, group3): 12 56 57 58 59 60 61 62 63 64 71 72 73 74 75 89 91 92 93 94 100 105 107 108 110 111 112 114 117 118 120 127 129 139 184
Cluster2: group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47
Cluster3: (group5,groupN): 50 51 52 53 54 57 58 91 201 205 207 210 212 214 216 217 218 219 220 221 225 700 701 702 705 706 708
Is there some existing algorithm or technique which may help me achieve this clustering with size constraint.
I tried to form clusters by finding the common elements between any two groups and including in the group if cluster size after inclusion is <90. But is there any existing algorithm in any of the programming language libraries like C++,python,java which may help me achieve this efficiently. If not, then is there any existing algorithm which achieves the same.
If possible, it will be great if the algorithm is optimal also.

There is no easy optimal solution. One approximation is as follows:
Pick the group with the largest size. Let its size be x
Pick the largest group such that its size is less than 90-x
Keep repeating step 2 until you cannot find such a group
Remove the selected groups and repeat the process starting from Step 1
Eg. You would pick group1 (or group4 or groupN) first is step 1. In step 2 you would pick group4. Now the size is 80 and there are no groups smaller than 90-80=10. So stop and remove these two groups. In the next iteration, you will select groupN, followed by group2, and at last group3. In the last iteration you have only one group, that is group5.

Java PBKDF2WithHmacSHA1 issue

I have three apps (.NET, Android, Java). The crypto involved consists of one of the three apps generating a message along with a key, IV, and salt to pass along to the other two. .Net <-> Android works perfectly for both asymmetric and symmetric encryption/decryption, and the key generating functions outputting the same values. The plain vanilla Java version is giving me problems.
The key, iv and salt are generated using a RNG such as
KeyGenerator kg = KeyGenerator.getInstance("AES");
SecureRandom sr = SecureRandom.getInstance("SHA1PRNG");
kg.init(len * 8, sr);
return kg.generateKey().getEncoded();
which are then converted to a hex string to be passed along. When I go to do the decryption on Android or Java, I convert the hex representation of the key back into a byte[] and pass it to the SecretKey function shown below.
public void SecretKey(byte[] key) {
char[] ca = null;
PBEKeySpec keySpec = null;
SecretKeyFactory factory = null;
_secretKey = null;
try {
ca = new char[key.length];
for (int x = 0; x < key.length; x++) {
ca[x] = (char)(key[x]);
}
keySpec = new PBEKeySpec(ca, _salt, 1024, 256);
factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
_secretKey = factory.generateSecret(keySpec);
.
.
.
Now, on the non-Android Java version I can verify that the password and salt are byte for byte identical as the .NET and Android version so there doesn't seem to be a conversion issue. However, the getEncoded() key output is different.
.Net
PW: 228 24 51 237 189 94 87 183 124 144 173 217 195 24 106 199 55 30 74 93 7 206 159 156 245 51 6 131 123 230 255 21
Salt: 87 241 248 158 129 101 47 36 255 31 30 26 211 50 204 156
Key: 204 26 176 226 255 40 25 163 60 85 75 208 230 192 214 150 136 5 155 55 228 199 98 36 230 84 210 6 164 113 48 128
Java:
PW: 228 24 51 237 189 94 87 183 124 144 173 217 195 24 106 199 55 30 74 93 7 206 159 156 245 51 6 131 123 230 255 21
Salt: 87 -15 -8 -98 -127 101 47 36 -1 31 30 26 -45 50 -52 -100
Key: -55 6 -4 80 96 9 85 18 72 -43 24 -48 -48 94 17 -113 74 108 -124 -118 -42 -29 -83 -88 -70 11 47 -4 4 -108 11 17
I'm using JDK1.7.0_45, have the non-restrictive policy files installed, along with BouncyCastle 1.50 set as the preferred provider using both policy and code paths.
On Android, I'm using SpongyCastle and .Net is just using the standard Microsoft CryptAPI.
Any help would be appreciated. Thanks.

The problem stems from the fact that PBKDF2 is designed to be used on passwords (text) not keys (random byte sequences that do not form valid encodings of text).
Following the recommendation in PKCS #5, the first step in key derivation should be conversion of the password text to bytes using UTF-8 encoding. Java does this, and with the particular key given in the post, the encoding is a 48-byte sequence. However, older versions of Android (and perhaps .NET) don't do a UTF-8 encoding; they simply discard all but the lower 8 bits of each character, giving you the 32-byte sequence provided in your post.
The newest Android versions, like Java, perform the character-to-byte conversion using UTF-8, and require use of a different key derivation algorithm to interoperate with old Android versions.
These problems can be avoided by not trying to use PBKDF2 here. It isn't improving security, it makes your code hard to write and read, and it introduces a number of interoperability pitfalls. All for no benefit.