We Keep Coding

LZString compressed data between PHP and Java

An Android app that I am writing acquires data compressed using LZString and sent out as base 64. I am using this implementation for LZString in Java along with this one in PHP. Both of these implementations are the top recomendations listed here by the author of the original JavaScript port of LZW.
I have had a really tough time understanding why the LZString data sent out by PHP throw up exceptions in Java. After much experiment and frayed nerves I have eventually worked out that the issue is down to apparent padding that is expected in Java and is missing in the data sent out from PHP. Take the following as examples
Original String being compressed
Betty bought a bit of butter but it was bitter so she bought some better butter to make the bitter butter better
a sentence I use for testing since with it multiple repetitions it is likely to compress well.
The PHP implementation of LZString spits out the following byte array
69 73 85 119 76 109 67 101 65 69 66 71 68 50 66 88 65 53 103 67 122 78 65
104 110 65 108 104 43 65 90 110 73 104 67 65 69 55 69 90 55 81 68 117 109 65
122 114 113 82 102 102 78 80 97 105 72 69 109 104 113 119 76 90 100 89 52 77
79 85 113 105 75 89 78 118 48 119 66 114 76 109 69 53 77 74 52 115 99 79 90
65
while the Java implementation generates the following byte array
69 73 85 119 76 109 67 101 65 69 66 71 68 50 66 88 65 53 103 67 122 78 65
104 110 65 108 104 43 65 90 110 73 104 67 65 69 55 69 90 55 81 68 117 109 65
122 114 113 82 102 102 78 80 97 105 72 69 109 104 113 119 76 90 100 89 52 77
79 85 113 105 75 89 78 118 48 119 66 114 76 109 69 53 77 74 52 115 99 79 90
65 **65 65 61 61**
You will note that the Java implementation tags on extra **AA==**.
I can at a pinch understand why there is an == - padding to get to the desired length multiple. However, I cannot understand why or where the AA are coming from.
I tested LZString.decompressFromBase64 in Java after tagging on an additional AA== and found that it works. On the other hand simply tagging on an == threw an exception. Further experiment revealed that tagging on ==== worked and so too did BB== indicating that these four bytes are simply used for padding and not put to any other use.
At this point I could quite simply append padding as appropriate in Java prior to doing LZString.decompressFromBase64. However, that I fear that would be a "solution" implemented without a full understanding of what is happening here. Perhaps someone here can shed some light?

Having issues with my ISO8583 packager while connecting to jpos channel

After packing my data and while trying to send to JPOS channel (server), i do receive the below error.
Length = 0030 Byte length(b): 48 :: Incoming data HEX(d):
3830300238000000C2820000303030303130303732323137313934363030303030363030303231383030303631373139
org.jpos.iso.IFA_LLNUM: Problem unpacking field 33 (java.lang.ArrayIndexOutOfBoundsException: 48) unpacking field=33,
consumed=42
org.jpos.iso.ISOException: org.jpos.iso.IFA_LLNUM: Problem unpacking field 33 (java.lang.ArrayIndexOutOfBoundsException: 48)
unpacking field=33, consumed=42
at org.jpos.iso.ISOBasePackager.unpack(ISOBasePackager.java:273)
at org.jpos.iso.ISOMsg.unpack(ISOMsg.java:416)
at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:903)
at org.jpos.iso.BaseChannel.receive(BaseChannel.java:671)
at org.jpos.iso.ISOServer$Session.run(ISOServer.java:130)
at org.jpos.util.ThreadPool$PooledThread.run(ThreadPool.java:71)
--- data ---
0000 38 30 30 02 38 00 00 00 C2 82 00 00 30 30 30 30 800.8.......0000 0010 31 30 30 37 32 32 31 37 31 39 34 36 30 30 30 30 1007221719460000 0020 30 36 30 30 30 32 31 38 30 30 30 36 31 37
31 39 0600021800061719
org.jpos.iso.IFA_LLNUM: Problem unpacking field 33 (java.lang.ArrayIndexOutOfBoundsException: 48) unpacking field=33,
consumed=42
org.jpos.iso.ISOException: org.jpos.iso.IFA_LLNUM: Problem unpacking field 33 (java.lang.ArrayIndexOutOfBoundsException: 48)
unpacking field=33, consumed=42
at org.jpos.iso.ISOBasePackager.unpack(ISOBasePackager.java:273)
at org.jpos.iso.ISOMsg.unpack(ISOMsg.java:416)
at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:903)
at org.jpos.iso.BaseChannel.receive(BaseChannel.java:671)
at org.jpos.iso.ISOServer$Session.run(ISOServer.java:130)
at org.jpos.util.ThreadPool$PooledThread.run(ThreadPool.java:71)
And, i am using the below java class to transport my packed data.
public static String networkTransport(String isoMessage) throws UnknownHostException, IOException {
Socket connection = new Socket("192.168.3.118", 1010);
BufferedOutputStream bos = new BufferedOutputStream(connection.getOutputStream());
OutputStreamWriter osw = new OutputStreamWriter(bos);
int len = isoMessage.length(); // get the length of the data
// SInce your packager name says Postilion, I think this will work.
osw.write(len >> 8); // send the length bytes in 2 bytes. this is the byte 1
// osw.write(len);// send the length bytes in 2 bytes. this is the byte 2
osw.write(isoMessage);
osw.flush();
byte[] arrOutut = new byte[4096];
int count = connection.getInputStream().read(arrOutut, 0, 4096);
String clientRequest = "";
for (int outputCount = 0; outputCount < count; outputCount++) {
char response = (char) arrOutut[outputCount];
clientRequest = clientRequest + response;
}
connection.close();
return clientRequest;
}
The challenge i am currently facing is how I can have a smooth flow with my JPOS channel.
All suggestions are highly welcomed.

Belew is how I would split your data.
383030 //echo message type as you said 0800.
But where is the starting 0 (0x30) ?
0238000000C28200 //bitmap 8 bytes - packed BCD
00303030303130303732323137313934363030303030363030303231383030303631373139 - data
Below are the bits you have turned on. Can you verify whether you have all the field data for the below turned on bits ? I don't understand why you need DE55 in an echo message.
0 0000
2 0010 7
3 0011 11, 12
8 1000 13
0 0000
0 0000
0 0000
0 0000
0 0000
0 0000
C 1100 41, 42
2 0011 47, 48
8 1000 49
2 0011 55, 56
0 0000
0 0000
On an assumption, I would split your data like below:
00 30 30 30 30 31 30 30 37 32 - transmission date mmddhhmmss
32 31 37 31 39 - trace number
34 36 30 30 30 30 - local time
30 36 30 30 - local date
30 32 31 38 30 30 30 36 - terminal id
31 37 31 39 - this is all the remaining data for bits 42, 47, 48, 49,
55 and 56.
So getting a null pointer is quite obvious.

I was able to resolve this issue, while making use of the JPOS library, but had to strimline it to using just the things i will be needing at my own end.
If you may want to use this method on your android device, these are the folders i actually used
Channel
Filter
Gui
Header
Packager
Validator and the whole java class here
or better still, use all the files and folders here

In packaging data for jpos server you have to check two details:
1) jpos server channel type (leading or trailing data)
2) jpos server packager
Please note that jpos server is not expecting raw stream data from clients.
On jpos.org site you can find very good written jpos manual.

h.264 udp stream decoding

I receive h.264 data throught UDP on Android. It looks, like a receive NAL units.
What I want to do, is to extract the
raw mpeg packets, that I can then push to my player. I use ijkplayer(build on ffmpeg) and it does not work if I push exactly the same data as I receive.
My question is: how can I extract raw h.264 packets. Is this stream wrapped in another protocol? Or is there possibility, that the stream is broken and some data is missing?
Data - shortened ... for brevity
0 0 0 1 97 0 -96 -101 -7 55 -117 1 82 -100 9 -107 -77 14 -20 -120 65 ...
0 0 0 1 97 0 80 38 -2 77 -1 57 -39 19 -20 86 -31 -47 11 122 -51 -82 ...
0 0 0 1 97 0 120 38 -2 77 -1 -118 7 -97 -80 -29 -2 -79 126 -25 47 3...
0 0 0 1 103 100 0 40 -84 -76 5 1 -19 -128 -86 64 0 0 3 0 -128 0 ...
0 0 0 1 104 -18 60 -80
0 0 0 1 97 0 -96 -102 25 51 -1 11 93 79 -78 94 -70 23 120 ...
0 0 0 1 97 0 80 38 -122 76 -1 16 80 -100 -84 -18 116 64 ...
0 0 0 1 97 0 120 38 -122 76 -1 26 -19 80 -103 -74 -31 44 ...
0 0 0 1 97 -102 57 51 -1 0 20 110 -37 37 -77 52 113 ...

0 0 0 1 is the h.264 start code. That is as "raw" as it gets. (there is no such think as an "mpeg packet")

I found on another forum that I need to wait for "0 0 0 1 9" which marks the beginning of Access unit and then I can feed the data to MediaCodec.

Clustering lists having maximum overlap with size restriction

I have the following group of numbers:
group1: 12 56 57 58 59 60 61 62 63 64 75 89 91 100 105 107 108 Group Size: 40
group2: 56 57 60 71 72 73 74 91 92 93 94 100 105 107 108 110 111 Group Size: 30
group3: 57 58 91 107 108 110 112 114 117 118 120 127 129 139 184 Group Size: 15
group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47 Group Size: 40
group5: 57 58 91 201 205 207 210 212 214 216 217 218 219 220 221 225 Group Size: 30
.
groupN: 50 51 52 53 54 210 214 216 219 225 700 701 702 705 706 708 Group Size: 40
Now I want to cluster together groups having maximum overlap such that after clustering, maximum size within a cluster does not exceed 90. For example here, the clusters are: (group1,group2,group3),(group5,groupN) and group4. The overlapping elements in the 3 groups are shown below:
Cluster1: (group1,group2,group3): 57 91 107 108 Cluster Size: (Group1_size+group2_size+group3_size =85 <90)
Cluster2: group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47 Cluster Size: (group4_size < 40)
Cluster3: (group5,groupN): 201 214 216 219 225 Cluster Size: (group5_size + groupN_size 70 <90)
If I include group5 in cluster1 then its size will be 85+30=115 and I want to return a size<90, therefore I can not include group4 in cluster1.
The elements in the respective clusters after removing the duplicate overlapping elements are:
Cluster1: (group1, group2, group3): 12 56 57 58 59 60 61 62 63 64 71 72 73 74 75 89 91 92 93 94 100 105 107 108 110 111 112 114 117 118 120 127 129 139 184
Cluster2: group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47
Cluster3: (group5,groupN): 50 51 52 53 54 57 58 91 201 205 207 210 212 214 216 217 218 219 220 221 225 700 701 702 705 706 708
Is there some existing algorithm or technique which may help me achieve this clustering with size constraint.
I tried to form clusters by finding the common elements between any two groups and including in the group if cluster size after inclusion is <90. But is there any existing algorithm in any of the programming language libraries like C++,python,java which may help me achieve this efficiently. If not, then is there any existing algorithm which achieves the same.
If possible, it will be great if the algorithm is optimal also.

There is no easy optimal solution. One approximation is as follows:
Pick the group with the largest size. Let its size be x
Pick the largest group such that its size is less than 90-x
Keep repeating step 2 until you cannot find such a group
Remove the selected groups and repeat the process starting from Step 1
Eg. You would pick group1 (or group4 or groupN) first is step 1. In step 2 you would pick group4. Now the size is 80 and there are no groups smaller than 90-80=10. So stop and remove these two groups. In the next iteration, you will select groupN, followed by group2, and at last group3. In the last iteration you have only one group, that is group5.

Java PBKDF2WithHmacSHA1 issue

I have three apps (.NET, Android, Java). The crypto involved consists of one of the three apps generating a message along with a key, IV, and salt to pass along to the other two. .Net <-> Android works perfectly for both asymmetric and symmetric encryption/decryption, and the key generating functions outputting the same values. The plain vanilla Java version is giving me problems.
The key, iv and salt are generated using a RNG such as
KeyGenerator kg = KeyGenerator.getInstance("AES");
SecureRandom sr = SecureRandom.getInstance("SHA1PRNG");
kg.init(len * 8, sr);
return kg.generateKey().getEncoded();
which are then converted to a hex string to be passed along. When I go to do the decryption on Android or Java, I convert the hex representation of the key back into a byte[] and pass it to the SecretKey function shown below.
public void SecretKey(byte[] key) {
char[] ca = null;
PBEKeySpec keySpec = null;
SecretKeyFactory factory = null;
_secretKey = null;
try {
ca = new char[key.length];
for (int x = 0; x < key.length; x++) {
ca[x] = (char)(key[x]);
}
keySpec = new PBEKeySpec(ca, _salt, 1024, 256);
factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
_secretKey = factory.generateSecret(keySpec);
.
.
.
Now, on the non-Android Java version I can verify that the password and salt are byte for byte identical as the .NET and Android version so there doesn't seem to be a conversion issue. However, the getEncoded() key output is different.
.Net
PW: 228 24 51 237 189 94 87 183 124 144 173 217 195 24 106 199 55 30 74 93 7 206 159 156 245 51 6 131 123 230 255 21
Salt: 87 241 248 158 129 101 47 36 255 31 30 26 211 50 204 156
Key: 204 26 176 226 255 40 25 163 60 85 75 208 230 192 214 150 136 5 155 55 228 199 98 36 230 84 210 6 164 113 48 128
Java:
PW: 228 24 51 237 189 94 87 183 124 144 173 217 195 24 106 199 55 30 74 93 7 206 159 156 245 51 6 131 123 230 255 21
Salt: 87 -15 -8 -98 -127 101 47 36 -1 31 30 26 -45 50 -52 -100
Key: -55 6 -4 80 96 9 85 18 72 -43 24 -48 -48 94 17 -113 74 108 -124 -118 -42 -29 -83 -88 -70 11 47 -4 4 -108 11 17
I'm using JDK1.7.0_45, have the non-restrictive policy files installed, along with BouncyCastle 1.50 set as the preferred provider using both policy and code paths.
On Android, I'm using SpongyCastle and .Net is just using the standard Microsoft CryptAPI.
Any help would be appreciated. Thanks.

The problem stems from the fact that PBKDF2 is designed to be used on passwords (text) not keys (random byte sequences that do not form valid encodings of text).
Following the recommendation in PKCS #5, the first step in key derivation should be conversion of the password text to bytes using UTF-8 encoding. Java does this, and with the particular key given in the post, the encoding is a 48-byte sequence. However, older versions of Android (and perhaps .NET) don't do a UTF-8 encoding; they simply discard all but the lower 8 bits of each character, giving you the 32-byte sequence provided in your post.
The newest Android versions, like Java, perform the character-to-byte conversion using UTF-8, and require use of a different key derivation algorithm to interoperate with old Android versions.
These problems can be avoided by not trying to use PBKDF2 here. It isn't improving security, it makes your code hard to write and read, and it introduces a number of interoperability pitfalls. All for no benefit.