An Android app that I am writing acquires data compressed using LZString and sent out as base 64. I am using this implementation for LZString in Java along with this one in PHP. Both of these implementations are the top recomendations listed here by the author of the original JavaScript port of LZW.
I have had a really tough time understanding why the LZString data sent out by PHP throw up exceptions in Java. After much experiment and frayed nerves I have eventually worked out that the issue is down to apparent padding that is expected in Java and is missing in the data sent out from PHP. Take the following as examples
Original String being compressed
Betty bought a bit of butter but it was bitter so she bought some better butter to make the bitter butter better
a sentence I use for testing since with it multiple repetitions it is likely to compress well.
The PHP implementation of LZString spits out the following byte array
69 73 85 119 76 109 67 101 65 69 66 71 68 50 66 88 65 53 103 67 122 78 65
104 110 65 108 104 43 65 90 110 73 104 67 65 69 55 69 90 55 81 68 117 109 65
122 114 113 82 102 102 78 80 97 105 72 69 109 104 113 119 76 90 100 89 52 77
79 85 113 105 75 89 78 118 48 119 66 114 76 109 69 53 77 74 52 115 99 79 90
65
while the Java implementation generates the following byte array
69 73 85 119 76 109 67 101 65 69 66 71 68 50 66 88 65 53 103 67 122 78 65
104 110 65 108 104 43 65 90 110 73 104 67 65 69 55 69 90 55 81 68 117 109 65
122 114 113 82 102 102 78 80 97 105 72 69 109 104 113 119 76 90 100 89 52 77
79 85 113 105 75 89 78 118 48 119 66 114 76 109 69 53 77 74 52 115 99 79 90
65 **65 65 61 61**
You will note that the Java implementation tags on extra **AA==**.
I can at a pinch understand why there is an == - padding to get to the desired length multiple. However, I cannot understand why or where the AA are coming from.
I tested LZString.decompressFromBase64 in Java after tagging on an additional AA== and found that it works. On the other hand simply tagging on an == threw an exception. Further experiment revealed that tagging on ==== worked and so too did BB== indicating that these four bytes are simply used for padding and not put to any other use.
At this point I could quite simply append padding as appropriate in Java prior to doing LZString.decompressFromBase64. However, that I fear that would be a "solution" implemented without a full understanding of what is happening here. Perhaps someone here can shed some light?
Related
This is a problem for us using vlingo-http backed by vlingo-wire, which was making direct use of Java NIO. We thought we were facing the infamous Netty Java NIO epoll bug that was fixed by Netty, so we applied Netty's fix to vlingo-wire. This didn't work. We finally gave up trying to solve it with direct NIO and even with incorporating Netty's fix. Thus, we switched to Netty in place of NIO, which works everywhere else, but not on Heroku.
Using Netty and some fine-grained logging we saw that for each request the server definitely replies. But for some reason the reply does not get past the Heroku router. For example:
2020-04-06T20:50:13.896574+00:00 app[web.1]: 20:50:13.896 [nioEventLoopGroup-3-4] DEBUG i.v.w.f.b.n.s.NettyInboundHandler - Request received
2020-04-06T20:50:13.896929+00:00 app[web.1]: 20:50:13.896 [nioEventLoopGroup-3-4] DEBUG i.n.handler.logging.LoggingHandler - [id: 0x1efddaa2, L:/172.18.27.14:59829 - R:/10.11.43.12:20716] READ COMPLETE
2020-04-06T20:50:13.897026+00:00 app[web.1]: Retrieving operations and...
2020-04-06T20:50:13.898394+00:00 app[web.1]: 20:50:13.898 [nioEventLoopGroup-3-4] DEBUG i.n.handler.logging.LoggingHandler - [id: 0x1efddaa2, L:/172.18.27.14:59829 - R:/10.11.43.12:20716] WRITE: 148B
2020-04-06T20:50:13.898395+00:00 app[web.1]: +-------------------------------------------------+
2020-04-06T20:50:13.898396+00:00 app[web.1]: | 0 1 2 3 4 5 6 7 8 9 a b c d e f |
2020-04-06T20:50:13.898397+00:00 app[web.1]: +--------+-------------------------------------------------+----------------+
2020-04-06T20:50:13.898397+00:00 app[web.1]: |00000000| 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0a |HTTP/1.1 200 OK.|
2020-04-06T20:50:13.898397+00:00 app[web.1]: |00000010| 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 61 70 |Content-Type: ap|
2020-04-06T20:50:13.898399+00:00 app[web.1]: |00000020| 70 6c 69 63 61 74 69 6f 6e 2f 76 6e 64 2e 63 61 |plication/vnd.ca|
2020-04-06T20:50:13.898400+00:00 app[web.1]: |00000030| 6c 63 75 6c 61 74 69 6f 6e 2b 6a 73 6f 6e 3b 76 |lculation+json;v|
2020-04-06T20:50:13.898400+00:00 app[web.1]: |00000040| 65 72 73 69 6f 6e 3d 31 2e 31 0a 43 6f 6e 74 65 |ersion=1.1.Conte|
2020-04-06T20:50:13.898400+00:00 app[web.1]: |00000050| 6e 74 2d 4c 65 6e 67 74 68 3a 20 35 33 0a 0a 5b |nt-Length: 53..[|
2020-04-06T20:50:13.898401+00:00 app[web.1]: |00000060| 0a 20 20 22 41 44 44 49 54 49 4f 4e 22 2c 0a 20 |. "ADDITION",. |
2020-04-06T20:50:13.898401+00:00 app[web.1]: |00000070| 20 22 53 55 42 54 52 41 43 54 49 4f 4e 22 2c 0a | "SUBTRACTION",.|
2020-04-06T20:50:13.898401+00:00 app[web.1]: |00000080| 20 20 22 4d 55 4c 54 49 50 4c 49 43 41 54 49 4f | "MULTIPLICATIO|
2020-04-06T20:50:13.898402+00:00 app[web.1]: |00000090| 4e 22 0a 5d |N".] |
2020-04-06T20:50:13.898402+00:00 app[web.1]: +--------+-------------------------------------------------+----------------+
2020-04-06T20:50:13.898467+00:00 app[web.1]: 20:50:13.898 [nioEventLoopGroup-3-4] DEBUG i.n.handler.logging.LoggingHandler - [id: 0x1efddaa2, L:/172.18.27.14:59829 - R:/10.11.43.12:20716] FLUSH
2020-04-06T20:50:13.898847+00:00 app[web.1]: 20:50:13.898 [nioEventLoopGroup-3-4] TRACE i.v.w.f.b.n.s.NettyInboundHandler - Reply sent
The conclusion was that perhaps we were not closing the client socket properly (even though using curl has the same problems). In any case we tried server socket eager closing to close the socket right after a response to an HTTP request. The assumption was that maybe the close would flush socket buffers (small responses). This also didn't work for us.
All of the above worked from the beginning on direct AWS use, and continues to work there.
We are currently looking into some ideas we got about timeouts and DNS configuration, but so far it's not panning out.
At this point we are stuck and don't know what to do or try next. We would sure like to support Heroku. If you have any clues to share we would appreciate it very much.
I want Netty (Reactor Netty, to be specific) to set the Content-Length header in my requests. Currently, even when I send a request body in my request, the Content-Length header is not set. Is it possible to configure Netty to set this automatically (set it to 0 if there is no request body)? Postman does this.
Thanks!
I can answer this question for netty.... In netty this is not done automatically and the only way how to do it is to either set it your self or add a ChannelOutboundHandler that does it depending on the FullHttpMessage it receives.
It depends on the content that you want to send. If it is of type Mono, then we will calculate the content length and send a FullHttpMessage. If it is of type Flux, we will consider this as a chunked content and thus we will not calculate the content length.
Here is an example with Mono:
public static void main(String[] args) {
String response =
HttpClient.create()
.wiretap(true)
.post()
.uri("https://postman-echo.com/post")
.send(Mono.just(Unpooled.wrappedBuffer("something".getBytes(Charset.defaultCharset()))))
.responseContent()
.aggregate()
.asString()
.block();
System.out.println(response);
}
In the logs you should be able to see this:
17:01:46.813 [reactor-http-nio-4] DEBUG reactor.netty.http.client.HttpClient - [id: 0x668bd78f, L:/xxx:xxx - R:postman-echo.com/34.239.20.132:443] WRITE: 118B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 50 4f 53 54 20 2f 70 6f 73 74 20 48 54 54 50 2f |POST /post HTTP/|
|00000010| 31 2e 31 0d 0a 75 73 65 72 2d 61 67 65 6e 74 3a |1.1..user-agent:|
|00000020| 20 52 65 61 63 74 6f 72 4e 65 74 74 79 2f 64 65 | ReactorNetty/de|
|00000030| 76 0d 0a 68 6f 73 74 3a 20 70 6f 73 74 6d 61 6e |v..host: postman|
|00000040| 2d 65 63 68 6f 2e 63 6f 6d 0d 0a 61 63 63 65 70 |-echo.com..accep|
|00000050| 74 3a 20 2a 2f 2a 0d 0a 63 6f 6e 74 65 6e 74 2d |t: */*..content-|
|00000060| 6c 65 6e 67 74 68 3a 20 39 0d 0a 0d 0a 73 6f 6d |length: 9....som|
|00000070| 65 74 68 69 6e 67 |ething |
+--------+-------------------------------------------------+----------------+
The question is: How to correctly set up a connection between the client, written on ActionScript, and a server, written on Java, using the NetConnection on the client side for live-streaming?
When using sockets (Socket, XMLSocket, DatagramSocket) in ActionScript, we need to provide security policy checking/answering at the server side, this process is absolutely clear and works fine.
But in case when we need to stream live media, we have to use NetConnection class instead of sockets. And this is the place where the problems begin.
It looks like the NetConnection works in different way, providing a special mechanism to connect with it at the server side. The standard security policy checking, at the server side, shows that there is different data incoming from the NetConnection's instance from the client.
Here is the example of what we receive from the simple socket :
60 112 111 108 105 99 121 45 102 105 108 101 45 114 101 113 117 101 115 116 47 62
And below is what we receive from NetConnection instance (5 tries) :
3 0 17 63 -43 -128 0 7 2 -111 101 -96 -14 -116 31 -60 56 -68 -59 33 -65 -89
3 0 17 -70 -73 -128 0 7 2 53 -16 15 -100 38 81 -84 15 -46 -53 35 112 -83
3 0 18 11 -36 -128 0 7 2 117 -99 -103 83 25 29 -68 86 25 16 86 36 -28
3 0 23 -60 -67 -128 0 7 2 21 -73 80 -12 80 -83 -52 68 15 37 -72 -47 11
3 0 24 36 4 -128 0 7 2 -25 42 -12 81 121 20 -52 57 -89 115 -112 76 -89
I know, NetConnection works fine with Flash Media Server, as well as Red5 and other Java-implemented web servers. So it's possible to implement this connection for sure but...
How can it be done? Any of your ideas, suggestions, examples etc. will be helpful! Thanks.
I have the following group of numbers:
group1: 12 56 57 58 59 60 61 62 63 64 75 89 91 100 105 107 108 Group Size: 40
group2: 56 57 60 71 72 73 74 91 92 93 94 100 105 107 108 110 111 Group Size: 30
group3: 57 58 91 107 108 110 112 114 117 118 120 127 129 139 184 Group Size: 15
group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47 Group Size: 40
group5: 57 58 91 201 205 207 210 212 214 216 217 218 219 220 221 225 Group Size: 30
.
groupN: 50 51 52 53 54 210 214 216 219 225 700 701 702 705 706 708 Group Size: 40
Now I want to cluster together groups having maximum overlap such that after clustering, maximum size within a cluster does not exceed 90. For example here, the clusters are: (group1,group2,group3),(group5,groupN) and group4. The overlapping elements in the 3 groups are shown below:
Cluster1: (group1,group2,group3): 57 91 107 108 Cluster Size: (Group1_size+group2_size+group3_size =85 <90)
Cluster2: group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47 Cluster Size: (group4_size < 40)
Cluster3: (group5,groupN): 201 214 216 219 225 Cluster Size: (group5_size + groupN_size 70 <90)
If I include group5 in cluster1 then its size will be 85+30=115 and I want to return a size<90, therefore I can not include group4 in cluster1.
The elements in the respective clusters after removing the duplicate overlapping elements are:
Cluster1: (group1, group2, group3): 12 56 57 58 59 60 61 62 63 64 71 72 73 74 75 89 91 92 93 94 100 105 107 108 110 111 112 114 117 118 120 127 129 139 184
Cluster2: group4: 1 2 4 6 7 8 9 10 17 18 20 41 42 43 45 47
Cluster3: (group5,groupN): 50 51 52 53 54 57 58 91 201 205 207 210 212 214 216 217 218 219 220 221 225 700 701 702 705 706 708
Is there some existing algorithm or technique which may help me achieve this clustering with size constraint.
I tried to form clusters by finding the common elements between any two groups and including in the group if cluster size after inclusion is <90. But is there any existing algorithm in any of the programming language libraries like C++,python,java which may help me achieve this efficiently. If not, then is there any existing algorithm which achieves the same.
If possible, it will be great if the algorithm is optimal also.
There is no easy optimal solution. One approximation is as follows:
Pick the group with the largest size. Let its size be x
Pick the largest group such that its size is less than 90-x
Keep repeating step 2 until you cannot find such a group
Remove the selected groups and repeat the process starting from Step 1
Eg. You would pick group1 (or group4 or groupN) first is step 1. In step 2 you would pick group4. Now the size is 80 and there are no groups smaller than 90-80=10. So stop and remove these two groups. In the next iteration, you will select groupN, followed by group2, and at last group3. In the last iteration you have only one group, that is group5.
I have a java card. How I can find the algorithms that the card support and use for it's cryptography? What happens when I click on the authenticate button in jcmanager (for example)? How I can authenticate with APDUs?
This is output of jcmanager when I clicked on Authenticate :
Open terminal ...
EstablishContext(): ...
Wait for card in a certain reader ...
Pick reader ...
**********************
Selecting Card Manager
***********************
-> 00 A4 04 00 08 A0 00 00 00 03 00 00 00
<- 6F 10 84 08 A0 00 00 00 03 00 00 00 A5 04 9F 65 01 FF 90 00
************
Init Update
*************
-> 80 50 00 00 08 D3 90 22 B2 C5 7C D4 DD
<- 00 00 11 60 01 00 7F 8B 0A F9 02 02 00 99 3E 01 33 1B 3F 8E 33 BA E4 AD 82 6E 3C C1 90 00
HostChallenge: D3 90 22 B2 C5 7C D4 DD
CardChallenge: 3E 01 33 1B 3F 8E
Card Calculated Card Cryptogram: 33 BA E4 AD 82 6E 3C C1
Derivation Data is 01 82 00 99 00 00 00 00 00 00 00 00 00 00 00 00
Host Cryptogram Data (to encrypt) 00 99 3E 01 33 1B 3F 8E D3 90 22 B2 C5 7C D4 DD 80 00 00 00 00 00 00 00
Card Cryptogram Data (to encrypt for verification) D3 90 22 B2 C5 7C D4 DD 00 99 3E 01 33 1B 3F 8E 80 00 00 00 00 00 00 00
S_ENC: CE 69 1B 1E C8 EC DB B0 0A 9B 18 4A 53 58 04 BB CE 69 1B 1E C8 EC DB B0
The Current session MAC key is F8 85 4D 94 19 BC 83 4C 99 BA E9 94 15 00 A6 B8
The Current session DEK key is 6D 72 48 D4 23 BF 3B 1C 7C 2F 1F BC 7C 04 E9 F6
Encrypted CardCryptoGram is 58 20 23 4E 14 8B FE AA F8 6D 14 20 3D 41 18 E4 33 BA E4 AD 82 6E 3C C1
Encrypted HostCryptoGram is 07 D0 B3 EB 0F 1B 7E 54 84 34 08 6C 5F D9 E5 55 4B 5F 0D F6 87 52 99 2E
-> 84 82 03 00 10 4B 5F 0D F6 87 52 99 2E 17 29 AA 68 12 98 CE 2D
<- 90 00
Authenticated
Is this right :
I send a random 8 byte number to the card:
-> 80 50 00 00 08 D3 90 22 B2 C5 7C D4 DD
and in the answer of my command, card give me a random numbers + the encrypted random number that I sent to the card. What happens next? my card and my reader with which algorithm encrypt and decrypt random numbers (cardchallenge and host challenge)?
you are totally mixing up things.
FIRST there is Global Platform. Global Platform specifies how to interact with the card manager/security domain: e.g. how to authenticate with the card manager, how to upload a cap file, how to install a cap file, to lock the card to unlock the card to change the keys you mentioned above etc...
SECOND there is the Java Card applet which you code in a way you want your smartcard to behave lateron.
your question is very unclear.
do you mean cryptographic algorithms regarding global platform or jor java card code?
for first identiy your card and read the corresponding Global Platform Specification, there are various authentication modes called scp01 scp02 ...
if you mean javacard: the JC Specification is not mandatory so its the manufactureres choice what crypto to implement. to find out you can either trial&error or you talk to the manufacturer
for authentication to the card manager you should always use a tool (either GPJ GPshell or JCOP tools<--last one is not freely available) generating the APDUs and crypto behind it instead of manually sending APDUs.
however, if you want to know the details read Global Platform Specification instead of asking on a forum
The algorithms supported by your particular card can be obtained via JCAlgTest project. JCAlgTest project page also provides results for 63+ cards gathered by other users. Use ATR of your card to search for similar one in JCAlgTest database. Alternatively, use ATR parsing tool and find match your card by name. The JCAlgTest project also provides performance comparison for selected cards.