I have some major problem detecting the true end of a http response within Socket (I have to use sockets as it was requested). We're communicating with a webservice that sends chuncked responses. I have no problems reading a response if it returns in one piece. However when it's being split all hell breaks loose :).
For example:
UserA -> RequestA -> Response[1] -> Processed
UserA -> RequestA -> Response[1] -> Processed
UserB -> RequestB -> a)Response[0,1] -> Processed[a.0]
UserB -> RequestB -> b)Response[0,1] -> Processed[a.1] <- From the previous line. And thus the true response to request B have to be processed again.
What is the preferred way to handle this kind of situation? Btw, the WS also returns the Content-Length header attribute, but honestly I have a head-ache from handling that. For that it seems like I have to read the headers fields to a ByteArrayOutputStream and check if it contains the Content-Length information. Then retrieve the actual length and wait until the is.available() reaches this value. Since the available method returns an estimation I do not trust it. So what would be the proper way?
The preferred way is to use existing code that already handles it: for example, HttpURLConnection or the Apache HTTP Client. There's no good reason for you to be reinventing the wheel.
The correct answer should be:
private static byte[] convert(final InputStream is) throws IOException {
final byte[] END_SIG = new byte[]{"\r".getBytes()[0], "\n".getBytes()[0]};
final List<Byte> streamBytes = new ArrayList<Byte>();
int readByte;
byte[] bytes;
// Read HTTP header:
while ((readByte = is.read()) != -1) {
streamBytes.add((byte) readByte);
if (streamBytes.size() > 4) {
int sbsize = streamBytes.size();
int rp = sbsize - 4;
int np = sbsize - 2;
int rn = sbsize - 3;
int nn = sbsize - 1;
if (END_SIG[0] == streamBytes.get(rp) && END_SIG[0] == streamBytes.get(np) && END_SIG[1] == streamBytes.get(rn) && END_SIG[1] == streamBytes.get(nn)) {
break;
}
}
}
// Convert to byte[]
bytes = new byte[streamBytes.size()];
for (int i = 0, iMAX = bytes.length; i < iMAX; ++i) {
bytes[i] = streamBytes.get(i);
}
// drop header
streamBytes.clear();
// Convert byte[] to String & retrieve the content-length:
final String HEADER = new String(bytes);
int startIndex = HEADER.indexOf("Content-Length:") + "Content-Length:".length() + 1;
int length = 0;
int I = startIndex;
while (Character.isDigit(HEADER.charAt(I++))) {
++length;
}
final String CL = HEADER.substring(startIndex, startIndex + length);
// Determine the number of bytes to read from now on:
int ContentLength = Integer.parseInt(CL);
while (streamBytes.size() < ContentLength) {
byte[] buffer = new byte[256];
int rc = is.read(buffer);
for (int irc = 0; irc < rc; ++irc) {
streamBytes.add(buffer[irc]);
}
}
// Convert to byte[]
bytes = new byte[streamBytes.size()];
for (int i = 0, iMAX = bytes.length; i < iMAX; ++i) {
bytes[i] = streamBytes.get(i);
}
return bytes;
}
And in one place this is the answer to a question.
Related
So I've been trying to read the content of a text file and write the content chunk by chunk alternately into e.g. 2 new files.
I already tried multiple ways to do that but it won't work (OutputStream and FileOutputStream seems to be the most suitable).
Before i tried to part the file in e.g. 3 Parts and wrote the first part in one file, the second part in another and so on. Which worked perfectly fine with OutputStream and FileOutputStream.
But it won't work when i want to do it alternately.
To do it alternately i use the round robin algorithm, which on its own works fine.
I would be really thankful if you could show me some examples to do it!
public void splitFile(String filePath, int numberOfParts, long sizeOfParts[]) throws FileNotFoundException, IOException, SQLException {
long bytes = 8;
OutputStream partsPath[] = new OutputStream[numberOfParts];
long bytePositition[] = new long[numberOfParts];
long copy_size[] = new long[numberOfParts];
for (int i = 0; i < numberOfParts; i++) {
copy_size[i] = sizeOfParts[i];
partsPath[i] = new FileOutputStream(path); //Gets Path from my Database (works)
//System.out.println(cloudsTable.getCloudsPathsFromDatabase(i) + '\\' + name + (i + 1) + fileType);
}
InputStream file = new FileInputStream(filePath);
while (true) {
boolean done = true;
for (int i = 0; i < numberOfParts; i++) {
if (copy_size[i] > 0) {
done = false;
if (copy_size[i] > bytes) {
copy_size[i] -= bytes;
bytePositition[i] += bytes;
System.out.println("file " + i + " " + bytePositition[i]);
readWrite(file, bytePositition[i], partsPath[i]);
} else {
bytePositition[i] += copy_size[i];
System.out.println("rest file " + i + " " + bytePositition[i]);
readWrite(file, bytePositition[i], partsPath[i]);
copy_size[i] = 0;
}
}
}
if (done == true) {
break;
}
}
file.close();
for (int i = 0; i < partsPath.length; i++) {
partsPath[i].close();
}
}
private void readWrite(InputStream file, long bytes, OutputStream path) throws IOException {
byte[] buf = new byte[(int) bytes];
while (file.read(buf) != -1) {
path.write(buf);
path.flush();
}
}
What the code does is, it only write the content of the Originalfile in the first-copied file and the following files are empty
EDIT:
To clarify what the code should do is write the first 8 bytes to go to file 1, second 8 bytes to go to file 2, third 8 bytes to go to file 3, fourth 8 bytes to go to file 1, and so on, round robin, until file 1 is sizeOfParts[0] long, file 2 is sizeOfParts[1] long, and file 3 is sizeOfParts[2] long.
The main problem is that the readWrite() method is only supposed to copy one 8-byte block of bytes, but has a loop that makes it copy all the remaining bytes in the input file.
In addition, the code should be enhanced to use try-finally to close the files, and to correctly handle end-of-file, in case the input file is shorter than the sum of parts.
I would eliminate the readWrite() method, and consolidate the logic to prevent duplicate code, like this:
public void splitFile(String inPath, long[] sizeOfParts) throws IOException, SQLException {
final int numberOfParts = sizeOfParts.length;
String[] outPath = new String[numberOfParts];
// Gets Paths from Database here
InputStream in = null;
OutputStream[] out = new OutputStream[numberOfParts];
try {
in = new BufferedInputStream(new FileInputStream(inPath));
for (int part = 0; part < numberOfParts; part++)
out[part] = new BufferedOutputStream(new FileOutputStream(outPath[part]));
byte[] buf = new byte[8];
long[] remain = sizeOfParts.clone();
for (boolean done = false; ! done; ) {
done = true;
for (int part = 0; part < numberOfParts; part++) {
if (remain[part] > 0) {
int len = in.read(buf, 0, (int) Math.min(remain[part], buf.length));
if (len == -1) {
done = true;
break;
}
remain[part] -= len;
System.out.println("file " + part + " " + (sizeOfParts[part] - remain[part]));
out[part].write(buf, 0, len);
done = false;
}
}
}
} finally {
if (in != null)
in.close();
for (int part = 0; part < out.length; part++)
if (out[part] != null)
out[part].close();
}
}
Suppose I receive bytes in chunks and I want to efficiently decode them to a string (that is going to be Unicode obviously), also I want to know, as soon as I can, if that string begins with a certain sequence.
One way could be:
public boolean inputBytesMatch(InputStream inputStream, String match) throws IOException {
byte[] buff = new byte[1024];
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
int len;
while ((len = inputStream.read(buff)) > 0){
byteArrayOutputStream.write(buff, 0, len);
String decoded = new String(byteArrayOutputStream.toByteArray(), Charset.defaultCharset());
if (decoded.startsWith(match)){
return true;
}
}
return false;
}
but this involves allocating a new array from the byteArrayOutputStream every time there is a new chunk and String will do another copy in the constructor. All this seems to me pretty inefficient. Also string will do a decode of the bytes in the constructor, every single time, doing it from the beginning once again.
How can I make this process faster?
Actually you don't need a ByteArrayOutputStream at all.
First turn your String match into a byte[], using your desired encoding.
Then just compare each incoming chunk with the next part of that array:
public boolean inputBytesMatch(InputStream inputStream, String match) throws IOException {
byte[] compare = match.getBytes(Charset.defaultCharset());
int n = compare.length;
int compareAt = 0;
byte[] buff = new byte[n];
int len;
while (compareAt < n && (len = inputStream.read(buff, 0, n-compareAt)) > 0) {
for (int i=0; i < len && compareAt < n; i++, compareAt++) {
if (compare[compareAt] != buff[i]) {
// found contradicting byte
return false;
}
}
}
// No byte was found which contradicts that the streamed data begins with compare.
// Did we actually read enough bytes?
return compareAt >= n;
}
You might find this version more readable:
public boolean inputBytesMatch(InputStream inputStream, String match) throws IOException {
byte[] compare = match.getBytes(Charset.defaultCharset());
int n = compare.length;
int compareAt = 0;
byte[] buff = new byte[n];
int len;
while (compareAt < n && (len = inputStream.read(buff, 0, n-compareAt)) > 0) {
if (!isSubArray(compare, compareAt, buff, len)) {
return false;
}
compareAt += len;
}
return compareAt >= n;
}
private boolean isSubArray(byte[] searchIn, int searchInOffset, byte[] searchFor, int searchForLength)
{
if (searchInOffset + searchForLength >= searchIn.length) {
// can not match
return false;
}
for (int i=0; i < searchForLength; i++) {
if (searchIn[searchInOffset+i] != searchFor[i]) {
return false;
}
}
return true;
}
I am trying to build a manual HTTP client (using sockets) along with a cache and I cant seem to figure out why the files are not saving to disk properly. It works pretty good for HTML files, but cant seem to work for other files types that re not text based like .gif. Could anyone tell me why? I am quite new to HTTP protocol and Socket programming in general.
The loop to grab the response.
InputStream inputStream = socket.getInputStream();
PrintWriter outputStream = new PrintWriter(socket.getOutputStream());
ArrayList<Byte> dataIn = new ArrayList<Byte>();
ArrayList<String> stringData = new ArrayList<String>();
//Indices to show the location of certain lines in arrayList
int blankIndex = 8;
int lastModIndex = 0;
int byteBlankIndex = 0;
try
{
//Get last modified date
long lastMod = getLastModified(url);
Date d = new Date(lastMod);
//Construct the get request
outputStream.print("GET "+ "/" + pathName + " HTTP/1.1\r\n");
outputStream.print("If-Modified-Since: " + ft.format(d)+ "\r\n");
outputStream.print("Host: " + hostString+"\r\n");
outputStream.print("\r\n");
outputStream.flush();
//Booleans to prevent duplicates, only need first occurrences of key strings
boolean blankDetected = false;
boolean lastModDetected = false;
//Keep track of current index
int count = 0;
int byteCount = 0;
//While loop to read response
String buff = "";
byte t;
while ( (t = (byte) inputStream.read()) != -1)
{
dataIn.add(t);
//Check for key lines
char x = (char) t;
buff = buff + x;
//For the first blank line (signaling the end of the header)
if(x == '\n')
{
stringData.add(buff);
if(buff.equals("\r\n") && !blankDetected)
{
blankDetected = true;
blankIndex = count;
byteBlankIndex = byteCount + 2;
}
//For the last modified line
if(buff.contains("Last-Modified:") && !lastModDetected)
{
lastModDetected = true;
lastModIndex = count;
}
buff = "";
count++;
}
//Increment count
byteCount++;
}
}
The the code to parse through response and write file to disk.
String catalogKey = hostString+ "/" + pathName;
//Get the directory sequence to make
String directoryPath = catalogKey.substring(0, catalogKey.lastIndexOf("/") + 1);
//Make the directory sequence if possible, ignore the boolean value that results
boolean ignoreThisBooleanVal = new File(directoryPath).mkdirs();
//Setup output file, and then write the contents of dataIn (excluding header) to the file
PrintWriter output = new PrintWriter(new FileWriter(new File(catalogKey)),true);
for(int i = byteBlankIndex + 1 ; i < dataIn.size(); i++)
{
output.print(new String(new byte[]{ (byte)dataIn.get(i)}, StandardCharsets.UTF_8));
}
output.close();
byte t;
while ( (t = (byte) inputStream.read()) != -1)
The problem is here. It should read:
int t;
while ( (t = inputStream.read()) != -1)
{
byte b = (byte)t;
// use b from now on in the loop.
The issue is that a byte of 0xff in the input will be returned to the int as 0xff, but to the byte as -1, so you are unable to distinguish it from end of stream.
And you should use a FileOutputStream, not a FileWriter, and you should not accumulate potentially binary data into a String or StringBuffer or anything to do with char. As soon as you've got to the end of the header you should open a FileOutputStream and just start copying bytes. Use buffered streams to make all this more efficient.
Not much point in any of these given that HttpURLConnection already exists.
I tried to get MOTD of the remote server, but I can't get colors. When MOTD is colored, the plugin isn't working.
I know why, but I don't know how to resolve it.
public PingServer(String host, int port) {
this.host = host;
this.port = port;
try {
socket.connect(new InetSocketAddress(host, port));
OutputStream out = socket.getOutputStream();
InputStream in = socket.getInputStream();
out.write(0xFE);
int b;
StringBuffer str = new StringBuffer();
while ((b = in.read()) != -1) {
if (b != 0 && b > 16 && b != 255 && b != 23 && b != 24) {
str.append((char) b);
}
}
data = str.toString().split("§");
data[0] = data[0].substring(1, data[0].length());
} catch (IOException e) {
e.printStackTrace();
}
}
According to the specification, the plugin will get response like this: MOTD§ONLINEPLAYERS§MAXPLAYERS, which should be split on § to get the different portions. However, § is also used for chat messages, and I'm not sure how to differentiate between the two. How can I work around this?
You're currently using the legacy server list ping, designed for beta 1.8 to 1.3. That one is triggered via sending just FE to the server. While current servers still support this ping, it's very old and has several flaws (including the one you found).
You should instead perform the current ping. While this is slightly more complicated, you don't need to implement much of the protocol to actually perform it.
There's only one complicated portion of the protocol you need to know about: VarInts. These are somewhat complicated because they take a varying number of bytes depending on the value. And as such, you have a packet length that can be somewhat hard to calculate.
/** See http://wiki.vg/Protocol_version_numbers. 47 = 1.8.x */
private static final int PROTOCOL_VERSION_NUMBER = 47;
private static final int STATUS_PROTOCOL = 1;
private static final JsonParser PARSER = new JsonParser();
/** Pings a server, returning the MOTD */
public static String pingServer(String host, int port) {
this.host = host;
this.port = port;
try {
socket.connect(new InetSocketAddress(host, port));
OutputStream out = socket.getOutputStream();
InputStream in = socket.getInputStream();
byte[] hostBytes = host.getBytes("UTF-8");
int handshakeLength =
varIntLength(0) + // Packet ID
varIntLength(PROTOCOL_VERSION_NUMBER) + // Protocol version number
varIntLength(hostBytes.length) + hostBytes.length + // Host
2 + // Port
varIntLength(STATUS_PROTOCOL); // Next state
writeVarInt(handshakeLength, out);
writeVarInt(0, out); // Handshake packet
writeVarInt(PROTOCOL_VERSION_NUMBER, out);
writeVarInt(hostBytes.length, out);
out.write(hostBytes);
out.write((port & 0xFF00) >> 8);
out.write(port & 0xFF);
writeVarInt(STATUS_PROTOCOL, out);
writeVarInt(varIntLength(0));
writeVarInt(0); // Request packet (has no payload)
int packetLength = readVarInt(in);
int payloadLength = readVarInt(in);
byte[] payloadBytes = new int[payloadLength];
int readLength = in.read(payloadBytes);
if (readLength < payloadLength) {
throw new RuntimeException("Unexpected end of stream");
}
String payload = new String(payloadBytes, "UTF-8");
// Now you need to parse the JSON; this is using GSON
// See https://github.com/google/gson
// and http://www.javadoc.io/doc/com.google.code.gson/gson/2.8.0
JsonObject element = (JsonObject) PARSER.parse(payload);
JsonElement description = element.get("description");
// This is a naive implementation; it assumes a specific format for the description
// rather than parsing the entire chat format. But it works for the way the
// notchian server impmlements the ping.
String motd = ((JsonObject) description).get("text").getAsString();
return motd;
} catch (IOException e) {
e.printStackTrace();
return null;
}
}
public static int varIntLength(int value) {
int length = 0;
do {
// Note: >>> means that the sign bit is shifted with the rest of the number rather than being left alone
value >>>= 7;
length++;
} while (value != 0);
}
public static void writeVarInt(int value, OutputStream out) {
do {
byte temp = (byte)(value & 0b01111111);
// Note: >>> means that the sign bit is shifted with the rest of the number rather than being left alone
value >>>= 7;
if (value != 0) {
temp |= 0b10000000;
}
out.write(temp);
} while (value != 0);
}
public static int readVarInt(InputStream in) {
int numRead = 0;
int result = 0;
int read;
do {
read = in.read();
if (read < 0) {
throw new RuntimeException("Unexpected end of stream");
}
int value = (read & 0b01111111);
result |= (value << (7 * numRead));
numRead++;
if (numRead > 5) {
throw new RuntimeException("VarInt is too big");
}
} while ((read & 0b10000000) != 0);
return result;
}
The current ping does use JSON, which means you need to use GSON. Also, this implementation makes some assumptions about the chat format; this implementation could break on custom servers that implement chat more completely, but it'll work for servers that embed § into the motd instead of using the more complete chat system (this includes the Notchian server implementation).
If you need to use the legacy ping, you can assume that the 2nd-to-last § marks the end of the MOTD (rather than the 1st §). Something like this:
String legacyPingResult = str.toString();
String[] data = new String[3];
int splitPoint2 = legacyPingResult.lastIndexOf('§');
int splitPoint1 = legacyPingResult.lastIndexOf('§', splitPoint2 - 1);
data[0] = legacyPingResult.substring(0, splitPoint1);
data[1] = legacyPingResult.substring(splitPoint1 + 1, splitPoint2);
data[2] = legacyPingResult.substring(splitPoint2 + 1);
However, I still don't recommend using the legacy ping.
My assignment: I have a queue-system provided by the IBM MQ-Series. I need to get all messages of a "Dead-Letter-Queue". The RFH2-Header and the original payload were assembled in the Dead-Letter. I the original payload is mostly a zip-file or string. Since there is the possibilty of the payload being a zip-file, I need to save the payload as a byte[] in java. The RFH2 should be a Linked HashMap.
What I've got so far:
public ZMessageDTO(final ZMessageType messageType, MQMessage message) throws ZDAEQException {
this.mqMessage = message;
this.messageType = messageType;
List<byte[]> payloadRFH2List = this.split("</usr>".getBytes(StandardCharsets.UTF_8));
rfh2 = getRFH2OfXMLBytes(payloadRFH2List.get(0));
payload = payloadRFH2List.get(1);
}
public boolean isMatch(byte[] pattern, byte[] input, int pos) {
for (int i = 0; i < pattern.length; i++) {
if (pattern[i] != input[pos + i]) {
return false;
}
}
return true;
}
public List<byte[]> split(byte[] pattern) throws ZDAEQException {
try {
byte[] input = new byte[(int) mqMessage.getMessageLength()];
for (int i = 0; i < (int) mqMessage.getMessageLength(); i++) {
input[i] = mqMessage.readByte();
}
List<byte[]> l = new LinkedList<byte[]>();
int blockStart = 0;
for (int i = 0; i < input.length; i++) {
if (isMatch(pattern, input, i)) {
l.add(Arrays.copyOfRange(input, blockStart, i));
blockStart = i + pattern.length;
i = blockStart;
}
}
l.add(Arrays.copyOfRange(input, blockStart, input.length));
return l;
} catch (IOException e) {
throw new ZDAEQException("Couldn't split dead letter" + e);
}
}
What happens by now:
If I create a new Message out of the data a get, it cuts the rfh2 by the 2066th char. If i call the original payload, I get a NullPointer.
What's the crux?
As I mentionend in the assignment, the RFH2 and the original payload are both assembled in the "Dead-Letter-Payload" (I hope you unterstand what I'm trying to tell). What I tried so far, is to search the payload for the </usr>-ending tag and save the two parts. I saved the first part as a String and the second part as a byte[]. I have absolutely no Idea if this works or not.
My question:
Is it possible to split the Dead-Letter-Payload in two parts and save the first one as a string, and the second one as a byte[]?
The MQRFH2 structure is very complex. If you don't understand it then why aren't you using the MQRFH2 class?
IBM provides the MQRFH2 class so that developers don't have to deal with the structure.
i.e.
mqMsg.seek(0);
MQRFH2 rfh2 = new MQRFH2(mqMsg);
byte[] data = new byte[mqMsg.getDataLength()];
mqMsg.readFully(data);
System.out.println("Message data: "+new String(data));