I am a Java developer
I'm trying to stream an unknown amount of binary data from client to server over http put request.
For some reason, my server is written in c# (.net core) and for convenience, my client (that i use only for test) is written in java.
As i'm waiting for unknown amount of data and in order to test my server rest api, my java client open a binary file and send binary packet over http put request (160 bytes by packet for example).
When i begin streaming data from client, i'm not able to consume the body (http stream) in server side until the client finished to send all binary packets.
http headers in use use: Connection-KeepAlive, Content-Type=> application/octetstream, transfer-encoding => chunked
So, I would like to consume the body without waiting the entire body.
Any hint of how to perform it ?
I'm thinking about removing await operator before async method that consume body and wait until client abort or close the connection in server side but i don't know if this is a good idea.
Below some piece of related code:
server side written in c#
[HttpPut]
public async Task<IActionResult> SendData(string id, string device, string token){
...
using (MemoryStream ms = new MemoryStream()){
await HttpContext.Request.Body.CopyToAsync(ms);
}
...
//do stuff
}
java client
HttpURLConnection conn = (HttpURLConnection) new URL(
"http://localhost:8080/test"
).openConnection();
byte[] data = Files.readAllBytes(Paths.get(FILE_TO_SEND_PATH));
ByteArrayInputStream is = new ByteArrayInputStream(data);
conn.setRequestMethod("PUT");
conn.setRequestProperty("Content-Type", "application/octet-stream");
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setChunkedStreamingMode(160);
//conn.setFixedLengthStreamingMode(is.available());
conn.setDefaultUseCaches(false);
conn.setUseCaches(false);
conn.connect();
OutputStream out = conn.getOutputStream();
//InputStream i = conn.getInputStream();
byte[] bytes = new byte[160];
int read = 0;
while ((read = is.read(bytes)) > 0) {
out.write(bytes, 0, read);
System.out.println(read);
TimeUnit.MICROSECONDS.sleep(125 * read);
out.flush();
}
out.close();
Related
Am making webservice calls to HTTPS server from an android application. Below is the code snippet, with which am able to make web service calls successfully and getting response.
My Question is, do we need to perform any additional step to encrypt data before making call to HTTPS server?
Because, from android profiler am able to see all my Web Requests in plain text format. My understanding is that request will gets encrypted before making HTTPS call.
public static WebServiceResp makeWebServiceCall(String XML, String urlPath) throws IOException{
//Code to make a web service HTTP request
String responseString = "";
String outputString = "";
String wsURL = urlPath;
URL url = new URL(wsURL);
URLConnection connection = url.openConnection();
HttpsURLConnection httpConn = (HttpsURLConnection)connection;
ByteArrayOutputStream bout = new ByteArrayOutputStream();
//System.out.println(XML);
byte[] buffer = new byte[XML.length()];
buffer = XML.getBytes();
bout.write(buffer);
byte[] b = bout.toByteArray();
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty("Content-Length",
String.valueOf(b.length));
httpConn.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
httpConn.setRequestMethod("POST");
httpConn.setRequestProperty("Cache-Control", "no-cache");
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
OutputStream out = httpConn.getOutputStream();
//Write the content of the request to the outputstream of the HTTP Connection.
out.write(b);
out.close();
//Ready with sending the request.
//Check the status
int status = httpConn.getResponseCode();
Log.d(TAG, "makeWebServiceCall: "+"Processing Status: "+status);
BufferedReader in;
if (status <= 200) {
//Read the response.
Log.d(TAG, "makeWebServiceCall: Getting Input Stream");
InputStreamReader isr =
new InputStreamReader(httpConn.getInputStream());
in = new BufferedReader(isr);
}else{
//Read the response.
Log.d(TAG, "makeWebServiceCall: Getting Error Stream");
InputStreamReader isr =
new InputStreamReader(httpConn.getErrorStream());
in = new BufferedReader(isr);
}
//Write the SOAP message response to a String.
while ((responseString = in.readLine()) != null) {
outputString = outputString + responseString;
}
Log.d(TAG, "makeWebServiceCall: WebServiceResponse " + outputString);
//Parse the String output to a org.w3c.dom.Document and be able to reach every node with the org.w3c.dom API.
Document document = Utils.parseXmlFile(outputString);
//NodeList nodeLst = document.getElementsByTagName("GetWeatherResult");
// String weatherResult = nodeLst.item(0).getTextContent();
//System.out.println("Weather: " + weatherResult);
//Write the SOAP message formatted to the console.
WebServiceResp webServiceResp = new WebServiceResp();
webServiceResp.setDocument(document);
webServiceResp.setStatus(status);
return webServiceResp;
}
No. If you're sending it to an https website, the encryption is done as part of the protocol. You don't need to do any additional work.
No. The encryption that you see is on the network layer. The client which initiates the https call see what was sent and what was received. That is how https works.
When you look at chrome browser's network tab, you see what was sent and what was received. Now this is not a security problem, https is more about you doing things which make its difficult for anyone between the network to eavesdrop your data.
Now if you still want an additional level of security you can use certificate pinning
https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning
https://medium.com/#appmattus/android-security-ssl-pinning-1db8acb6621e
How can you add to network_security_config from MainActivity
So in this technique you basically say that the certificate hash that you expected is to have this content. And then if someone uses a trusted proxy with trusted CAs on the system, even after generating a valid certificate for the given domain the connections will not be established.
HTTPS is transparent to your application, all of the magic happens between Transport Layer(so it calls 'Transport Layer Security'), you may imagine encrypted telegrams in the old days, generals tell telegrapher messages in plain text, and telegrapher send them in encrypted form(maybe use some kind of codebook), anyone who didn't have the same codebook can't decrypt the message easily, and anyone who uses telegrams didn't care about the codebook(or even known about it, except those telegraphers on both side of the 'Transport Layer').
The encryption/decryption is done by built-in network client module provided by OS. So you needn't worry about it.
You can view plain texts with some client tools as they know exactly what they are sending/receiving. E.g. chrome developer tool. (Actually they don't care about encryption/decryption either).
I'm making a POST request using Java 8 like this:
URL url = new URL("http://target.server.com/doIt");
URLConnection connection = url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) connection;
byte[] soapBytes = soapRequest.getBytes();
httpConn.setRequestProperty("Host", "target.host.com");
httpConn.setRequestProperty("Content-Length", soapBytes.length+"");
httpConn.setRequestProperty("Content-Type", "application/soap+xml; charset=utf-8");
httpConn.setRequestMethod("POST");
httpConn.setConnectTimeout(5000);
httpConn.setReadTimeout(35000);
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
OutputStream out = httpConn.getOutputStream();
out.write(soapBytes);
out.close();
int statusCode;
try {
statusCode = httpConn.getResponseCode();
} catch (IOException e) {
InputStream stream = httpConn.getErrorStream();
if (stream == null) {
throw e;
} else {
// this never happens
}
}
My soap request contains a document ID and the target server (which hosts a third-party service that I do not own or have access to) returns a PDF document that matches the supplied ID.
Most of the time, the server returns a PDF doc and occasionally the status code is 500 when the document is not available. However, sometimes the call to getResponseCode() throws an IOException with "Invalid Http response".
I thought that a server would always have some response code to return, no matter what happens.
Does this mean that server is returning complete garbage that doesn't
match the expected format of a HTTP response?
Is there a way to get any more information about the actual response?
Is there a way to retrieve the raw textual response (if any)?
As AxelH points out, there must be something wrong when connecting with the remote server, and in this case you just can't get a valid response.
If you are in a testing environment, you can monitorize the connection at TCP level (not at HTTP level): Put a monitor between your client and the remote server which monitorizes all the TCP traffic exchanged between the two peers. If you are using Eclipse, you can create a TCP monitor.
I want to make http request with socket because I want to test how many sockets I can create my server. So I write and read from my server using OutputStream and InputStream. But I can't read from the input stream again after the first response. Do you know how to read the second response without closing the socket?
Here is my code:
Socket socket = new Socket();
socket.connect(new InetSocketAddress(address, 80), 1000);
socket.setSoTimeout(25*1000);
OutputStream os = socket.getOutputStream();
os.write(getRequest(host)); // some request as bytearray, it has Connection: Keep-Alive in the header
os.flush();
InputStream is = socket.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
String response = IOUtils.toString(bis);
System.out.println("RESPONSE = \n" + response); // this works fine
os.write(getRequestBodyBa()); // send another request, i can see it sent to server with wireshark
os.flush();
// try to read again but it always return empty string
response = IOUtils.toString(bis); // how to read the second response?????
System.out.println("RESPONSE = \n" + response);
os.close();
is.close();
socket.close();
Thanks.
I believe the HTTP standard are to close the connection after each response, unless the request has the Connection header set to keep-alive.
IOUtils.toString(InputStream) reads the stream to EOS, so there can't be anything left to read for next time. Don't use it. You need to parse the response headers, work out whether there is a Content-Length header, if so read the body for exactly that many bytes; if there is no Content-Length header (and no chunking) the connection is closed after the body so you can't send a second command; etc etc etc. It is endless. Don't use a Socket for this either: use an HTTP URL and URLConnection.
My client receive raw HTTP headers (including GET, POST, Multipart POST, etc.) and I want to send them to a server and get output.
But I don't want to parse whole request manually, then set all that parsed stuff to HttpClient...
Does an elegant way to do this (even something like code below)?
AGoodHttpClient response = new AGoodHttpClient(host, port, myHeaders);
InputStream in = response.getInputStream();
// ...
Edited
Let's say I have this code. How do I recognize EOS (-1 isn't working for HTTP/1.1). Is there a guaranteed way how to cut the connection, when transfer is done? I want something what will care about cutting a connection (something like HttpClient), but with direct access to sending headers (like outToServer.write(myHeaders)).
Socket connectionToServer = new Socket(host, port);
OutputStream outToServer = connectionToServer.getOutputStream();
outToServer.write(myHeaders.getBytes());
InputStream inputFromServer = connectionToServer.getInputStream();
byte[] buff = new byte[1024];
int count;
while ((count = inputFromServer.read(buff)) != -1) {
System.out.write(buff, 0, count);
}
Thanks for help!
I'm a problem with a HttpsURLConnection that I can't seem to solve. Basically, I'm sending up some info to a server and if some of that data is wrong, the server sends me a 500 response code. However, it also sends a message in the response telling me which bit of data was wrong. The problem is that the message is always empty when I read it in. I think this is because a filenotfound exception always gets thrown before the stream can be read. Am I right? I tried reading the errorstream as well but this is always empty. Here's a snippet:
conn = (HttpsURLConnection) connectURL.openConnection();
conn.setDoOutput(true);
conn.setConnectTimeout(30000);
conn.setReadTimeout(30000);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Length",
Integer.toString(outString.getBytes().length));
DataOutputStream wr = new DataOutputStream(conn
.getOutputStream());
wr.write(outString.getBytes());
wr.flush();
wr.close();
if(conn.getResponseCode>400{
String response = getErrorResponse(conn);
public String getErrorResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
//is = conn.getInputStream();
is = conn.getErrorStream();
// scoop up the reply from the server
int ch;
StringBuffer sb = new StringBuffer();
while ((ch = is.read()) != -1) {
sb.append((char) ch);
}
//System.out.println(sb.toString());
return sb.toString();
// return conferenceId;
}
catch (Exception e){
e.printStackTrace();
}
}
So just to follow up on this, here is how I solved it:
public static String getResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
if(conn.getResponseCode()>=400){
is = conn.getErrorStream();
}
else{
is=conn.getInputStream();
}
...read stream...
}
It seems that calling them like this produced an error stream with a message. Thanks for the suggestions!
Try setting content-type request property to "application/x-www-form-urlencoded"
The same is mentioned on this link:
http://developers.sun.com/mobility/midp/ttips/HTTPPost/
The Content-Length and Content-Type headers are critical because they tell the web server how many bytes of data to expect, and what kind, identified by a MIME type.
In MIDP clients the two most popular MIME types are application/octet-stream, to send raw binary data, and application/x-www-form-urlencoded, to send name-value pairs
Are you in control of the server? In other words, did you write the process that runs on the server and listens to the port you're trying to access?
If you did, then you should also be able to debug it and see why your process returns 404.
If you didn't, then describe your architecture (HTTP server, the component it invokes to respond to your HTTP(S) request, etc) and we'll take it from there.
In the very simplest case, of an HTTP server being an Apache server yielding control to some PHP script, it means that Apache couldn't assign your request to anything. Most likely a Web server misconfiguration. Provide some more details and we'll help you out.