I am working on a crawler and I have the following question: it works with simple HTTP requests, but not HTTPS, and I need to make an HTTPS request. I changed the port to 443 and try to send the same request, but I get 400 error. Obviously, I need to change something else, but I do not know what. I open socket and this is how I make the request:
String request
= "GET " + file
+ (file.endsWith("robots.txt") ? " HTTP/1.0\r\n" : " HTTP/1.1\r\n")
// " HTTP/1.1\r\n"
+ "User-Agent: " + CrawlerConfig.USER_AGENT + "\r\n"
// + ((!CrawlerConfig.SAVE_IMAGES) ? "Accept: text/html\r\n" : "")
// + "Accept: text/*\r\n"
+ (file.endsWith("robots.txt") ? "Connection: close\r\n" : "")
+ "Host: " + host + "\r\n" + "\r\n"/*
* + body
*/;
outStream.write(request.getBytes("US-ASCII"));
outStream.flush();
Try to send an OPTIONS request on the resource, needed request headers should be returned, some may be missing.
Related
I have built a Soap String message without the http header on top.
Is there any spring or Java EE technology that can be used to send the message to a soap server?
I want to have the ability to write to the logs what is sent exactly but also hiding sensitive data in the soap request such as passwords (when it is printed to the logs).
public class SoapRequestTest {
#Test
public void createHttp() throws IOException {
String soapRequest = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n" +
"<soap:Envelope xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n" +
" <soap:Body>\n" +
" <Divide xmlns=\"http://tempuri.org/\">\n" +
" <intA>int</intA>\n" +
" <intB>int</intB>\n" +
" </Divide>\n" +
" </soap:Body>\n" +
"</soap:Envelope>";
}
}
Now this soap message I want to use when calling the Divide method in http://www.dneonline.com/calculator.asmx?wsdl
I'm new to Java Spring security. I've been able to successfully add some HTTP headers and deploy them via the code I have below.
Once the basic HTTP headers were working, I also added headers for Strict-Transport-Security. It seems to work locally however, once deployed, the value gets overridden.
I've talked to some of our architects and they don't believe that there is anything sitting between this Java app and the browser which might be overwriting/replacing the values. They think it might be something in Spring that is fighting with the code I have here.
Curious if anyone has experience with this and could offer some advice.
I've include some screenshots of what Chrome is showing me for headers.
#Override
public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
response.setHeader("Content-Security-Policy",
"default-src https: http: 'unsafe-inline' 'unsafe-eval'; " +
"script-src 'self' 'unsafe-inline' 'unsafe-eval'; " +
"object-src 'none'; " +
"base-uri 'self'; " +
"frame-ancestors 'self'; " +
"form-action https: http: 'self'; "
);
response.setHeader("Strict-Transport-Security",
"max-age=0; " +
"Cache-Control: no-cache, no-store, must-revalidate, private; " +
"Pragma: no-cache; " +
"Expires: 0; " +
"includeSubDomains"
);
response.setHeader("Upgrade-Insecure-Requests", "1");
response.setHeader("X-Frame-Options", "DENY");
response.setHeader("X-XSS-Protection", "1; mode=block");
response.setHeader("Referrer-Policy", "no-referrer-when-downgrade");
super.postHandle(request, response, handler, modelAndView);
}
}```
[![local deployed changes][1]][1]
[![server deployed changes][2]][2]
[1]: https://i.stack.imgur.com/3LGEd.png
[2]: https://i.stack.imgur.com/8NAMn.png
I try to send a File via xmpp and smack
FileTransferManager manager = new FileTransferManager(
this.xmppConnection);
OutgoingFileTransfer transfer = manager
.createOutgoingFileTransfer(this.jid);
transfer.sendFile(file, "test");
while (!transfer.isDone()) {
LOGGER.info("Uploading File: " + file.getName() + " STATUS: "
+ transfer.getStatus() + " >> progress: " + 100
* transfer.getProgress());
try {
Thread.sleep(100);
} catch (InterruptedException ex) {
LOGGER.error("Exception: " + ex);
}
}
LOGGER.info("File transfer is done: " + file.getName() + " STATUS: "
+ transfer.getStatus() + " >> progress: " + 100
* transfer.getProgress());
LOGGER.info("Amount written: " + transfer.getAmountWritten());
LOGGER.info("Bytes sent: " + transfer.getBytesSent());
LOGGER.info("Peer: " + transfer.getPeer());
LOGGER.info("Error: " + transfer.getError());
LOGGER.info("Exception: ", transfer.getException());
The logs are:
Uploading File: image_P9.png STATUS: Initial >> progress: 0.0
Uploading File: image_P9.png STATUS: Negotiating Transfer >> progress: 0.0
File transfer is done: image_P9.png STATUS: Refused >> progress: 0.0
Amount written: -1
Bytes sent: -1
Peer: felix.infraview#jabber.de/IM+ Android
Error: null
Exception:
so I don't get a Error or Exception. The sending is just refused.
My goal is to send an inline-image to any android client.
STATUS: Refused
...
Peer: felix.infraview#jabber.de/IM+
It means the target peer does not support file transfer methods you offer to. XMPP SI File Transfer is a very complex specification which is not implemented by simple clients.
In fact, it does not properly implemented in Smack too, they can give you detailed description about error, but implemented only generic "refused".
How to get the status code or something better to know if an message was sent?
I already do this, but off course it don't work.
httppost.setEntity(new StringEntity(message.toString()));
// Execute HTTP Post Request
HttpResponse response = httpclient.execute(httppost);
String status = response.getStatusLine().toString();
Log.d(CLASS_TAG + "showOnScreenMessage", "Message sent! ");
Log.d(CLASS_TAG + "showOnScreenMessage", "Status : " + status );
// if response == 201 , the message has be received, Set True to acknowledgment_message
if ( status == "201"){
Log.d(CLASS_TAG + "showOnScreenMessage", "Message received! ");
}else{
// Do nothing already false
Log.d(CLASS_TAG + "showOnScreenMessage", "Message not received! ");
}
Thanks for your suggestions
String compare as it's say above.
status.equals("201")
Or you can use :
response.getStatusLine().getStatusCode() != HttpStatus.SC_OK
All status here
Change:
status == "201"
to:
status.equals("201")
I receive this warning
[javac] SSLTunnelSocketFactory.java:120: warning:
sun.net.www.protocol.http.HttpURLConnection is Sun proprietary API and
may be removed in a future release
[javac] + sun.net.www.protocol.http.HttpURLConnection.userAgent
for
if(NetworkUtils.isIPV6Compatible()&& NetworkUtils.isValidIPV6Address(host)){
msg = "CONNECT [" + host + "]:" + port + " HTTP/1.0\n"
+ "User-Agent: "
+ sun.net.www.protocol.http.HttpURLConnection.userAgent
+ "\r\n\r\n";
}
Do you have an idea what can I use instead (preferably not a an external jar but from the installed JVM)?
By default that variable is defined as follows:
String javaVersion = "Java/" + System.getProperty("java.version");
String userAgent = System.getProperty("http.agent") == null ? javaVersion : System.getProperty("http.agent") + " " + javaVersion;
So you can replace that variable with this one or with the name of your application or with any other string you like (since the resulting string is not the User-Agent of a common browser I assume that no JavaScript or any other code will check for it).