Right now I'm having some major issues with writing a simple portscanner / password checker for a work security project.
My basic goal is to write a quick little tool that opens up a text file, Scans on ports 21,23,80,502, and 8080 then simply writes the returned http status to a file (200, 404, whatever)
So far I've been trying to do this using httpclient and I've had very poor results.
My code goes something like this
public static void doHosts() throws Exception{
String filename = "C:\\test.txt";
String ip = "";
String port[] = {"21", "23", "80", "502", "8080"};
FileInputStream fstream = new FileInputStream("c:\\scan.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
while ((ip = br.readLine()) != null) {
HttpParams httpParameters = new BasicHttpParams();
int timeoutConnection = 3000;
int timeoutSocket = 5000;
HttpConnectionParams.setConnectionTimeout(httpParameters, timeoutConnection);
HttpConnectionParams.setSoTimeout(httpParameters, timeoutSocket);
HttpHost targetHost = new HttpHost(ip);
DefaultHttpClient httpclient = new DefaultHttpClient(httpParameters);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials("blah", "blah");
httpclient.getCredentialsProvider().setCredentials(new AuthScope(targetHost.getHostName(), targetHost.getPort()), creds);
NegotiateSchemeFactory nsf = new NegotiateSchemeFactory();
httpclient.getAuthSchemes().register(AuthPolicy.SPNEGO, nsf);
// Create AuthCache instance
AuthCache authCache = new BasicAuthCache();
// Generate BASIC scheme object and add it to the local auth cache
BasicScheme basicAuth = new BasicScheme();
authCache.put(targetHost, basicAuth);
// Add AuthCache to the execution context
BasicHttpContext localcontext = new BasicHttpContext();
localcontext.setAttribute(ClientContext.AUTH_CACHE, authCache);
httpclient = wrapClient(httpclient);
HttpGet get;
for (int i = 0; i < port.length; i++) {
get = new HttpGet("http://" + ip + ":" + port[i] + "/");
HttpResponse response = httpclient.execute(get);
try {
BufferedWriter out = new BufferedWriter(new FileWriter("filename", true));
out.write("ip: "+ ip + " : "+port+ " - " + response.getStatusLine().getStatusCode()); //HTTP status returned off request );
} catch (IOException e) {
}
}
}
}
My issue thus far has been that it opens the text file, hits a single bad result, doesn't serialize it, then just dies with an exception.
I'm not sure how to make it continue with an exception (I know some sites will say "not up" that's why it's a port scanner.
Any help would be wonderful and would probably save some poor interns from manually checking about 6000 ip's tomorrow.
You are not handling exceptions thrown by HttpClient when the connection cannot be made. You should not assume that httpclient.execute(get) will always return without exception.
If you want your program to ignore these conditions and keep on trucking, you need to catch these exceptions.
Alternatively, it seems like you might want to give nmap a try.
You don't say in your post, but my guess is it's throwing ClientProtocolException on the first request to port 21, which is to be expected as port 21 does not speak HTTP. You are not catching this exception, so your program dies.
What do you mean by "bad results"? Cases where nmap said a port was open or closed and it wasn't?
If you do decide to continue down the nmap path and want to integrate nmap with some Java code, have a look at Nmap4j on sourceforge.net. It's a simple Java wrapper around Nmap that allows you to access the results as Java objects.
Related
I am digging for quite a while and I am wondering how do I open an HttpClient connection in Java (Android) and then close the socket(s) right away without getting CLOSE_WAIT and TIME_WAIT TCP statuses while I am checking network monitoring tools.
What I am doing is (Found this solution on stackoverflow site):
String url = "http://example.com/myfile.php";
String result = null;
InputStream is = null;
StringBuilder sb = null;
try {
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
HttpResponse response = httpclient.execute(httppost);
HttpEntity entity = response.getEntity();
is = entity.getContent();
} catch (Exception e) {
Log.e("log_tag", "Error in http connection" + e.toString());
}
// convert response to string
try {
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, "iso-8859-1"), 8);
sb = new StringBuilder();
sb.append(reader.readLine() + "\n");
String line = "0";
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
is.close();
result = sb.toString();
} catch (Exception e) {
}
Toast.makeText(getApplicationContext(), result, Toast.LENGTH_LONG).show();
After I run this code - The PHP file is executed well, I get the response back to TOAST, BUT - when I analyze the networking environment of my mobile device with external network analyzer tool - I see that the connection(s) stay in CLOSE_WAIT or/and TIME_WAIT for about 1 minute and only then they move to CLOSED state.
The problem is:
I am calling the above function every ~2 to 5 seconds in an infinite loop, which result over time a huge amount of CLOSE_WAITs and TIME_WAITs - which affect the overall performance of my Android app, until it gets stuck and useless !
What I want to do is (And need your answer if possible):
I wish to really close the connection RIGHT AWAY after I TOAST the response message without any open sockets. No TIME_WAIT and no CLOSE_WAIT. No left overs at all - close all communication IMMEDIATELY at the split second that I run code that should do so. I don't need the connection anymore until the next iteration of the loop.
How can I accomplish that ?
I have in mind that I don't want the application to halt or have poor performance over time, since it should run in a service/stay open forever.
I would really appreciate if you could write simple code that work after I do copy-paste.
I am new to Java and Android, so I will try to figure out the code that you write, so please keep it as simple as possible. Thanks a lot !
Question asker.
try using HttpURLConnection class. refer to following link :
http://android-developers.blogspot.de/2011/09/androids-http-clients.html
in the finally clause just call disconnect. Example code ..
try {
HttpURLConnection locConn = (HttpURLConnection) locurl.openConnection();
//URL url = locConn.getURL();
locConn.setRequestProperty("Authorization", basicAuth);
locConn.setRequestMethod("GET");
locConn.setRequestProperty("Content-Type", "application/json");
locConn.setRequestProperty("X-Myauthtoken", userCredentials);
retc = locConn.getResponseCode();
reader = new BufferedReader(new InputStreamReader(locConn.getInputStream()));
String sessionK = null;
readVal = reader.readLine();
if (retc == 200) {
}
}catch (...)
{
//handle exception
}finally {
//disconnect here
locConn.disconnect();
}
HttpClient httpclient = new HttpClient();
GetMethod httpget = new GetMethod("http://www.myhost.com/");
try {
httpclient.executeMethod(httpget);
Reader reader = new InputStreamReader(httpget.getResponseBodyAsStream(), httpget.getResponseCharSet());
// consume the response entity
} finally {
httpget.releaseConnection();
}
The problem
In my Android app, I am trying to add a user to the /_security document of my CouchDB database via HTTP PUT. If I am trying to do this by authenticating my admin user using Cookie-authentication or by simply inserting the admin data into the url like in the following, I receive an error.
URL on which the PUT is directed (if not using Cookie-authentication):
http://admin_name:admin_password#url:port/databasename/_security
Error I receive in both cases:
Authentication error: Unable to respond to any of these
challenges: {} {"error":"unauthorized","reason":"You are not a db or
server admin."}
If I am doing this via command-line using curl, the user is inserted without any problems:
~$ curl -X PUT http://admin:pw#ip:port/databasename/_security -d '{"admins":{"names":[],"roles":[]},"members":{"names":["participant_1"],"roles":[]}}'
> {"ok":true}
My aproach
As soon as I authenticate using the "Authorization" option in one of my HTTP PUT's headers, athentication is no problem anymore.
private boolean putJSONWithAuthentication(String userName, String password, String json, String url) {
// url = http://url:port/databasename/_security
// json = {"admins":{"names":[],"roles":[]},"members":{"names":["participant_1"],"roles":[]}}
HttpClient client = new DefaultHttpClient();
HttpPut put = new HttpPut(url);
String authenticationData = userName+":"+password;
String encoding = Base64.encodeToString(authenticationData.getBytes(Charset.forName("utf-8")), Base64.DEFAULT);
try {
StringEntity stringEntity = new StringEntity(json,"utf-8");
put.setEntity(stringEntity);
put.setHeader("Content-type", "application/json; charset=utf-8");
put.setHeader("Accept", "application/json");
put.setHeader("Authorization", "Basic " + encoding);
HttpResponse response = client.execute(put);
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
String line = "";
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
return true;
} catch (IOException e) {
e.printStackTrace();
return false;
}
}
However, I am receiving this error:
> {"error":"bad_request","reason":"invalid_json"}
If I am inserting my user-JSON as a usual document using the above method, e.g., to http://url:port/databasename/new_document, the JSON is inserted without any errors. Consequently, I guess the JSON string should be formatted correctly.
Thus, my questions is, what am I missing here? It seems like I cannot authenticate and put data in the request body at once. How do I correctly insert a user to the /_security document of a database from code?
Instead of creating the basic auth header manually, can you try create it using UsernamePasswordCredentials, e.g.
HttpPut put = new HttpPut(url);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(userName, password);
put.addHeader( BasicScheme.authenticate(creds,"US-ASCII",false) );
...
HttpResponse response = client.execute(put);
I am using httpclient lib from apache. I managed to get an HttpResponse by sending a GET request to the server. Now what I am trying to do is to send that response that I got to a clientSocket output stream.
So basically I want to send whatever I received from the server to the open client connection. Since I am using HttpClient I get the response in the form of an HttpResponse object. I tried the following:
private void forwardRequest(String header, String url){
try {
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(url);
CloseableHttpResponse response;
//Adding the request headers to httpget
String lines[] = header.split("\\n");
for (String str : lines) {
String parts[] = str.split(":", 2);
httpget.addHeader(parts[0], parts[1]);
}
HttpResponse respone;
response = httpclient.execute(httpget);
//It works till here I can read from the response and print out the html page
//But after this I don't know how to send it to client
OutputStream bos = clientSocket.getOutputStream();
PrintWriter pw = new PrintWriter(bos);
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
while ((line = rd.readLine()) != null) {
pw.println(line);
//bos.write(line.getBytes()); //This also doesn't work
}
response.close();
}
Also clientSocket is a global variable which is associcated with a ServerSocket like:
clientSocket = serverSocket.accept();
I don't expect a full solution. Just point me in the right direction.. Thanks a ton!
EDIT:
I tried the following based on what EJP suggested.. It's still not working. I was wondering if it was correctly implemented?
int portNumber = 8012; // port on which the program listens
ServerSocket serverSocket =
new ServerSocket(portNumber); //the socket at which the program listens
Socket clientSocket = serverSocket.accept(); //clientSocket of the program
Socket toServer = new Socket("localhost", 8089); //proxy server to which program connects
PrintWriter out =
new PrintWriter(toServer.getOutputStream(), true);
PrintWriter outClient =
new PrintWriter(clientSocket.getOutputStream(), true);
BufferedReader in = new BufferedReader(
new InputStreamReader(clientSocket.getInputStream()));
BufferedReader inServer = new BufferedReader(
new InputStreamReader(toServer.getInputStream()));
) {
String inputLine;
while ((inputLine = in.readLine()) != null) {
out.println(inputLine); //Writing to proxy server
outClient.println(inServer.readLine()); //writing back to original request sender
System.out.println(inputLine);
}
The client made an HTTP request, so it will be expecting an HTTP response. If the global clientSocket is just a raw TCP socket and not an HttpClient, then you need to add the HTTP response protocol header yourself.
You have the content from the server, you'll want to first return an HTTP response 200 OK, then empty line with carriage return + linefeed (CR+LF), then Content-length: , then the document. If you are just proxying text documents, then you could convert to a string here, but otherwise, I would just pass the mime type, charset, and entity through as the raw bytes as the web server responded, that way you can proxy any document, including images or binary files.
It will look something like this:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: length
<html> ...
</html>
To pass the http headers through from the server:
HttpEntity entity = response.getEntity();
// technically you should check the HTTP response rather than assume it is a 200
int statusCode = httpResp.getStatusLine().getStatusCode();
if(statusCode != 200)
... // do something with non 200 responses ?
clientSocket.write("HTTP/1.1 200 OK\r\n");
Header[] responseHeaders = response.getAllHeaders();
for(Header header : responseHeaders) {
clientSocket.write(header.toString() + "\r\n");
}
clientSocket.write("\r\n"); // empty line required
// Use BufferedInputStream to deal in bytes
BufferedInputStream input = new BufferedInputStream(entity.getContent());
byte[] buf = new byte[8192];
int bytesRead;
while ((bytesRead = input.read(buf, 8192)) > 0) {
clientSocket.write(buf, bytesRead);
}
I say "something like this", don't take this literal, I doubt it compiles. I don't have dev station in front of me, but this is the general idea.
NOTE: Since you are using the Apache client lib, you should be able to use the specific HTTP client instead of writing the raw protocol. This will abstract the HTTP protocol away somewhat. I'll update the answer later if nobody else provides a better one.
If you're just forwarding requests and responses you don't have any need to engage in the HTTP protocol at all beyond the first line of the request. If the client knows you're the proxy you will get either a GET request with the full URL or else a CONNECT request ditto. All you have to do is connect to the target and then just copy bytes in both directions simultaneously.
I am running MVC 4 on my server and to save a bit of data for my users I figured I would enable GZip encoding, to do this I simply used:
(C#)
Response.AddHeader("Content-Encoding", "gzip");
Response.Filter = new GZipStream(Response.Filter, CompressionMode.Compress);
In my android application I use:
(Java)
String response = "";
DefaultHttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
try {
HttpResponse execute = client.execute(httpGet);
InputStream content = execute.getEntity().getContent();
BufferedReader buffer = new BufferedReader(
new InputStreamReader(content));
String s = "";
while ((s = buffer.readLine()) != null) {
response += s;
}
} catch (Exception e) {
e.printStackTrace();
}
return response;
When I use GZip the Java code nuts out and causes GC to run, I was never patient enough to wait for it to return.
When I took off GZip from the server it ran perfectly fine. The function to get the response returns straight away with no problem.
I tried adding this to the java code:
httpGet.addHeader("Accept-Encoding", "gzip");
With no success.
Question is, is there something I'm not getting? Can I not put the response in a stream if it is using GZip? Am I meant to use the stream and uncompress it after?
What am I doing wrong?
Instead of using
DefaultHttpClient client = new DefaultHttpClient();
you can use
ContentEncodingHttpClient client = new ContentEncodingHttpClient();
which is a subclass of DefaultHttpClient and supports GZIP content.
You need Apache HttpClient 4.1 for this.
If you have Apache HttpClient 4.2, you should use
DecompressingHttpClient client = new DecompressingHttpClient();
if you have Apache HttpClient 4.3, you should use the HttpClientBuilder
I am testing the uploading of files to a dataset on CKAN / datahub.io through a Java client of the API.
public String uploadFile()
throws CKANException {
String returned_json = this._connection.MultiPartPost("", "");
System.out.println("r: " + returned_json);
return returned_json;
}
and
protected String MultiPartPost(String path, String data)
throws CKANException {
URL url = null;
try {
url = new URL(this.m_host + ":" + this.m_port + path);
} catch (MalformedURLException mue) {
System.err.println(mue);
return null;
}
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
String fileName = "D:\\test.jpg";
FileBody bin = new FileBody(new File(fileName),"image/jpeg");
StringBody comment = new StringBody("Filename: " + fileName);
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("bin", bin);
reqEntity.addPart("comment", comment);
HttpPost postRequest = new HttpPost("http://datahub.io/api/storage/auth/form/2013-01-24T130158/test.jpg");
postRequest.setEntity(reqEntity);
postRequest.setHeader("X-CKAN-API-Key", this._apikey);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
System.out.println("status code: " + statusCode);
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
System.out.println("body: " + body);
} catch (IOException ioe) {
System.out.println(ioe);
} finally {
httpclient.getConnectionManager().shutdown();
}
return body;
}
2 responses I get to my POST request:
a 413 error ("request entity too large") when the jpeg I try to upload is 2.83 Mb. This disappears when I shrink the file to a smaller size. Is there a limit to file size uploads?
a 500 error ("internal server error"). This is where I am stuck. It might have to do with the fact that my dataset on datahub.io is not "datastore enabled"? (I see a disabled "Data API" button next to my resource files in the dataset, with a tooltip saying:
"Data API is unavailable for this resource as DataStore is disabled"
=> is it a possible reason for this 500 error? If so, how could I enable it from the client side? (pointers to Python code would be useful!)
Thx!
PS: the dataset I am using for testing purposes: http://datahub.io/dataset/testapi
Only someone with access to the exception log could tell you why the 500 is occurring.
However, I'd check your request is the same as what you'd get from the python client that was written alongside the datastore: https://github.com/okfn/ckanclient/blob/master/ckanclient/init.py#L546
You're sending the "bin" image buffer and "comment" file_key in your multipart request. Note the file_key must be changed for every upload, so add in a timestamp or something. And maybe you need to add in a Content-Type: for the binary.
I have been going through the same kind of troubles as the poster of this question. After quite a bit of trial and error, I came up with a solution to the problem. In my case, I had some control over the CKAN repository that I wanted to upload to. If you don't, your problem might be impossible to solve...
I assume you are using the 1.8 version of CKAN?
First of all, check whether the CKAN repository has been set up to allow file upload and if not, configure it to allow that. This can be done on the server using the steps posted here: http://docs.ckan.org/en/ckan-1.8/filestore.html#local-file-storage
The 413 error that you mentioned should be adressed next. This has to do with the general configuration of the server. In my case, the CKAN was hosted through nginx. I added a "client_max_body_size 100M" line to the nginx.conf file. See this post for instance: http://recursive-design.com/blog/2009/11/18/nginx-error-413-request-entity-too-large/
Then there is only the 500 error left. At the time of this writing, the api documentation of CKAN is still a little immature... It does indeed say that you have to build a request like you have made for file upload. However, this request is just to ask for permission for the file upload. If your credentials check out for file upload (not every user may be allowed to upload files), the response holds an object telling you where to send your file to... Because of the unclear api, you ended up merging these two requests.
The following scenario shows a follow up of two requests to handle the file upload. It might be that some steps in the scenario work out differently in your case, because of a repository that has been set up a little differently. If you get error messages, please be sure to check the response's body for clues!
Here is the authentication request that I used:
String body = "";
String generatedFilename=null;
HttpClient httpclient = new DefaultHttpClient();
try {
// create new identifier for every file, use time
SimpleDateFormat dateFormatGmt = new SimpleDateFormat("yyyyMMMddHHmmss");
dateFormatGmt.setTimeZone(TimeZone.getTimeZone("GMT"));
String date=dateFormatGmt.format(new Date());
generatedFilename=date +"/"+filename;
HttpGet getRequest = new HttpGet(this.CKANrepos+ "/api/storage/auth/form/"+generatedFilename);
getRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(getRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
throw new IllegalStateException("File reservation failed, server responded with code: "+statusCode+
"\n\nThe message was: "+body);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
Now, if all goes well, the server responds with a json object holding the parameters to use when doing the actual file upload. In my case, the object looked like:
{file_key:"some-filename-to-use-when-uploading"}
Be sure to check the json object though, as I'm given to understand that there may be custom ckan repositories that require more or different parameters.
These responses can then be used in the actual file upload:
File file = new File("/tmp/file.rdf");
String body = "";
HttpClient httpclient = new DefaultHttpClient();
try {
FileBody bin = new FileBody(file,"application/rdf+xml");
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("file", bin);
reqEntity.addPart("key", new StringBody(filename));
HttpPost postRequest = new HttpPost(this.CKANrepos+"/storage/upload_handle");
postRequest.setEntity(reqEntity);
postRequest.setHeader(CKANapiHeader, this.CKANapi);
HttpResponse response = httpclient.execute(postRequest);
int statusCode = response.getStatusLine().getStatusCode();
BufferedReader br = new BufferedReader(
new InputStreamReader((response.getEntity().getContent())));
String line;
while ((line = br.readLine()) != null) {
body += line;
}
if(statusCode!=200){
getWindow().showNotification("Upload Statuscode: "+statusCode,
body,
Window.Notification.TYPE_ERROR_MESSAGE);
}
}finally {
httpclient.getConnectionManager().shutdown();
}
as you can see, the file_key property has now been transformed into the simple 'key' property. I don't know why.
This will get your file uploaded. The response to this upload request will hold a json object telling you where the file got uploaded to. edit: actually it seems that my ckan responded with a simple html page to tell me that the file got uploaded... I had to parse the page to confirm that the file was uploaded correctly :(
In my case, the file was at
this.CKANrepos +"/storage/f/"+location
where location is the filename returned in the authentication phase.
In the previous code fragments:
//the location of your ckan repository, including /api and possibly version, e.g.
this.CKANrepos = "http://datahub.io/api/3/";
this.CKANapiHeader="X-CKAN-API-Key";
this.CKANapi = "your ckan api key here";