I have written a test web crawler class that attempts to search Google, as shown:
public class WebCrawler {
String query;
public WebCrawler(String search)
{
query = search;
}
public void connect()
{
HttpURLConnection connection = null;
try
{
String url = "http://www.google.com/search?q=" + query;
URL search = new URL(url);
connection = (HttpURLConnection)search.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.setDoInput(true);
connection.setUseCaches(false);
connection.setAllowUserInteraction(false);
connection.connect();
BufferedReader read = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while((line = read.readLine())!=null)
{
System.out.println(line);
}
read.close();
}
catch(MalformedURLException e)
{
e.printStackTrace();
}
catch(ProtocolException e)
{
e.printStackTrace();
}
catch(IOException e)
{
e.printStackTrace();
}
finally
{
connection.disconnect();
}
}
}
When I try to run it with a test query "test" though, I get a HTTP response 403 error-- what am I missing? This is my first time doing any networking stuff with Java.
403 == forbidden, which makes sense because you're a robot trying to access a part of google that they don't want robots accessing. Google's robots.txt pretty clearly specifies that you shouldn't be scraping /search.
Google provides a search API which allows 100 queries per day. They provide libraries and examples of how to interface with it in most languages, including Java. More than that, you've gotta pay.
Related
I have the below method to check an incoming url is valid or not and based on this i need to redirect to error page.
/*
* This method is used to validate the URL and return a response
*/
private static int isValidUrl(String qrUrl) {
int code = 0;
try {
URL url = new URL(qrUrl);
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setRequestMethod("GET");
connection.connect();
code = connection.getResponseCode();
logger.info("Response code is {} for the url", code);
} catch(MalformedURLException e) {
logger.error("A MalformedURLException Occured in QR Code servlet during URL Validation", e);
} catch (IOException e) {
logger.error("An IOException Occured in QR Code servlet during URL Validation", e);
}
return code;
}
The below line of code is giving medium security vulnerability(Description:
This web server request could be used by an attacker to expose internal services and filesystem) during code scan. How can I fix it?
In my project i am verifying the links for few landing pages from an excel sheet using Selenium and Java. Below is the code i am using:
public static void verifyConnection(String linkUrl) throws IOException {
URL url = new URL(linkUrl);
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try {
if (urlConnection.getResponseCode() == 200) {
System.out.println("resp 200");
} else {
System.out.println("resp error");
}
} catch (Exception e) {
e.printStackTrace();
} finally {
urlConnection.disconnect();
}
}
The problem here is, when the linkUrl is something like https://www.google.com/ the response works fine, but when i do the same for Spanish version of our site with links like https://es.google.com/ or https://espanol.google.com/ i get Connection timed out exception.
Could someone advise, what i am missing?
Note: The above google links may not be valid, it is just to explain my scenario.
I have a Java servlet web server and an android application that communicate with each other. What is the proper way of passing a custom object (EX: User object) from server to android.
Example:
User logs in through android device, the users email and password are sent to the server for authentication. Server now needs to send back to the android device the User object.
If you can attach an example or link to tutorial that would be great.
https://www.youtube.com/watch?v=Gyaay7OTy-w
make a class that extends AsyncTask
implement override methods (Alt+Insert)
In do in background:
Make an http url connection
try {
URL url=new URL(urladdress);
HttpURLConnection con= (HttpURLConnection)url.openConnection();
con.setRequestMethod("GET");
con.setRequestMethod("POST")
con.setDoOutput(true);
con.setDoInput(true);
return con;
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
now download the data from this connection
if(con==null) {
Toast.makeText(ctx,"Connection is null",Toast.LENGTH_LONG).show();
}
try{
InputStream is = new BufferedInputStream(con.getInputStream());
BufferedReader br = new BufferedReader(new InputStreamReader(is));
String line;
StringBuffer jsonData= new StringBuffer();
while ((line=br.readLine())!=null)
{
jsonData.append(line+"/n");
}
br.close();
is.close();
return jsonData.toString();
} catch (IOException e) {
e.printStackTrace();
}
now parse this jsonData
try{
JSONArray ja = new JSONArray(jsonData);
JSONObject jo = null;
//parse Data
}
In onPostExecute call the required java file which will receive data
write a script that convert data recived from your server and send data in json format
Note- these code belongs three different classes so don't try copy/paste
Use Retrofit, is quite simple and straightforward.
Retrofit has given almost all the API's to make server call and to receive response. Internally they also use GSON to do the parsing. you can go through the official website you will get more info and a tutorial.
Hope it helps
I'm building a project in which I want a method to make a simple http GET request in order to send two variables to an website via URL.
In a normal java project I would likely use java.net or apache and solve the issue in a matter of minutes. In JavaME, due to my lack of experience I'm not really being able to fulfill the task.
Basically what I want to do is having an url like google.com/index.php?v1=x&v=y
being able to do a get request in order to send those variables via URL.
Any tips?
Here's an example of how you could do something like that.
HttpConnection connection = null;
InputStream inputstream = null;
String url = null;
StringBuffer dataReceived = null;
url = "http://www.google.com/index.php?v1=x&v=y";
dataReceived = new StringBuffer();
try {
connection = (HttpConnection) Connector.open(url);
connection.setRequestMethod(HttpConnection.GET);
connection.setRequestProperty("Content-Type", "text/plain");
connection.setRequestProperty("Connection", "close");
if (connection.getResponseCode() == HttpConnection.HTTP_OK) {
inputstream = connection.openInputStream();
int ch;
while ((ch = inputstream.read()) != -1 ) {
dataReceived.append((char) ch);
}
} else {
// Connection not ok
}
} catch (Exception e) {
// Something went wrong
} finally {
if (inputstream != null) {
try {
inputstream.close();
} catch (Exception e) {
}
}
if (connection != null) {
try {
connection.close();
} catch (Exception e) {
}
}
}
Note: I didn't test this specific code. I just edited some code I had lying around from a previous project of mine, so you may need to fix a few errors.
I'm writing a client j2me app that connects to a php based api using a post method. For security purposes the app and the api should interact within a session with a 30 minutes timeout. My problem is that, the application user has to keep on logging in even when the session time out is not yet done. My php code is fine because I've tested it on a browser and it works fine; however but the application fails and I have to keep on logging in. Might I be missing something? These are the headers am sending to the server using my Java app.
String phonenumber = this.phonenumberTextbox.getString();
String password = this.passwordTextBox.getString();
HttpConnection httpConn = null;
String url = "http://192.168.56.1/?action=login-user";
InputStream is;
OutputStream os;
is = null;
os = null;
String sReply = "";
boolean loggedIn = false;
try {
// Open an HTTP Connection object
httpConn = (HttpConnection) Connector.open(url);
// Setup HTTP Request to POST
httpConn.setRequestMethod(HttpConnection.POST);
httpConn.setRequestProperty("User-Agent", "Profile/MIDP-1.0 Confirguration/CLDC-1.0");
httpConn.setRequestProperty("Accept_Language", "en-US");
//Content-Type is must to pass parameters in POST Request
httpConn.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
os = httpConn.openOutputStream();
String params;
params = "phonenumber=" + phonenumber + "&password=" + password;
os.write(params.getBytes());
// Read Response from the Server
StringBuffer sb = new StringBuffer();
is = httpConn.openDataInputStream();
int chr;
while ((chr = is.read()) != -1) {
sb.append((char) chr);
}
// Web Server just returns stuff
sReply = sb.toString();
} catch (IOException ex) {
System.out.println("Cannot find server");
} finally {
if (is != null) {
try {
is.close();
} catch (IOException ex) {
}
}
if (os != null) {
try {
os.close();
} catch (IOException ex) {
}
}
if (httpConn != null) {
try {
httpConn.close();
} catch (IOException ex) {
}
}
}
// do something with sReply
Default practice in PHP these days** when dealing for sessions if for PHP to generate a cookie for the client to store. Your java client must store this cookie (called phpsessid by default, but any PHP programmer worth their salt will change the cookie name), and present it in the HTTP headers on subsequent requests.
I'm not very familiar with the HTTP support in Java, but if it's anything like curl it will include options for setting and retrieving cookies.
** It used to be possible to pass session tokens by a query string (a GET parameter) when making the request, but as this was horribly insecure and leaked potentially sensitive information, it's never used any more and may even no longer be available as an option.