how do I search for existence of a word in a webpage given its url say "www.microsoft.com". Do I need to download this webpage to perform this search ?
You just need to make http request on web page and grab all its content after that you can search necessary words in it, below code might help you to do so.
public static void main(String[] args) {
try {
URL url;
URLConnection urlConnection;
DataOutputStream outStream;
DataInputStream inStream;
// Build request body
String body =
"fName=" + URLEncoder.encode("Atli", "UTF-8") +
"&lName=" + URLEncoder.encode("Þór", "UTF-8");
// Create connection
url = new URL("http://www.example.com");
urlConnection = url.openConnection();
((HttpURLConnection)urlConnection).setRequestMethod("POST");
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
urlConnection.setUseCaches(false);
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
urlConnection.setRequestProperty("Content-Length", ""+ body.length());
// Create I/O streams
outStream = new DataOutputStream(urlConnection.getOutputStream());
inStream = new DataInputStream(urlConnection.getInputStream());
// Send request
outStream.writeBytes(body);
outStream.flush();
outStream.close();
// Get Response
// - For debugging purposes only!
String buffer;
while((buffer = inStream.readLine()) != null) {
System.out.println(buffer);
}
// Close I/O streams
inStream.close();
outStream.close();
}
catch(Exception ex) {
System.out.println("Exception cought:\n"+ ex.toString());
}
}
i know how i would do this in theory - use cURL or some application to download it, store the contents into a variable, then parse it for whatever you need
Yes, you need to download page content and search inside it for what you want. And if it happens that you want to search the whole microsoft.com website then you should either write your own web crawler, use an existing crawler or use some search engine API like Google's.
Yes, you'll have to download the page, and, to make sure to get the complete content, you'll want to execute scripts and include dynamic content - just like a browser.
We can't "search" something on a remote resource, that is not controlled by us and no webservers offers a "scan my content" method by default.
Most probably you'll want to load the page with a browser engine (webkit or something else) and perform the search on the internal DOM structure of that engine.
If you want to do the search yourself, then obviously you have to download the page.
If you're planning on this approach, i recommend Lucene (unless you want a simple substring search)
Or you could have a webservice that does it for you. You could request the webservice to grep the url and post back its results.
You could use a search engine's API. I believe Google and Bing (http://msdn.microsoft.com/en-us/library/dd251056.aspx) have ones you can use.
Related
I'm currently calling a local .json file in my Android app using the following line
InputStream inputStream = context.getAssets().open("cyclist.json");
I simply want to switch it to pull the .json from a webservice instead. What is the best way to do this?
Please, please, don't reinvent the wheel.
Use existing libraries Volley by Google (video from I/O talk), Retrofit by Square, RoboSpice and countless others are there to serve you. Further, search before posting
Supposing you already set up a server to respond to requests, I would try something like this:
URL url = new URL("http://www.mydomain.com/slug");
URLConnection urlConnection = url.openConnection();
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
try {
readStream(in);
} finally {
in.close();
}
See URLConnection for details.
I know Android enforces limitations in downloading stuff from a server. You might have to execute the code in another thread, using the AsyncTask. Again, I'm not sure if this is required for your particular purpose.
I am trying to append some information to a text file kept on webserver using java using:
public class Main {
public static void main(String[] args) {
try {
URL url = new URL("http://www.abcd.com/info.txt");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if (connection != null) {
System.out.println("Established URL connection");
}
connection.setDoOutput(true);
connection.setRequestMethod("PUT");
connection.setDoInput(true);
connection.setUseCaches(false);
connection.setRequestProperty("Content-Type", "text/html");
System.out.println(connection.getOutputStream().toString());
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
writer.write("This is a sample text");
writer.close();
}
catch (Exception ex) {
ex.printStackTrace();
}
}
}
Neither the text file is not being updated nor getting any error.. The reason for doing this is - I have developed a small software and the updates for this will be kept on web site. If any user updates the data, this code will update the text file. This way I will be able to get the information of user who have updated.
As far as I know, you first need to get the data written in the file, to client, Using a GET call , then append the data, and the finally do a POST call to rewrite the file with appended data
You would have to make the changes at server side to do that. You cannot acheive the same using HttpURLConnection.
You can try using FTP if its feasible for you. In case of FTP you should download the file, append the text and upload the same again.
I'm a bit confused - you're attempting to open an HTTP connection to a file and modify it on the fly?
I feel like I might be missing something - plain HTTP doesn't support this. Can you imagine the nightmare it would be if everybody could go around overwriting everybody else's websites (without authentication, even, as your code seems to suggest)?
What you're doing here is calling PUT on the /info.txt resource with your text as the entity body. I'm fairly sure that never has and never will overwrite the corresponding file.
What you need to do is either go through a protocol that supports file writing (WebDav, FTP...) or write server-side code that accepts a content submission (through, for example, a POST or PUT call with an entity body on a specific resource), analyses that input and modify its local file system.
Again, I might be misunderstanding your question entirely, in which case I apologise if I come off as somewhat patronising.
I have a situation where a intermediate servlet needs to be introduced which will handle requests from existing project and redirect the manipulated response to either existing project or the new one. This servlet will act as an interface to login into the new project from some other application.
So currently I use the following code to get back response in jsp as an xml.
var jqxhr =$.post("http://abhishek:15070/abc/login.action",
{ emailaddress: "ars#gmail.com",
projectid: "123" },
function(xml)
{
if($(xml).find('isSuccess').text()=="true")
{
sessiontoken=$(xml).find('sessiontoken').text();
setCookie("abcsessionid", sessiontoken , 1);
setCookie("abcusername",e_add,1);
}
}
)
.error(function() {
if(jqxhr.responseText == 'INVALID_SESSION') {
alert("Your Session has been timed out");
window.location.replace("http://abhishek:15070/abc/index.html");
}else {
alert( jqxhr.responseText);
}
});
xml content
<Response>
<sessiontoken>334465683124</sessiontoken>
<isSuccess>true</isSuccess>
</Response>
but now I want the same thing to be done using servlet, is it possible?
String emailid=(String) request.getParameter("emailaddress");
String projectid=(String) request.getParameter("projectid");
Update
I just came up with something.
Is it possible to return back a html page with form (from servlet), whose on body load it will submit a form and on submission of this form it will receive the response xml which will get processed.
Use java.net.URLConnection or Apache HttpComponents Client. Then, parse the returned HTTP response with a XML tool like as JAXB or something.
Kickoff example:
String emailaddress = request.getParameter("emailaddress");
String projectid = request.getParameter("projectid");
String charset = "UTF-8";
String query = String.format("emailaddress=%s&projectid=%s",
URLEncoder.encode(emailaddress, charset),
URLEncoder.encode(projectid, charset));
URLConnection connection = new URL("http://abhishek:15070/abc/login.action").openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);
try {
connection.getOutputStream().write(query.getBytes(charset));
}
finally {
connection.getOutputStream().close();
}
InputStream response = connection.getInputStream();
// ...
See also:
Using java.net.URLConnection to fire and handle HTTP requests
HttpClient tutorial and examples
Actually, what you probably want is not an intermediate servlet at all. What you probably want is called a servlet filter and writing one is not particularly hard. I've written one in the past and I just started on a new one yesterday.
An article like this one or this one lays out pretty simply how you can use a servlet filter to intercept calls to specific URLs and then redirect or reject from there. If the incoming URL matches the pattern for the filter, it will get a shot at the request and response and it can then make a choice whether or not to pass it on to the next filter in line.
I don't know if all third party security solutions do it like this, but at least CAS seemed to be implemented that way.
I am trying to download an vcalendar using a java application, but I can't download from a specific link.
My code is:
URL uri = new URL("http://codebits.eu/s/calendar.ics");
InputStream in = uri.openStream();
int r = in.read();
while(r != -1) {
System.out.print((char)r);
r = in.read();
}
When I try to download from another link it works (ex: http://www.mysportscal.com/Files_iCal_CSV/iCal_AUTO_2011/f1_2011.ics). Something don't allow me to download and I can't figure out why, when I try with the browser it works.
I'd follow this example. Basically, get the response code for the connection. If it's a redirect (e.g. 301 in this case), retrieve the header location and attempt to access the file using that.
Simplistic Example:
URL uri = new URL("http://codebits.eu/s/calendar.ics");
HttpURLConnection con = (HttpURLConnection)uri.openConnection();
System.out.println(con.getResponseCode());
System.out.println(con.getHeaderField("Location"));
uri = new URL(con.getHeaderField("Location"));
con = (HttpURLConnection)uri.openConnection();
InputStream in = con.getInputStream();
You should check what that link actually provides. For example, it might be a page that has moved, which gives you back an HTTP 301 code. Your browser will automatically know to go and fetch it from the new URL, but your program won't.
You might want to try, for example, wireshark to sniff the actual traffic when you do the browser request.
I think too that there is a redirect. The browser downloads from ssl secured https://codebits.eu/s/calendar.ics. Try using a HttpURLConnection, it should follow redirects automatically:
HttpURLConnection con = (HttpURLConnection)uri.openConnection();
InputStream in = con.getInputStream();
I have a java application that sends text to a sql database on a server. Currently my java application takes the text, puts it into the url, then sends it to a php page on the server that takes it with GET and puts it in the database. that works fine to an extent, the problem is, that i need to be able to send lots of text, and i keep getting 414, uri to long errors. is there a better way to do this?
ok, i tried what you said, and read the tutorial, but something is not working. here is my code that i tried
public void submitText(String urls,String data) throws IOException{
URL url = new URL(urls);
URLConnection con = url.openConnection();
con.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
con.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
out.write(data);
out.flush();
}
submitText(server + "publicPB.php", "param=" + text);
here is my php code
$param = $_POST['param'];
$sql = "UPDATE table SET cell='{$param}' WHERE 1";
mysql_query($sql);
...
im pretty sure its not a problem with the php as the php worked fine with GET, and thats all i change with it, my problem i think is that im not 100% sure how to send data to it with the java
Use a POST instead of a GET and send the text as the request body. You can only pass so much data to a URL. E.g.:
// Assuming 'input' is a String and contains your text
URL url = new URL("http://hostname/path");
URLConnection con = url.openConnection();
con.setRequestProperty("Content-Type", "text/plain; charset=utf-8");
con.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(con.getOutputStream(), "UTF-8");
out.write(input);
out.close();
See Reading from and Writing to a URLConnection for more details.
Why don't you use POST to send data across to PHP page? GET does have a smaller limit of content.
Use POST requests, which do not have content length limits.
POST requests do not have length content limits and are much secure than GET requests ;)
If using SQL Server I would look into leveraging BCP. You can write the file and call BCP from within Java, and it will send the information directly to your database.