What I want to do is to open an internet page in my browser (chrome) and get the html source code of the page just opened with my java application.
I don't want to get the source code of an url, I want a program that connects to the browser and gets the html code of the page that is open.
For example, if I open youtube in my browser, I want my application to get the current pages html code (in that case youtube code). Sorry if my english is not very good.
You can do this:
import java.util.*;
public static void main(String[] args) {
Scanner input = new Scanner(System.in);
URL url;
InputStream is = null;
BufferedReader br;
String line;
try {
String urlInput = input.nextLine();
url = new URL(urlInput);
is = url.openStream(); // throws an IOException
br = new BufferedReader(new InputStreamReader(is));
while ((line = br.readLine()) != null) {
System.out.println(line);
}
} catch (MalformedURLException mue) {
mue.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
if (is != null) is.close();
} catch (IOException ioe) {
// nothing to see here
}
}
}
I got this from here: How do you Programmatically Download a Webpage in Java
Try this out:
You must pass in the URL as the argument and you'll have the HTML code
public static void main(String[] args) throws IOException {
URL u = null;
try {
u = new URL(args[0]);
} catch (MalformedURLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
BufferedReader in = new BufferedReader(new InputStreamReader(u.openStream()));
String line = null;
while((line = in.readLine()) != null){
System.out.print(line);
}
}
Related
Consider following
Code
private String url = "https://celestrak.org/NORAD/elements/resource.txt";
#Override
public Boolean crawl() {
try {
// Timeout is set to 20s
Connection connection = Jsoup.connect(url).userAgent(USER_AGENT).timeout(20 * 1000);
Document htmlDocument = connection.get();
// 200 is the HTTP OK status code
if (connection.response().statusCode() == 200) {
System.out.println("\n**Visiting** Received web page at " + url);
} else {
System.out.println("\n**Failure** Web page not recieved at " + url);
return Boolean.FALSE;
}
if (!connection.response().contentType().contains("text/plain")) {
System.out.println("**Failure** Retrieved something other than plain text");
return Boolean.FALSE;
}
System.out.println(htmlDocument.text()); // Here it print whole text file in one line
} catch (IOException ioe) {
// We were not successful in our HTTP request
System.err.println(ioe);
return Boolean.FALSE;
}
return Boolean.TRUE;
}
Output
SCD 1 1 22490U 93009B 16329.83043855 .00000228 00000-0 12801-4 0 9993 2 22490 24.9691 122.2579 0043025 337.9285 169.5838 14.44465946256021 TECHSAT 1B (GO-32) 1 25397U ....
I am trying to read an online-txt file (from https://celestrak.org/NORAD/elements/resource.txt). Problem is that while I print or save the body's text it prints whole online-txt file in one line. But I want to read it as splited by \n so that I can read it line by line. Am I making mistake while reading online-txt file?
I am using JSoup.
you can do it without using jsoup in the following manner:
public static void main(String[] args) {
String data;
try {
data = IOUtils.toString(new URL("https://celestrak.com/NORAD/elements/resource.txt"));
for (String line : data.split("\n")) {
System.out.println(line);
}
} catch (IOException e1) {
e1.printStackTrace();
}
}
the above code uses org.apache.commons.io.IOUtils
if adding the commons library is a issue you can use the below code:
public static void main(String[] args) {
URLReader reader;
try {
reader = new URLReader(new URL("https://celestrak.com/NORAD/elements/resource.txt"));
BufferedReader bufferedReader = new BufferedReader(reader);
String sCurrentLine;
while ((sCurrentLine = bufferedReader.readLine()) != null) {
System.out.println(sCurrentLine);
}
bufferedReader.close();
} catch (MalformedURLException e1) {
e1.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
Since the file is already delimited by line separator, we can simple take the input stream from URL to read the contents
String url = "https://celestrak.com/NORAD/elements/resource.txt";
List<String> text = new BufferedReader(new InputStreamReader(new URL(url).openStream())).lines().collect(Collectors.toList());
To convert to a String
String content = new BufferedReader(new InputStreamReader(new URL(url).openStream())).lines()
.collect(Collectors.joining(System.getProperty("line.separator")));
My query is how to change how to change address in URL (http://localhost:8080/HELLO_WORLD). I change HELLO_WORLD to desire word.
#Override
public Response serve(IHTTPSession session) {
String answer = "";
BufferedReader reader = null;
try {
reader = new BufferedReader(
new InputStreamReader(appContext.getAssets().open("block.html")));
// do reading, usually loop until end of file reading
String mLine;
while ((mLine = reader.readLine()) != null) {
//process line
answer += mLine;
}
} catch (IOException e) {
//log the exception
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
//log the exception
Log.d("BABAR", "EXception occured in serve()");
}
}
}
return newFixedLengthResponse(answer);
}
please suggest me how to change
I don´t know if this is what you want, but you can try.
You have to follow the steps:
1- Create a local to store your server files;
2-Then change the response in the class that is implementing the NanoHttp server to something like this:
#Override
public Response serve(IHTTPSession session) {
String answer = "";
try{
FileReader filereader = new FileReader(contextoMain.local(localyourstorethefiles)+"/yourfolder/yourfile.html");
BufferedReader reader = new BufferedReader(filereader);
String line = "";
while ((line = reader.readLine()) != null) {
answer += line;
}
reader.close();
}catch(IOException ioe) {
Log.w("Httpd", ioe.toString());
}
return newFixedLengthResponse(answer);
}
3 - Then, call the localhost:8080 without putting the 8080/yourfolder/yourfile
I've been trying to figure out how to read a HttpURLConnection. According to this example: http://www.vogella.com/tutorials/AndroidNetworking/article.html , the following code should work. However, readStream never fires, and I'm not logging any lines.
I do get that the InputStream is passed through the buffer and all, but for me the logic breaks down in the readStream method, and then mostly the empty string 'line' and the while statement. What exactly is happening there / should happen there, and how would I be able to fix it? Also, why do I have to create the url in the Try statement? It gives back a Unhandled Exception; java.net.MalformedURLException.
Thanks in advance!
static String SendURL(){
try {
URL url = new URL("http://www.google.com/");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
readStream (con.getInputStream());
} catch (Exception e) {
e.printStackTrace();
}
return ("Done");
}
static void readStream(InputStream in) {
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(in));
String line = "";
while ((line = reader.readLine()) != null) {
Log.i("Tag", line);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
There are a bunch of things wrong with the code I posted in the question. Here is a working example:
public class GooglePlaces extends AsyncTask {
public InputStream inputStream;
public GooglePlaces(Context context) {
String url = "https://www.google.com";
try {
HttpRequest httpRequest = requestFactory.buildGetRequest(new GenericUrl(url));
HttpResponse httpResponse = httpRequest.execute();
inputStream = httpResponse.getContent();
} catch (IOException e) {
e.printStackTrace();
}
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder builder = new StringBuilder();
try {
for (String line = null; (line = bufferedReader.readLine()) != null;) {
builder.append(line).append("\n");
Log.i("GooglePlacesTag", line);
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
It appears you are not connecting your HTTPUrlClient try con.connect()
I am trying to develop a small web crawler, which downloads the web pages and search for links in a specific section. But when i am running this code, links in "href" tag are getting shortened. like :
original link : "/kids-toys-action-figures-accessories/b/ref=toys_hp_catblock_actnfigs?ie=UTF8&node=165993011&pf_rd_m=ATVPDKIKX0DER&pf_rd_s=merchandised-search-4&pf_rd_r=267646F4BB25430BAD0D&pf_rd_t=101&pf_rd_p=1582921042&pf_rd_i=165793011"
turned into : "/kids-toys-action-figures-accessories/b?ie=UTF8&node=165993011"
can anybody help me please. below is my code :
package test;
import java.io.*;
import java.net.MalformedURLException;
import java.util.*;
public class myFirstWebCrawler {
public static void main(String[] args) {
String strTemp = "";
String dir="d:/files/";
String filename="hello.txt";
String fullname=dir+filename;
try {
URL my_url = new URL("http://www.amazon.com/s/ref=lp_165993011_ex_n_1?rh=n%3A165793011&bbn=165793011&ie=UTF8&qid=1376550433");
BufferedReader br = new BufferedReader(new InputStreamReader(my_url.openStream(),"utf-8"));
createdir(dir);
while(null != (strTemp = br.readLine())){
writetofile(fullname,strTemp);
System.out.println(strTemp);
}
System.out.println("index of feature category : " + readfromfile(fullname,"Featured Categories"));
} catch (Exception ex) {
ex.printStackTrace();
}
}
public static void createdir(String dirname)
{ File d= new File(dirname);
d.mkdirs();
}
public static void writetofile(String path, String bbyte)
{
try
{
FileWriter filewriter = new FileWriter(path,true);
BufferedWriter bufferedWriter = new BufferedWriter(filewriter);
bufferedWriter.write(bbyte);
bufferedWriter.newLine();
bufferedWriter.close();
}
catch(IOException e)
{System.out.println("Error");}
}
public static int readfromfile(String path, String key)
{
String dir="d:/files/";
String filename="hello1.txt";
String fullname=dir+filename;
linksAndAt[] linksat=new linksAndAt[10];
BufferedReader bf = null;
try {
bf = new BufferedReader(new FileReader(path));
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
String currentLine;
int index =-1;
try{
Runtime.getRuntime().exec("cls");
while((currentLine = bf.readLine()) != null)
{
index=currentLine.indexOf(key);
if(index>0)
{
writetofile(fullname,currentLine);
int count=0;
int lastIndex=0;
while(lastIndex != -1)
{
lastIndex=currentLine.indexOf("href=\"",lastIndex);
if(lastIndex != -1)
{
lastIndex+="href=\"".length();
StringBuilder sb = new StringBuilder();
while(currentLine.charAt(lastIndex) != '\"')
{
sb.append(Character.toString(currentLine.charAt(lastIndex)));
lastIndex++;
}
count++;
System.out.println(sb);
}
}
System.out.println("\n count : " + count);
return index;
}
}
}
catch(FileNotFoundException f)
{
f.printStackTrace();
System.out.println("Error");
}
catch(IOException e)
{try {
bf.close();
} catch (IOException e1) {
e1.printStackTrace();
}}
return index;}
}
This feels to me like a situation where the server app is responding differently to requests from your desktop browser and your Java-based crawler. That could be because your browser is passing cookies in its requests which your Java-based crawler is not (such as session-persisting cookies), or it could be because your desktop browser passes a different User-Agent header than your crawler does, or it could be because other request headers are different between your desktop browser and your Java crawler.
When writing crawling apps, this is one of the biggest issues one runs into: it's easy to forget that the same URL requested by different clients won't always respond with the same code. Not sure if that's what's happening to you here, but it's very common.
I used following method to write data to a file in one android application
private void writeFileToInternalStorage() {
String eol = System.getProperty("line.separator");
BufferedWriter writer = null;
try{
writer = new BufferedWriter(new OutputStreamWriter(openFileOutput("myFile.txt", MODE_WORLD_WRITEABLE|MODE_WORLD_READABLE)));
writer.write("Hello world!");
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
if (writer != null)
{
try
{
writer.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
}
Then I tried to read that file from another android application using this method
private void readFileFromInternalStorage(){
String eol = System.getProperty("line.separator");
BufferedReader input = null;
try
{
input = new BufferedReader(new InputStreamReader(openFileInput("myFile1.txt")));
String line;
StringBuffer buffer = new StringBuffer();
while ((line = input.readLine()) != null)
{
buffer.append(line + eol);
}
TextView tv = (TextView) findViewById(R.id.textView);
tv.setText(buffer.toString().trim());
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
if (input != null)
{
try
{
input.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
}
Second method can't read the file. I added read write permissions also, but it shows only blank screen. What can be the error and how can I correct that ??. I'm new to Android programming and need your help.
Thanks!
The problem is
openFileOutput("myFile.txt", MODE_WORLD_WRITEABLE|MODE_WORLD_READABLE))
The documentation says:
This file is written to a path relative to your app within the
So the case is you are writing file in path relative to application 1 and trying to read it from
path relative to application 2.
You should be able to call Environment.getExternalStorageDirectory() to get the root path to the SD card and use that to create a FileOutputStream. From there, just use the standard java.io routines.
Look below snippet to write file to SD card.
private void writeToSDCard() {
try
{
File file = new File(Environment.getExternalStorageDirectory(),
"filename");
BufferedWriter writer = new BufferedWriter(new FileWriter(file));
writer.write("Hello World");
writer.close();
} catch (IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Look below snippet to read file saved on SD card.
private void readFileFromSDCard() {
File directory = Environment.getExternalStorageDirectory();
// Assumes that a file article.rss is available on the SD card
File file = new File(directory + "/article.rss");
if (!file.exists()) {
throw new RuntimeException("File not found");
}
Log.e("Testing", "Starting to read");
BufferedReader reader = null;
try {
reader = new BufferedReader(new FileReader(file));
StringBuilder builder = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
builder.append(line);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Your best bet is to place it into the scdcard into something like /sdcard/Android/data/package/shared/