What I wanna do is get the content of this URL :
https://www.aviationweather.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&stationString=CYQB&hoursBeforeNow=2
and copy it to a file so I can parse it and use the elements.
Here is what I have so far :
package test;
import java.io.*;
import java.net.*;
import org.apache.commons.io.FileUtils;
public class JavaGetUrl {
#SuppressWarnings("deprecation")
public static void main(String[] args) throws FileNotFoundException {
URL u;
InputStream is = null;
DataInputStream dis;
String s = null;
try {
u = new URL(
"https://www.aviationweather.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&stationString=CYQB&hoursBeforeNow=2");
is = u.openStream(); // throws an IOException
dis = new DataInputStream(new BufferedInputStream(is));
while ((s = dis.readLine()) != null) {
System.out.println(s);
FileUtils.writeStringToFile(new File("input.txt"), s);
}
} catch (MalformedURLException mue) {
System.out.println("Ouch - a MalformedURLException happened.");
mue.printStackTrace();
System.exit(1);
} catch (IOException ioe) {
System.out.println("Oops- an IOException happened.");
ioe.printStackTrace();
System.exit(1);
} finally {
try {
is.close();
} catch (IOException ioe) {
}
}
}
}
The problem is that the content of s does not show up in input.txt.
If I replace s by any other strings it works. So I guess it's a problem with the data of s. Is it because it's xml?
Thank you all for the help.
The file is probably getting over-written.
You should use "append" mode to get file appended with data(from readLine).
public static void writeStringToFile(File file,
String data,
boolean append)
as you are already using apaches commons-io, you can also simply use
FileUtils.copyURLToFile(URL, File)
see https://commons.apache.org/proper/commons-io/javadocs/api-2.4/org/apache/commons/io/FileUtils.html#copyURLToFile(java.net.URL,%20java.io.File)
Related
I am trying to write a test for the writeMessage() method. But I have no idea how to start, since I need to test an OutputStream. This should be something like a small chat. It should read a message from console, write it to the text file and than print all messages that have been written to the file.
This is for a university project.
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.PrintStream;
public class ChatIntImplement implements ChatI {
public static String readMessage() throws IOException, NullPointerException{
InputStream is = System.in;
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String message = null;
try {
message = br.readLine();
}catch(IOException ex) {
System.err.println("couldn't write data (fatal)");
System.exit(0);
}
return message;
}
public static void messageToFile(String message) throws IOException {
try {
String filename = "savedMessage.txt";
OutputStream os = new FileOutputStream(filename, true);
PrintStream ps = new PrintStream(os);
ps.println(message);
}catch (FileNotFoundException ex) {
System.err.println("couldn't open file - fatal");
System.exit(0);
}
}
public static void showMessages() {
InputStream is = null;
try {
is = new FileInputStream("savedMessage.txt");
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String message;
while((message = br.readLine())!= null) {
System.out.println(message);
}
}catch(FileNotFoundException ex) {
System.err.println("couln't open file -fatal");
System.exit(0);
}catch(IOException e) {
System.err.println("couldn't read data (fatal)");
System.exit(0);
}
}
#Override
public void writeMessage(String message){
try {
messageToFile(message);
showMessages();
}catch (IOException e) {
System.err.println("couldn't write data (fatal");
System.exit(0);
}
}
#Override
public void exit() {
// TODO Auto-generated method stub
}
public static void main(String[] args) {
ChatIntImplement chat = new ChatIntImplement();
try {
String message = readMessage();
chat.writeMessage(message);
}catch (IOException e) {
System.err.println("couldn't write data (fatal");
System.exit(0);
}catch (NullPointerException npe) {
System.err.println("Du hast nichts eingegeben");
System.exit(0);
}
}
}
I think the problem you are having getting started is because of how you have structured your class. All of your methods are static, and each method's dependencies, such as the various input and output streams you are using, are created as needed inside each method. For instance, in your messageToFile(...) method:
OutputStream os = new FileOutputStream(filename, true);
A much better approach to your project would be design your class using a traditional object-oriented class design that makes use of a pattern called Dependency Injection
The idea here is that your class that implements your ChatI interface would use member-variables for your input and output streams and the other objects, which would be initialized in the constructor for the class. Then you could control, and most importantly, get access to those streams from within your unit tests - something like this perhaps:
#Test
public void givenAMessageString_When() {
String expectedInput="expectedString";
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ChatIntImplement chat = new ChatIntImplement(baos);
...
chat.writeMessage(expectedInput);
string output = new String(baos.toByteArray());
assertThat(output).equals(expectedInput);
}
Am using nio2 to read the external file in my desktop using eclipse. I am getting the exception for the following code.
"java.nio.file.NoSuchFileException: C:\Users\User\Desktop\JEE\FirstFolder\first.txt"
Kindly advise how to resolve it? Tried using command prompt also. Getting the same exception.
public class ReadingExternalFile {
public static void main(String[] args) {
Path p1= Paths.get("C:\\Users\\User\\Desktop\\FirstFolder\\first.txt");
System.out.println(p1.toString());
System.out.println(p1.getRoot());
try(InputStream in = Files.newInputStream(p1);
BufferedReader reader = new BufferedReader(new InputStreamReader(in)))
{
System.out.println("Inside try");
String line=null;
while((line=reader.readLine())!=null){
if (!line.equals("")) {
System.out.println(line);
}
//System.out.println(line);
}
} catch (IOException e) {
System.out.println( e);
}
}
}
I dont understand why you are using a Path object, you can simply make the file using the File object and just using the string as the path, and then wraping it in a file reader object then wrapping that in a buffered reader, the end should look something like this:
public static void main(String[] args) {
try {
File file = new File("C:\\Users\\User\\Desktop\\FirstFolder\\first.txt");
FileReader fr = new FileReader(file);
BufferedReader bfr = new BufferedReader(fr);
System.out.println(bfr.readLine());
bfr.close();
} catch (IOException e){
e.printStackTrace();
}
}
don't forget to close your streams after reading and writing, also use readable names (don't do what I've done, use meaningful names!)
Try below code hope this will help you.
Path p1= Paths.get("C:\\Users\\user\\Desktop\\FirstFolder\\first.txt");
try(
BufferedReader reader = Files.newBufferedReader(p1, Charset.defaultCharset()))
{
System.out.println("Inside try");
String line=null;
while((line=reader.readLine())!=null){
if (!line.equals("")) {
System.out.println(line);
}
//System.out.println(line);
}
} catch (IOException e) {
System.out.println( e);
}
Try this.
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
public class BufferedReaderExample {
public static void main(String[] args) {
BufferedReader br = null;
try {
String sCurrentLine;
br = new BufferedReader(new FileReader("C:\\Users\\User\\Desktop\\FirstFolder\\first.txt"));
while ((sCurrentLine = br.readLine()) != null) {
System.out.println(sCurrentLine);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null)br.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
public static void main(String[] args) {
try {
File file = new File("C:\\Users\\User\\Desktop\\FirstFolder\\first.txt");
FileReader freader = new FileReader(file);
BufferedReader bufreader = new BufferedReader(freader);
System.out.println(bufreader.readLine());
bufreader.close();
} catch (IOException e){
e.printStackTrace();
}
}
First of all, I am but a lowly web-programmer so have very little experience with actual programming.
I have been given a list of 30,000 urls and I am not going to waste my time clicking each one to check if they are valid - is there a way to read through the text file that they are in and have a program check each line?
The code I currently have is in java as really that's all I know so if there's a better language again, please let me know.
Here is what I have so far:
public class UrlCheck {
public static void main(String[] args) throws IOException {
URL url = new URL("http://www.google.com");
//Need to change this to make it read from text file
try {
InputStream inp = null;
try {
inp = url.openStream();
} catch (UnknownHostException ex) {
System.out.println("Invalid");
}
if (inp != null) {
System.out.println("Valid");
}
} catch (MalformedURLException exc) {
exc.printStackTrace();
}
}
}
First you read the file line by line using a BufferedReader and check each line. Below code should work. It is upto you to decide what to do when you encounter an invalid URL. You could just print it as I showed or write to another file.
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import java.rmi.UnknownHostException;
public class UrlCheck {
public static void main(String[] args) throws IOException {
BufferedReader br = new BufferedReader(new FileReader("_filename"));
String line;
while ((line = br.readLine()) != null) {
if(checkUrl(line)) {
System.out.println("URL " + line + " was OK");
} else {
System.out.println("URL " + line + " was not VALID"); //handle error as you like
}
}
br.close();
}
private static boolean checkUrl(String pUrl) throws IOException {
URL url = new URL(pUrl);
//Need to change this to make it read from text file
try {
InputStream inp = null;
try {
inp = url.openStream();
} catch (UnknownHostException ex) {
System.out.println("Invalid");
return false;
}
if (inp != null) {
System.out.println("Valid");
return true;
}
} catch (MalformedURLException exc) {
exc.printStackTrace();
return false;
}
return true;
}
}
The checkUrl method can be simplified as below as well
private static boolean checkUrl(String pUrl) {
URL url = null;
InputStream inp = null;
try {
url = new URL(pUrl);
inp = url.openStream();
return inp != null;
} catch (IOException e) {
e.printStackTrace();
return false;
} finally {
try {
if (inp != null) {
inp.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
You could just use httpURLConnection. If it is not valid you won't get anything back.
HttpURLConnection connection = null;
try{
URL myurl = new URL("http://www.myURL.com");
connection = (HttpURLConnection) myurl.openConnection();
//Set request to header to reduce load
connection.setRequestMethod("HEAD");
int code = connection.getResponseCode();
System.out.println("" + code);
} catch {
//Handle invalid URL
}
I am unsure of your experience but a multi-threaded solution is possible here. As you read through the text file store the urls in a thread-safe structure and allow a number of threads to go and attempt to open these connections. This will make for a more efficient solution as it may take a while to test the 30000 urls while you are reading them in.
Check out a producer-consumer example if you are unsure:
http://www.journaldev.com/1034/java-blockingqueue-example-implementing-producer-consumer-problem
public class UrlCheck {
public static void main(String[] args) {
try {
URL url = new URL("http://www.google.com");
//Open the Http connection
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
//Get the http response code
int responceCode = connection.getResponseCode();
if (responceCode == HttpURLConnection.HTTP_OK) //if the http response code is 200 OK so the url is valid
{
System.out.println("Valid");
} else //Else the url is not valid
{
System.out.println("Invalid");
}
} catch (MalformedURLException ex) {
System.out.println("Invalid");
} catch (IOException ex) {
System.out.println("Invalid");
}
}
}
import java.net.URL;
import java.io.*;
import java.net.MalformedURLException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Test {
public static void main(String args[]) {
try {
processHTMLFromLink(new URL("http://fwallpapers.com"));
} catch (MalformedURLException ex) {
Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
}
}
public static int processHTMLFromLink(URL url) {
InputStream is = null;
DataInputStream dis;
String line;
int count = 0;
try {
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
while ((line = in.readLine()) != null) {
System.out.println(line);
}
} catch (MalformedURLException mue) {
System.out.println(mue.toString());
} catch (IOException ioe) {
System.out.println(ioe.toString());
} finally {
try {
is.close();
} catch (IOException ioe) {
// nothing to see here
}
}
return count;
}
}
error:
java.io.IOException: Server returned HTTP response code: 403 for URL: http://fwallpapers.com
Exception in thread "main" java.lang.NullPointerException
at Test.processHTMLFromLink(Test.java:38)
at Test.main(Test.java:15)
Java Result: 1
It is working fine on browser. But I am getting null point exceptions. this code works fine with other links. can anyone help me out with this. How can I get content while i am getting 403 error.
This is an old post but if people wanted to know how this works.
a 403 means acces-denied.
There is a work around for this.
If you want to able to do this you have to set a user agant parameter to 'fool' the website
This is how my old method looked like:
private InputStream read() {
try {
return url.openStream();
}
catch (IOException e) {
String error = e.toString();
throw new RuntimeException(e);
}
}
Changed it to: (And it works for me!)
private InputStream read() {
try {
HttpURLConnection httpcon = (HttpURLConnection) url.openConnection();
httpcon.addRequestProperty("User-Agent", "Mozilla/4.0");
return httpcon.getInputStream();
} catch (IOException e) {
String error = e.toString();
throw new RuntimeException(e);
}
}
Your mistake is swallowing the exception.
When I run my code, I get an HTTP 403 - "forbidden". The web server won't allow you to do this.
My code works perfectly for http://www.yahoo.com.
Here's how I do it:
package url;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;
/**
* UrlReader
* #author Michael
* #since 3/20/11
*/
public class UrlReader {
public static void main(String[] args) {
UrlReader urlReader = new UrlReader();
for (String url : args) {
try {
String contents = urlReader.readContents(url);
System.out.printf("url: %s contents: %s\n", url, contents);
} catch (Exception e) {
e.printStackTrace();
}
}
}
public String readContents(String address) throws IOException {
StringBuilder contents = new StringBuilder(2048);
BufferedReader br = null;
try {
URL url = new URL(address);
br = new BufferedReader(new InputStreamReader(url.openStream()));
String line = "";
while (line != null) {
line = br.readLine();
contents.append(line);
}
} finally {
close(br);
}
return contents.toString();
}
private static void close(Reader br) {
try {
if (br != null) {
br.close();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
This is now a completely different question so I have edited your title.
According to your edit, you aren't getting null pointer exceptions, you are getting HTTP 403 status, which means 'Forbidden', which means you can't access that resource.
I tried this code to download a file from my company's ftp site. The file gets downloaded but has 0 kb in size. Any idea? Thanks a lot !
package org.kodejava.example.commons.net;<br/><br/>
import org.apache.commons.net.ftp.FTPClient;
import java.io.IOException;
import java.io.FileOutputStream;
public class FtpDownloadDemo {
public static void main(String[] args) {
FTPClient client = new FTPClient();
FileOutputStream fos = null;
try {
client.connect("ftp.domain.com");
client.login("admin", "secret");
String filename = "sitemap.xml";
fos = new FileOutputStream(filename);
client.retrieveFile("/" + filename, fos);
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (fos != null) {
fos.close();
}
client.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}