Display real time data from external website in Swing interface? - java

I've tried googling the web but every question seems to address web development. I'm simply wondering if it is even possible to fetch data from internet (game results and in game events) that is updated every second ,or every 10 second and so on ,from a website that's not mine and to display it in a Java desktop client with the Swing library interface? And if yes, what is the best method?
ThankYou

Yes, you can do it. You should use java.net package to work with network.
Data fetching depends on the site from which you are going to fetch data, for example:
If site have API, like Stack Overflow, you should use it.
If data is presented on the page, you can use parser like jsoup (if page is html, of course)

I get stock data when requested, rather than on a timer, but you can look at my code and see how I get the stock data.
Here's what the JPanel looks like. It's the panel on the right.
This is the HistoricalDataRunnable class.
package com.ggl.stock.picker.controller;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import javax.swing.SwingUtilities;
import com.ggl.stock.picker.model.Company;
import com.ggl.stock.picker.model.StockDay;
import com.ggl.stock.picker.model.StockHistory;
import com.ggl.stock.picker.model.StockPickerModel;
import com.ggl.stock.picker.view.StockPickerFrame;
public class HistoricalDataRunnable implements Runnable {
private static final String URLString =
"http://www.google.com/finance/historical?output=csv&q=";
private Company company;
private StockPickerFrame frame;
private StockPickerModel model;
public HistoricalDataRunnable(StockPickerFrame frame,
StockPickerModel model, Company company) {
this.frame = frame;
this.model = model;
this.company = company;
}
#Override
public void run() {
InputStream is = null;
BufferedReader br = null;
try {
String symbol = company.getStockSymbol();
URL url = new URL(URLString + symbol);
URLConnection hc = url.openConnection();
hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; "
+ "Intel Mac OS X 10.4; en-US; rv:1.9.2.2) "
+ "Gecko/20100316 Firefox/3.6.2");
is = hc.getInputStream();
br = new BufferedReader(new InputStreamReader(is));
processCSVFile(br);
} catch (MalformedURLException e) {
e.printStackTrace();
String message = e.getMessage();
message = "<html>" + message;
setMessageLabel(message);
} catch (IOException e) {
String message = e.getMessage();
message = "<html>" + message;
setMessageLabel(message);
} finally {
closeReaders(is, br);
}
}
private void processCSVFile(BufferedReader br) throws IOException {
String line = "";
int count = 0;
StockHistory history = new StockHistory(company);
while ((line = br.readLine()) != null) {
if (count > 0) {
StockDay stockDay = createStockDay(line);
if (stockDay != null) {
history.addStockDay(stockDay);
}
}
count++;
}
if (history.calculateNumbers()) {
model.addStockHistory(history);
addStockHistory();
setMessageLabel(" ");
} else {
String message = "<html>There is no data for "
+ company.getCompanyName();
setMessageLabel(message);
}
}
private StockDay createStockDay(String line) {
String[] parts = line.split(",");
if (parts[1].equals("-"))
return null;
double open = Double.valueOf(parts[1]);
double high = Double.valueOf(parts[2]);
double low = Double.valueOf(parts[3]);
double close = Double.valueOf(parts[4]);
long volume = Long.valueOf(parts[5]);
return new StockDay(parts[0], open, high, low, close, volume);
}
private void addStockHistory() {
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
frame.addStockHistory();
}
});
}
private void setMessageLabel(final String text) {
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
frame.setMessageLabel(text);
;
}
});
}
private void closeReaders(InputStream is, BufferedReader br) {
try {
if (br != null)
br.close();
if (is != null)
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
This class makes a URL connection with Google, and returns the historical stock information. This information is captured in a StockHistory instance, and stored in the StockPickerModel instance.
The User-Agent property is set to simulate a browser. Some web sites don’t allow programs to access their web server. By setting the User-Agent property, you can pretend to be a web browser. Your program should respect the web server, and not submit too many requests. How much is too many depends on the web server.
This class also updates the view. The only way we’ll know when the request is complete is when the HistoricalDataRunnable method receives a response from Google. It’s up to this class to update the model and the view.
Since this class is run in a separate thread, we have to call the SwingUtilities invokeLater method to execute any Swing GUI commands. That’s why the addStockHistory and setMessageLabel methods are enclosed in an invokeLater method.
This class displays any errors in the JLabel message. The stock might not be kept by Google. The stock may not have any stock day values. These error messages are displayed.
To see the rest of the code, take a look at my Stock Picker Using Java Swing article.

Related

Unable to run class in a separate thread/window

I am trying to run a multi-user chat client java programme as part of another java programme.
How do I implement it in such a way that I can open up the chat client from the main programme? I have attempted to start it using ProcessBuilder but it causes the whole programme to crash.
The class to start the chat client and the client client itself is shown below respectively
--------------------- Class to start chat client ---------------------
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
public class ChatCommand extends Command {
public static final String COMMAND_WORD = "chat";
public static final String MESSAGE_USAGE = COMMAND_WORD + ":\n" + "Opens up a separate chat programme\n\t"
+ "Example: " + COMMAND_WORD;
public static final String MESSAGE_SUCCESS = "Initialising chat!";
public static void main(String[] args) {
new ChatCommand();
}
public ChatCommand() {
try {
int result = compile("seedu.addressbook.communications.ChatClient");
System.out.println("javac returned " + result);
result = run("seedu.addressbook.communications.ChatClient");
} catch (IOException | InterruptedException ex) {
ex.printStackTrace();
}
}
public int run(String clazz) throws IOException, InterruptedException {
ProcessBuilder pb = new ProcessBuilder("java", clazz);
pb.redirectError();
pb.directory(new File("src"));
Process p = pb.start();
InputStreamConsumer consumer = new InputStreamConsumer(p.getInputStream());
consumer.start();
int result = p.waitFor();
consumer.join();
System.out.println(consumer.getOutput());
return result;
}
public int compile(String file) throws IOException, InterruptedException {
ProcessBuilder pb = new ProcessBuilder("javac", file);
pb.redirectError();
pb.directory(new File("src"));
Process p = pb.start();
InputStreamConsumer consumer = new InputStreamConsumer(p.getInputStream());
consumer.start();
int result = p.waitFor();
consumer.join();
System.out.println(consumer.getOutput());
return result;
}
public class InputStreamConsumer extends Thread {
private InputStream is;
private IOException exp;
private StringBuilder output;
public InputStreamConsumer(InputStream is) {
this.is = is;
}
#Override
public void run() {
int in = -1;
output = new StringBuilder(64);
try {
while ((in = is.read()) != -1) {
output.append((char) in);
}
} catch (IOException ex) {
ex.printStackTrace();
exp = ex;
}
}
public StringBuilder getOutput() {
return output;
}
public IOException getException() {
return exp;
}
}
public CommandResult execute() {
ChatClient cc = new ChatClient();
try {
cc.main(new String[]{"a", "b"});
} catch (Exception e) {
System.out.println("aaa");
}
commandHistory.addHistory(COMMAND_WORD);
return new CommandResult(MESSAGE_SUCCESS);
}
}
--------------------------- Chat Client --------------------------
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.Socket;
import javax.swing.JFrame;
import javax.swing.JOptionPane;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
import javax.swing.JTextField;
/* A simple Swing-based client for the chat server. Graphically
* it is a frame with a text field for entering messages and a
* textarea to see the whole dialog.
*
* The client follows the Chat Protocol which is as follows.
* When the server sends "SUBMITNAME" the client replies with the
* desired screen name. The server will keep sending "SUBMITNAME"
* requests as long as the client submits screen names that are
* already in use. When the server sends a line beginning
* with "NAMEACCEPTED" the client is now allowed to start
* sending the server arbitrary strings to be broadcast to all
* chatters connected to the server. When the server sends a
* line beginning with "MESSAGE " then all characters following
* this string should be displayed in its message area.
*/
public class ChatClient {
private BufferedReader in;
private PrintWriter out;
private JFrame frame = new JFrame("MediChat");
private JTextField textField = new JTextField(40);
private JTextArea messageArea = new JTextArea(8, 40);
/* Constructs the client by laying out the GUI and registering a
* listener with the textfield so that pressing Return in the
* listener sends the textfield contents to the server. Note
* however that the textfield is initially NOT editable, and
* only becomes editable AFTER the client receives the NAMEACCEPTED
* message from the server.
*/
public ChatClient() {
// Layout GUI
textField.setEditable(false);
messageArea.setEditable(false);
frame.getContentPane().add(textField, "North");
frame.getContentPane().add(new JScrollPane(messageArea), "Center");
frame.pack();
// Add Listeners
textField.addActionListener(new ActionListener() {
/* Responds to pressing the enter key in the textfield by sending
* the contents of the text field to the server. Then clear
* the text area in preparation for the next message.
*/
public void actionPerformed(ActionEvent e) {
out.println(textField.getText());
textField.setText("");
}
});
}
/* Prompt for and return the address of the server.
*/
private String getServerAddress() {
return JOptionPane.showInputDialog(
frame,
"Enter IP Address of the Server:",
"Welcome to MediChat!",
JOptionPane.QUESTION_MESSAGE);
}
/* Prompt for and return the desired screen name.
*/
private String getName() {
return JOptionPane.showInputDialog(
frame,
"Choose a screen name:",
"Screen name selection",
JOptionPane.PLAIN_MESSAGE);
}
/* Connects to the server then enters the processing loop.
*/
private void run() throws IOException {
// Make connection and initialize streams
String serverAddress = getServerAddress();
Socket socket = new Socket(serverAddress, 9001);
in = new BufferedReader(new InputStreamReader(
socket.getInputStream()));
out = new PrintWriter(socket.getOutputStream(), true);
// Process all messages from server, according to the protocol.
while (true) {
String line = in.readLine();
if (line.startsWith("SUBMITNAME")) {
out.println(getName());
} else if (line.startsWith("NAMEACCEPTED")) {
textField.setEditable(true);
} else if (line.startsWith("MESSAGE")) {
messageArea.append(line.substring(8) + "\n");
}
}
}
/**
* Runs the client as an application with a closeable frame.
*/
public static void main(String[] args) throws Exception {
ChatClient client = new ChatClient();
client.frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
client.frame.setVisible(true);
client.run();
}
}
You are seriously overcomplicating things:
don't make compiling your client code part of any Java class. Define a project setup in your IDE, or on the command line using gradle for example. Then use that to separately compile your classes whenever you change something. Running javac in your classes manually is seriously wrong!
and then, just make sure that all compiled class files are available on the classpath of your jvm. Don't bother to use reflection or anything else that is based on class names as raw strings.
most importantly: you use other classes by instantiating objects directly. The main method should only be used when you want to run a class standalone from the command line!
so instead of using class names as string, simply normally import the classes to use, and use new to create objects of them.
beyond that, separate your concerns. The client is the client, the server is the server. It is absolutely not a good idea to have the server start client instances. Meaning: rather create a third class, maybe called SetupTestEnvironment that first starts the server and a few clients for testing purposes.

How to give file as input and work in multiple threads?

I have this code to find out how to get the status code from a URL:
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
/**
* #author Crunchify.com
*
*/
class j {
public static void main(String args[]) throws Exception {
String[] hostList = { "http://example.com", "http://example2.com","http://example3.com" };
for (int i = 0; i < hostList.length; i++) {
String url = hostList[i];
String status = getStatus(url);
System.out.println(url + "\t\tStatus:" + status);
}
}
public static String getStatus(String url) throws IOException {
String result = "";
try {
URL siteURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection) siteURL
.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
int code = connection.getResponseCode();
result = Integer.toString(code);
} catch (Exception e) {
result = "->Red<-";
}
return result;
}
}
I have checked it for small input it works fine. But I have millions of domains which I need to scan. I have a file containing it.
I want to know how I can give file as an input to this code.
I want the code to work in Multiple Threads. Say Thread count should be more than 20000, so that my output will be faster.
How I can write the out to another file?
Kindly help me. If possible I would like to know which the Bandwidth Savvy method to do the same job. I want to make the code faster anyways. how I can do these thing with the code I have?
Java Version:
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
This does what you want:
Input list file (c://lines.txt)
http://www.adam-bien.com/
http://stackoverflow.com/
http://www.dfgdfgdfgdfgdfgertwsgdfhdfhsru.de
http://www.google.de
The Thread:
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.Callable;
public class StatusThread implements Callable<String> {
String url;
public StatusThread(String url) {
this.url = url;
}
#Override
public String call() throws Exception {
String result = "";
try {
URL siteURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection) siteURL.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
int code = connection.getResponseCode();
result = Integer.toString(code);
} catch (Exception e) {
result = "->Red<-";
}
return url + "|" + result;
}
}
And the main program:
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.stream.Stream;
public class CallableExample {
public static void main(String[] args) throws IOException {
// Number of threads
int numberOfThreads = 10;
// Input file
String sourceFileName = "c://lines.txt"; // Replace by your own
String targetFileName = "c://output.txt"; // Replace by your own
// Read input file into List
ArrayList<String> urls = new ArrayList<>();
try (Stream<String> stream = Files.lines(Paths.get(sourceFileName ))) {
stream.forEach((string) -> {
urls.add(string);
});
} catch (IOException e) {
e.printStackTrace();
}
// Create thread pool
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(numberOfThreads);
List<Future<String>> resultList = new ArrayList<>();
// Launch threads
for(String url : urls) {
StatusThread statusGetter = new StatusThread(url);
Future<String> result = executor.submit(statusGetter);
resultList.add(result);
}
// Use results
FileWriter writer;
writer = new FileWriter(targetFileName);
for (Future<String> future : resultList) {
try {
String oneResult = future.get().split("\\|")[0] + " -> " + future.get().split("\\|")[1];
// Print the results to the console
System.out.println(oneResult);
// Write the result to a file
writer.write(oneResult + System.lineSeparator());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
writer.close();
// Shut down the executor service
executor.shutdown();
}
}
Don't forget to:
Create your input file and point to it (c://lines.txt)
Change the number of threads to get the best result
You will have issues sharing a file across threads. Much better to read the file and then spawn a thread to process each record in the file.
Creating a thread is none trivial resource wise so a thread pool would be useful so threads can be reused.
Do you want all threads to write to a single file?
I would do that using a shared list between the threads and the writer. others may have a better idea.
How to do all this depends on Java version.
You can use the ExecutorService and set the thread number to use.
The ExecutorService instance will handle for your the threads management.
You just need to provide it the tasks to execute and invoking all tasks executions.
When all the task are performed you can get the result.
In the call() method of The Callable implementation we return a String with a separator to indicate the url and the response code of the request.
For example : http://example3.com||301, http://example.com||200, etc...
I have not written the code to read a file and store in another file the result of the tasks. You should not have great difficulty to implement it.
Here is the main class :
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class Main {
public static void main(String[] args) throws InterruptedException {
String[] hostList = { "http://example.com", "http://example2.com", "http://example3.com" };
int nbThreadToUse = Runtime.getRuntime().availableProcessors() - 1;
ExecutorService executorService = Executors.newFixedThreadPool(nbThreadToUse);
Set<Callable<String>> callables = new HashSet<Callable<String>>();
for (String host : hostList) {
callables.add(new UrlCall(host));
}
List<Future<String>> futures = executorService.invokeAll(callables);
for (Future<String> future : futures) {
try {
String result = future.get();
String[] keyValueToken = result.split("\\|\\|");
String url = keyValueToken[0];
String response = keyValueToken[1];
System.out.println("url=" + url + ", response=" + response);
} catch (ExecutionException e) {
e.printStackTrace();
}
}
executorService.shutdown();
}
}
Here is UrlCall, the Callable implementation to perform a call to the url.
UrlCall takes in its constructor the url to test.
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.Callable;
public class UrlCall implements Callable<String> {
private String url;
public UrlCall(String url) {
this.url = url;
}
#Override
public String call() throws Exception {
return getStatus(url);
}
private String getStatus(String url) throws IOException {
try {
URL siteURL = new URL(url);
HttpURLConnection connection = (HttpURLConnection) siteURL.openConnection();
connection.setRequestMethod("HEAD");
connection.connect();
int code = connection.getResponseCode();
return url + "||" + code;
} catch (Exception e) {
//FIXME to log of course
return url + "||exception";
}
}
}
I agree with Thread pool approach exposed here.
Multi-threading consists in exploiting the time the others threads spend to wait (I guess int his case: the distant site response). It does not multiply processing power. Then about 10 threads seem reasonable (more depending on hardware).
An important point that seem to have been neglected in answer I read is that OP talk about millions of domains. Then I would discourage loading whole file in memory in a list iterated over afterwards. I would rather merge all in a single loop (file reading), instead of 3 (read, ping, write).
stream.forEach((url) -> {
StatusThread statusGetter = new StatusThread(url, outputWriter);
Future<String> result = executor.submit(statusGetter);
});
outputWriter would be a type with a synchronized method to write into an output stream.

Implements of thread in Android doesn't work

I'm writing a basic application in Android, the application will be connected to MySql server by quest in PHP, in Android Internet connection have to make in diffrent thread, so I create class which implements Runnable interface.
package com.company.opax.loginmysql;
import android.util.Log;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
/**
* Created by opax on 30.08.2015.
*/
public class HttpPostMethod implements Runnable{
private String fileInHost;
private ArrayList<PostParameters> postParameterses;
private StringBuffer postResult;
public HttpPostMethod(String fileInHost, ArrayList<PostParameters> postParameterses){
this.fileInHost = fileInHost;
this.postParameterses = new ArrayList<PostParameters>(postParameterses);
}
public String getResult() {
return postResult.toString();
}
#Override
public void run() {
try {
String urlParameters = generateParameters();
URLConnection conn = initializeUrlConnection();
OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream());
writer.write(urlParameters);
writer.flush();
String line;
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while ((line = reader.readLine()) != null) {
postResult.append(line);
}
writer.close();
reader.close();
} catch (IOException e) {
Log.e("Exception", this.getClass().getName() + " name: " + e.toString());
}
}
private URLConnection initializeUrlConnection() throws MalformedURLException {
URL url = new URL(fileInHost);
URLConnection conn;
try {
conn = url.openConnection();
conn.setDoOutput(true);
}catch(IOException e){
throw new MalformedURLException();
}
return conn;
}
private String generateParameters(){
StringBuffer finishPostQuery = new StringBuffer();
for(PostParameters p : postParameterses){
finishPostQuery.append(p.getNameParam());
finishPostQuery.append("=");
finishPostQuery.append(p.getValueParam());
finishPostQuery.append("&");
}
if(!finishPostQuery.toString().equals("login=seba&password=pass&"))
throw new AssertionError("blad generatora zapytania: " + finishPostQuery);
return finishPostQuery.toString();
}
}
and login class:
public class Login {
private User user;
private final String paramLogin = "login";
private final String paramPass = "password";
public Login(User user){
this.user = user;
}
public boolean tryLogin(){
try{
ArrayList<PostParameters> postParameterses = new ArrayList<>();
postParameterses.add(new PostParameters(paramLogin, user.getUserName()));
postParameterses.add(new PostParameters(paramPass, user.getUserPass()));
HttpPostMethod httpPostMethod = new HttpPostMethod("http://blaba.php", postParameterses);
httpPostMethod.run();
Log.i("bla", httpPostMethod.getResult());
} catch (Exception e) {
Log.i("Exception", e.toString());
}
return false;
}
}
I'm trying to connect in other thread, but I still have an error: 'android.os.NetworkOnMainThreadException'
I would be grateful for the all suggestion what I do wrong.
Instead of:
httpPostMethod.run();
do:
new Thread(httpPostMethod).start();
In case your login call failed for some reasons (timeout, wrong login), you should report that somehow to user - this is what AsyncTask class is for. It allows you to run background code in doInBackkground, and after network operation ends - in onPostExecute you can execute UI related stuff - like show errors/results.
I suggest you two things.
First use AsyncTask instead of pure java threads.
But the main advice is to use a library that make http requests.
I like to use Retrofit, it may handle all request and thread part for you, but there are others.

Parse HTML links from a google query

First the revised code which throws javax.swing.text.ChangedCharSetException:
import java.io.*;
import java.net.*;
public class Main
{
public static void main(String[] args) throws IOException, Exception
{
String query = "#pragma";
Socket s = new Socket("google.com",80);
PrintStream p = new PrintStream(s.getOutputStream());
p.print("GET /search?q="+query+" HTTP/1.0\r\n");
p.print("User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)\r\n");
p.print("Connection: close\r\n\r\n");
InputStreamReader in = new InputStreamReader(s.getInputStream());
BufferedReader buffer = new BufferedReader(in);
// String line;
//
// while ((line = buffer.readLine()) != null)
// { System.out.println(line); }
HTMLUtils.ParseLinks (buffer);
in.close();
}
}
import java.io.BufferedReader;
import java.io.IOException;
//import java.io.FileReader;
import java.io.Reader;
import java.util.List;
import java.util.ArrayList;
import javax.swing.text.html.parser.ParserDelegator;
import javax.swing.text.html.HTMLEditorKit.ParserCallback;
import javax.swing.text.html.HTML.Tag;
import javax.swing.text.html.HTML.Attribute;
import javax.swing.text.MutableAttributeSet;
public class HTMLUtils
{
private HTMLUtils() {}
public static List<String> extractLinks(Reader reader) throws IOException
{
final ArrayList<String> list = new ArrayList<String>();
ParserDelegator parserDelegator = new ParserDelegator();
ParserCallback parserCallback = new ParserCallback()
{
public void handleText(final char[] data, final int pos) { }
public void handleStartTag(Tag tag, MutableAttributeSet attribute, int pos)
{
if (tag == Tag.A) {
String address = (String) attribute.getAttribute(Attribute.HREF);
list.add(address);
}
}
public void handleEndTag(Tag t, final int pos) { }
public void handleSimpleTag(Tag t, MutableAttributeSet a, final int pos) { }
public void handleComment(final char[] data, final int pos) { }
public void handleError(final java.lang.String errMsg, final int pos) { }
};
parserDelegator.parse(reader, parserCallback, false);
return list;
}
public static void ParseLinks(BufferedReader buffer) throws Exception{
//FileReader reader = new FileReader("buffer");
List<String> links = HTMLUtils.extractLinks(buffer);
for (String link : links) {
System.out.println(link);
}
}
}
Notice that the user agent is for IE in this example.
Now I Have 3 problems:
How/can I pass the HTMLUtils.ParseLinks method a "raw buffer" instead of an HTML file she's expecting (I can write the buffer to a file but I guess that is unnecessary)
I don't know how to enter inverted commas (" ") inside the query statment in order to get the whole string i.e.: query=" "New York Yankees" "
Is it so complicated to get the User-Agent string from the host machine ???
link text
I have to say that it is imported class that I use and I don't really understand whats going on there. I'll try to understand when it will work [-8
THNX
Have a read of http://code.google.com/apis/ajaxsearch/, it's going to be a lot easier to get the data out of a JSON string than digging through acres of HTML. There's an open source Java class for digesting JSON: http://www.json.org/java/. Transferring the JSON will require a lot less bandwidth too!
If you want to do it in Java, you should consider using XPath to extract all links from the response. Therefore you first have to convert the response to XML. Then you can apply an XPath query like
//a/#href
to extract all href attributes for links. You can modify the query to only include links from the Google results and not from advertisements etc.
Here is another Tutorial to get you started.
Happy coding.
BTW: To avoid mistakes when you create your HTTP request and (even more important) to avoid unnecessary work, you could use a library like Apache Commons HTTPClient. This would reduce your work to:
HttpClient client = new HttpClient();
HttpMethod method = new GetMethod("http://www.google.com/search?q=" + query);
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
String response = new String(method.getResponseBody());

Download a file from the internet using java : How to authenticate?

Thanks to this thread How to download and save a file from Internet using Java?
I know how to download a file, now my problem is that I need to authenticate on the sever from which I'm dowloading. It's an http interface to a subversion server. Which field do I need to look up into ?
Using the code posted in the last comment, I get this exception:
java.io.IOException: Server returned HTTP response code: 401 for URL: http://myserver/systemc-2.0.1.tgz
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1305)
at java.net.URL.openStream(URL.java:1009)
at mypackage.Installer.installSystemc201(Installer.java:29)
at mypackage.Installer.main(Installer.java:38)
Thanks,
You extend the Authenticator class and register it. The javadocs at the link explain how.
I don't know if this works with the nio method that got the accepted answer to the question, but it for sure works for the old fashioned way that was the answer under that one.
Within the authenticator class implementation, you are probably going to use a PasswordAuthentication and override the getPasswordAuthentication() method of your Authenticator implementation to return it. That will be the class which is passed the user name and password you need.
Per your request, here is some sample code:
public static final String USERNAME_KEY = "username";
public static final String PASSWORD_KEY = "password";
private final PasswordAuthentication authentication;
public MyAuthenticator(Properties properties) {
String userName = properties.getProperty(USERNAME_KEY);
String password = properties.getProperty(PASSWORD_KEY);
if (userName == null || password == null) {
authentication = null;
} else {
authentication = new PasswordAuthentication(userName, password.toCharArray());
}
}
protected PasswordAuthentication getPasswordAuthentication() {
return authentication;
}
And you register it in the main method (or somewhere along the line before you call the URL):
Authenticator.setDefault(new MyAuthenticator(properties));
The usage is simple, but I find the API convoluted and kind of backwards for how you typically think about these things. Pretty typical of singleton design.
This is some code I wrote that fetches a website and displays the contents to System.out. It uses Basic authentication:
import java.net.*;
import java.io.*;
public class foo {
public static void main(String[] args) throws Exception {
URL yahoo = new URL("http://www.MY_URL.com");
String passwdstring = "USERNAME:PASSWORD";
String encoding = new
sun.misc.BASE64Encoder().encode(passwdstring.getBytes());
URLConnection uc = yahoo.openConnection();
uc.setRequestProperty("Authorization", "Basic " + encoding);
InputStream content = (InputStream)uc.getInputStream();
BufferedReader in =
new BufferedReader (new InputStreamReader (content));
String line;
while ((line = in.readLine()) != null) {
System.out.println (line);
}
in.close();
}
Problems with the above code:
This code isn't production-ready (but it gets the point across.)
The code yields this compiler warning:
foo.java:11: warning: sun.misc.BASE64Encoder is Sun proprietary API and may be removed in a future release
sun.misc.BASE64Encoder().encode(passwdstring.getBytes());
^ 1 warning
One really should use the Authenticator class, but for the life of me, I could not figure out how and I couldn't find any examples either, which just goes to show that the Java people don't actually like it when you use their language to do cool things. :-P
So the above isn't a good solution, but it does work and could easily be modified later.
Write your overriding class for Authenticator:
import java.net.Authenticator;
import java.net.PasswordAuthentication;
public class MyAuthenticator extends Authenticator {
private static String username = "";
private static String password = "";
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication (MyAuthenticator.username,
MyAuthenticator.password.toCharArray());
}
public static void setPasswordAuthentication(String username, String password) {
MyAuthenticator.username = username;
MyAuthenticator.password = password;
}
}
Write your main class:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.Authenticator;
import java.net.MalformedURLException;
import java.net.URL;
public class MyMain{
public static void main(String[] args) {
URL url;
InputStream is = null;
BufferedReader br;
String line;
// Install Authenticator
MyAuthenticator.setPasswordAuthentication("Username", "Password");
Authenticator.setDefault (new MyAuthenticator ());
try {
url = new URL("Your_URL_Here");
is = url.openStream(); // throws an IOException
br = new BufferedReader(new InputStreamReader(is));
while ((line = br.readLine()) != null) {
System.out.println(line);
}
} catch (MalformedURLException mue) {
mue.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} finally {
try {
if (is != null) is.close();
} catch (IOException ioe) {
// nothing to see here
}
}
}
}
Have you tried building your URL in the form http://user:password#domain/path?
I would suggest checking out HttpClient from apache http://hc.apache.org/httpclient-3.x/ it makes downloading/authenticating very easy
This open source library, http://spnego.sourceforge.net, also has some examples on how to use it's SpnegoHttpURLConnection class.
One of the constructors in the class allows you to pass-in a username and password.
Take a look at the class's java doc for the examples.

Categories