I have created a web scraper which brings the market data of share rates from the website of stock exchange. www.psx.com.pk in that site there is a hyperlink of Market Summary. From that link I have to scrap the data. I have created a program which is as follows.
package com.market_summary;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.Iterator;
import java.util.Locale;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
public class ComMarket_summary {
boolean writeCSVToConsole = true;
boolean writeCSVToFile = true;
boolean sortTheList = true;
boolean writeToConsole;
boolean writeToFile;
public static Document doc = null;
public static Elements tbodyElements = null;
public static Elements elements = null;
public static Elements tdElements = null;
public static Elements trElement2 = null;
public static String Dcomma = ",";
public static String line = "";
public static ArrayList<Elements> sampleList = new ArrayList<Elements>();
public static void createConnection() throws IOException {
System.setProperty("http.proxyHost", "191.1.1.202");
System.setProperty("http.proxyPort", "8080");
String tempUrl = "http://www.psx.com.pk/index.php";
doc = Jsoup.connect(tempUrl).get();
System.out.println("Successfully Connected");
}
public static void parsingHTML() throws Exception {
File fold = new File("C:\\market_smry.csv");
fold.delete();
File fnew = new File("C:\\market_smry.csv");
for (Element table : doc.getElementsByTag("table")) {
for (Element trElement : table.getElementsByTag("tr")) {
trElement2 = trElement.getElementsByTag("td");
tdElements = trElement.getElementsByTag("td");
FileWriter sb = new FileWriter(fnew, true);
if (trElement.hasClass("marketData")) {
for (Iterator<Element> it = tdElements.iterator(); it.hasNext();) {
if (it.hasNext()) {
sb.append("\r\n");
}
for (Iterator<Element> it2 = trElement2.iterator(); it.hasNext();) {
Element tdElement2 = it.next();
final String content = tdElement2.text();
if (it2.hasNext()) {
sb.append(formatData(content));
sb.append(" | ");
}
}
System.out.println(sb.toString());
sb.flush();
sb.close();
}
}
System.out.println(sampleList.add(tdElements));
}
}
}
private static final SimpleDateFormat FORMATTER_MMM_d_yyyy = new SimpleDateFormat("MMM d, yyyy", Locale.US);
private static final SimpleDateFormat FORMATTER_dd_MMM_yyyy = new SimpleDateFormat("dd-MMM-YYYY", Locale.US);
public static String formatData(String text) {
String tmp = null;
try {
Date d = FORMATTER_MMM_d_yyyy.parse(text);
tmp = FORMATTER_dd_MMM_yyyy.format(d);
} catch (ParseException pe) {
tmp = text;
}
return tmp;
}
public static void main(String[] args) throws IOException, Exception {
createConnection();
parsingHTML();
}
}
Now, the problem is when I execute this program it should create a .csv file but what actually happens is it's not creating any file. When I debug this code I found that program is not entering in the loop. I don't understand that why it is doing so. While when I run the same program on the other website which have slightly different page structure it is running fine.
What I understand that this data is present in the #document which is a virtual element and doesn't mean anything that's why program can't read it while there is no such thing in other website. Kindly, help me out to read the data inside the #document element.
Long Story Short
Change your temp url to http://www.psx.com.pk/phps/index1.php
Explanation
There is no table in the document of http://www.psx.com.pk/index.php.
Instead it is showing it's content in two frameset.
One is dummy with url http://www.psx.com.pk/phps/blank.php.
Another one is the real page which is showing actual data and it's url is
http://www.psx.com.pk/phps/index1.php
I'm using docx4j in an XPages application to create Word documents containing content from an XPage. The Word document (in .docx format) is created based on a template (also in docx.format). This all works fine. However, when I change the template from a .docx to a .dotx format, the Word document (.docx) which is generated cannot be opened. On trying to open the document, I get an error saying that the content causes problems.
Can anyone tell me how to convert a .dotx file to a .docx file using docx4j?
The code I am currently using is:
import java.util.ArrayList;
import java.util.List;
import java.util.StringTokenizer;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.JAXBException;
import org.docx4j.XmlUtils;
import org.docx4j.openpackaging.exceptions.Docx4JException;
import org.docx4j.openpackaging.packages.WordprocessingMLPackage;
import org.docx4j.wml.ContentAccessor;
import org.slf4j.impl.*;
import java.io.FileInputStream;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import org.docx4j.wml.*;
import org.apache.commons.lang3.StringUtils;
import java.util.Enumeration;
import java.util.Map;
import java.util.Iterator;
import java.util.Vector;
import lotus.domino.Document;
import lotus.domino.*;
import org.docx4j.openpackaging.parts.WordprocessingML.DocumentSettingsPart;
import org.docx4j.jaxb.Context;
import org.docx4j.openpackaging.parts.relationships.RelationshipsPart;
import org.docx4j.openpackaging.parts.relationships.Namespaces;
public class JavaTemplateDocument {
public void mainCode(Session session, Document currDoc, String empLang, String templateType, String sArt) throws Exception {
Database dbCurr = session.getCurrentDatabase();
String viewName = "vieTemplateLookup";
View tview = dbCurr.getView(viewName);
Vector viewKey = new Vector();
viewKey.addElement(empLang);
viewKey.addElement(templateType);
Document templateDoc = tview.getDocumentByKey(viewKey);
if (tview.getDocumentByKey(viewKey) == null ) System.out.println("templateDoc is NULL");
Item itmNotesFields = templateDoc.getFirstItem("NotesFieldList");
Item itmWordFields = templateDoc.getFirstItem("WordFieldList");
Vector<String[]> notesFields = itmNotesFields.getValues();
Vector<String[]> wordFields = itmWordFields.getValues();
int z = notesFields.size();
int x = wordFields.size();
Enumeration e1 = notesFields.elements();
Enumeration e2 = wordFields.elements();
WordprocessingMLPackage template = getTemplate("C:\\Temp\\AZG Sample Template.dotx","C:\\Temp\\AZG Sample Template.docx");
for (int y = 0; y < x; y++) {
if (currDoc.hasItem(String.valueOf(notesFields.elementAt(y)))) {
Item itmNotesName = currDoc.getFirstItem(String.valueOf(notesFields.elementAt(y)));
replacePlaceholder(template, itmNotesName.getText(), String.valueOf(wordFields.elementAt(y))); }
else {
replacePlaceholder(template, "", String.valueOf(wordFields.elementAt(y)));
}
}
writeDocxToStream(template, "C:\\Temp\\AZG Sample Document.docx");
createResponseDocument(dbCurr, currDoc, templateDoc, sArt);
}
private void createResponseDocument(Database dbCurr, Document currDoc, Document templateDoc, String sArt) throws NotesException{
Document respDoc = dbCurr.createDocument(); // create the response document
String refVal = currDoc.getUniversalID();
respDoc.appendItemValue("IsDocTemplate", "1");
if (currDoc.hasItem("Name")) {
respDoc.appendItemValue("Name", currDoc.getItemValue("Name"));}
else {System.out.println("Name is not available"); }
if (currDoc.hasItem("Firstname")) {
respDoc.appendItemValue("Firstname", currDoc.getItemValue("Firstname"));}
else {System.out.println("Firstname is not available"); }
if (currDoc.hasItem("ReferenceTypeTexts")) {
respDoc.appendItemValue("ReferenceTypeTexts", currDoc.getItemValue("ReferenceTypeTexts"));}
else {System.out.println("ReferenceTypeTexts is not available"); }
if (currDoc.hasItem("ReferenceType")) {
respDoc.appendItemValue("ReferenceType", currDoc.getItemValue("ReferenceType"));}
else {System.out.println("ReferenceType is not available"); }
System.out.println("Append Form value");
respDoc.appendItemValue("Form", "frmRespTempl");
respDoc.makeResponse(currDoc);
RichTextItem body = respDoc.createRichTextItem("Body");
body.embedObject(1454, "", "C:\\Temp\\AZG Sample Document.docx", null);
respDoc.save();
}
/*
* Create a simple word document that we can use as a template.
* For this just open Word, create a new document and save it as template.docx.
* This is the word template we'll use to add content to.
* The first thing we need to do is load this document with docx4j.
*/
private WordprocessingMLPackage getTemplate(String source, String target) throws Docx4JException, FileNotFoundException, IOException {
String WORDPROCESSINGML_DOCUMENT = "application/vnd.openxmlformats- officedocument.wordprocessingml.document.main+xml";
final ContentType contentType = new ContentType(WORDPROCESSINGML_DOCUMENT);
String templatePath = source;
File sourceFile = new File(source);
File targetFile = new File(target);
copyFileUsingFileChannels(sourceFile, targetFile);
WordprocessingMLPackage template = WordprocessingMLPackage.load(new FileInputStream(targetFile));
ContentTypeManager ctm = wordMLPackage.getContentTypeManager();
ctm.addOverrideContentType(new URI("/word/document.xml"),WORDPROCESSINGML_DOCUMENT);
DocumentSettingsPart dsp = new DocumentSettingsPart();
CTSettings settings = Context.getWmlObjectFactory().createCTSettings();
dsp.setJaxbElement(settings);
wordMLPackage.getMainDocumentPart().addTargetPart(dsp);
// Create external rel
RelationshipsPart rp = RelationshipsPart.createRelationshipsPartForPart(dsp);
org.docx4j.relationships.Relationship rel = new org.docx4j.relationships.ObjectFactory().createRelationship();
rel.setType( Namespaces.ATTACHED_TEMPLATE );
rel.setTarget(templatePath);
rel.setTargetMode("External");
rp.addRelationship(rel); // addRelationship sets the rel's #Id
settings.setAttachedTemplate(
(CTRel)XmlUtils.unmarshalString("<w:attachedTemplate xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\" xmlns:r=\"http://schemas.openxmlformats.org/officeDocument/2006/relationships\" r:id=\"" + rel.getId() + "\"/>", Context.jc, CTRel.class)
);
return template;
}
private static List<Object> getAllElementFromObject(Object obj, Class<?> toSearch) {
List<Object> result = new ArrayList<Object>();
if (obj instanceof JAXBElement) obj = ((JAXBElement<?>) obj).getValue();
if (obj.getClass().equals(toSearch))
result.add(obj);
else if (obj instanceof ContentAccessor) {
List<?> children = ((ContentAccessor) obj).getContent();
for (Object child : children) {
result.addAll(getAllElementFromObject(child, toSearch));
}
}
return result;
}
/*
* This will look for all the Text elements in the document, and those that match are replaced with the value we specify.
*/
private void replacePlaceholder(WordprocessingMLPackage template, String name, String placeholder ) {
List<Object> texts = getAllElementFromObject(template.getMainDocumentPart(), Text.class);
for (Object text : texts) {
Text textElement = (Text) text;
if (textElement.getValue().equals(placeholder)) {
textElement.setValue(name);
}
}
}
/*
* write the document back to a file
*/
private void writeDocxToStream(WordprocessingMLPackage template, String target) throws IOException, Docx4JException {
File f = new File(target);
template.save(f);
}
/*
* Example code for replaceParagraph
*
String placeholder = "SJ_EX1";
String toAdd = "jos\ndirksen";
replaceParagraph(placeholder, toAdd, template, template.getMainDocumentPart());
*/
private void replaceParagraph(String placeholder, String textToAdd, WordprocessingMLPackage template, ContentAccessor addTo) {
// 1. get the paragraph
List<Object> paragraphs = getAllElementFromObject(template.getMainDocumentPart(), P.class);
P toReplace = null;
for (Object p : paragraphs) {
List<Object> texts = getAllElementFromObject(p, Text.class);
for (Object t : texts) {
Text content = (Text) t;
if (content.getValue().equals(placeholder)) {
toReplace = (P) p;
break;
}
}
}
// we now have the paragraph that contains our placeholder: toReplace
// 2. split into seperate lines
String as[] = StringUtils.splitPreserveAllTokens(textToAdd, '\n');
for (int i = 0; i < as.length; i++) {
String ptext = as[i];
// 3. copy the found paragraph to keep styling correct
P copy = (P) XmlUtils.deepCopy(toReplace);
// replace the text elements from the copy
List<?> texts = getAllElementFromObject(copy, Text.class);
if (texts.size() > 0) {
Text textToReplace = (Text) texts.get(0);
textToReplace.setValue(ptext);
}
// add the paragraph to the document
addTo.getContent().add(copy);
}
// 4. remove the original one
((ContentAccessor)toReplace.getParent()).getContent().remove(toReplace);
}
/*
* A set of hashmaps that contain the name of the placeholder to replace and the value to replace it with.
*
* Map<String,String> repl1 = new HashMap<String, String>();
repl1.put("SJ_FUNCTION", "function1");
repl1.put("SJ_DESC", "desc1");
repl1.put("SJ_PERIOD", "period1");
Map<String,String> repl2 = new HashMap<String, String>();
repl2.put("SJ_FUNCTION", "function2");
repl2.put("SJ_DESC", "desc2");
repl2.put("SJ_PERIOD", "period2");
Map<String,String> repl3 = new HashMap<String, String>();
repl3.put("SJ_FUNCTION", "function3");
repl3.put("SJ_DESC", "desc3");
repl3.put("SJ_PERIOD", "period3");
replaceTable(new String[]{"SJ_FUNCTION","SJ_DESC","SJ_PERIOD"}, Arrays.asList(repl1,repl2,repl3), template);
*/
private void replaceTable(String[] placeholders, List<Map<String, String>> textToAdd,
WordprocessingMLPackage template) throws Docx4JException, JAXBException {
List<Object> tables = getAllElementFromObject(template.getMainDocumentPart(), Tbl.class);
// 1. find the table
Tbl tempTable = getTemplateTable(tables, placeholders[0]);
List<Object> rows = getAllElementFromObject(tempTable, Tr.class);
// first row is header, second row is content
if (rows.size() == 2) {
// this is our template row
Tr templateRow = (Tr) rows.get(1);
for (Map<String, String> replacements : textToAdd) {
// 2 and 3 are done in this method
addRowToTable(tempTable, templateRow, replacements);
}
// 4. remove the template row
tempTable.getContent().remove(templateRow);
}
}
private Tbl getTemplateTable(List<Object> tables, String templateKey) throws Docx4JException, JAXBException {
for (Iterator<Object> iterator = tables.iterator(); iterator.hasNext();) {
Object tbl = iterator.next();
List<?> textElements = getAllElementFromObject(tbl, Text.class);
for (Object text : textElements) {
Text textElement = (Text) text;
if (textElement.getValue() != null && textElement.getValue().equals(templateKey))
return (Tbl) tbl;
}
}
return null;
}
private static void addRowToTable(Tbl reviewtable, Tr templateRow, Map<String, String> replacements) {
Tr workingRow = (Tr) XmlUtils.deepCopy(templateRow);
List<?> textElements = getAllElementFromObject(workingRow, Text.class);
for (Object object : textElements) {
Text text = (Text) object;
String replacementValue = (String) replacements.get(text.getValue());
if (replacementValue != null)
text.setValue(replacementValue);
}
reviewtable.getContent().add(workingRow);
}
private static void copyFileUsingFileChannels(File source, File dest)
throws IOException {
FileChannel inputChannel = null;
FileChannel outputChannel = null;
try {
inputChannel = new FileInputStream(source).getChannel();
outputChannel = new FileOutputStream(dest).getChannel();
outputChannel.transferFrom(inputChannel, 0, inputChannel.size());
} finally {
inputChannel.close();
outputChannel.close();
}
}
}
Broadly, there are a few things that comprise the difference between a template (.dotx) and a document (.docx). This means you have a few things that you need to do -- it's not as simple as just changing the file extension, whether you're saving a doc as a template, or attempting to create a document from a template.
Hopefully this outline will assist:
First do what you've already done: your new document should be a file copy of the template
Change your new WordprocessingMLPackage's document type as appropriate (see WORDPROCESSINGML_TEMPLATE in the ContentTypes class)
Create an attached template and attach it to your document: see the sample code on Github for more detail on that (TemplateAttach.java sample).
Good luck!
Let's hack it.
New office formats are just ZIPs with many XML configurations and data. Try to save identical document as template and document in MS Word. IMHO the core of your problem is in (packed) file [Content_Types].xml.
They differ in the property:
ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.template.main+xml"
ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"
I would expect #benpoole's advice should work (it should alter the content of said file). If that is not the case, simply hack the content of it inside the file (it is just ordinary ZIP archive, remember).
Disclaimer: there IS difference in few more files, that might need tweaking to make it work.
I would say that you need to change the returning filename to a dotx from docx
do a filecopy from docx to dotx and change this row
body.embedObject(1454, "", "C:\\Temp\\AZG Sample Document.dotx", null);
What I wanted to do is create a global FileInputStream object and use it a long of my application.
I have the follow class which create my FileInputStream objec and return the result of my query over the XML file:
package Engine;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.ArrayList;
import com.ximpleware.AutoPilot;
import com.ximpleware.VTDException;
import com.ximpleware.VTDGen;
import com.ximpleware.VTDNav;
public class BACENGQueryXMLbyVTD {
public String query;
public String path;
public BACENGQueryXMLbyVTD(String query, String path) {
this.query = query;
this.path = path;
}
public ArrayList query() throws IOException, VTDException {
System.out.println(path);
File f = new File(path);
//System.out.println(f);
FileInputStream fis = new FileInputStream(f);
//System.out.println(fis);
byte[] xmlContent = new byte[(int) f.length()];
fis.read(xmlContent);
VTDGen vg = new VTDGen();
vg.setDoc(xmlContent);
vg.parse(false);
VTDNav vn = vg.getNav();
AutoPilot ap = new AutoPilot(vn);
int node = 0;
ap.selectXPath(query);
int i;
ArrayList<String> interfaces = new ArrayList<String>();
while ((i = ap.evalXPath()) != -1) {
vn.push();
interfaces.add(vn.toString(i + 1));
}
ap.resetXPath();
return interfaces;
}
}
This class is called from my main class which has a loop.
for (int c = 0; c < nodeNames.size(); c++) {
String Queries2 = "/import_data/group/node[#name='" +nodeNames.get(c) + "']/interface/#network_value";
BACENGQueryXMLbyVTD getNodesInterfaces = new BACENGQueryXMLbyVTD(Queries2, TransitPath);
listOfNodeInterfaces = getNodesInterfaces.query();
}
It's working fine, however in order to reduce the consume of IO resource over my server HD. I would like to create an unique FileInputStream object and use it for each query whcih has to be executed.
Could somebody point out the way to do it?
Separate your concerns - BACENGQueryXMLbyVTD is both loading the data and executing the query.
First load the file into a byte[] outside your loop, then pass it to BACENGQueryXMLbyVTD. You might also want to pass the query as an argument to the query method.
You'll end up with a BACENGQueryXMLbyVTD that looks like this (with the disclaimer that I'm not familiar with VTD-XML, so this might the creation of objects from that API might not work exactly like this):
public class BACENGQueryXMLbyVTD
{
private byte[] doc;
public BACENGQueryXMLbyVTD(byte[] doc)
{
this.doc = doc;
}
public List<String> query(String query) throws IOException, VTDException
{
VTDGen generator = new VTDGen();
generator.setDoc(doc);
generator.parse(false);
VTDNav navigator = generator.getNav();
AutoPilot autoPilot = new AutoPilot(navigator);
autoPilot.selectXPath(query);
List<String> nodeInterfaces = new ArrayList<String>();
int i;
while ((i = autoPilot.evalXPath()) != -1)
{
navigator.push();
nodeInterfaces.add(navigator.toString(i + 1));
}
return nodeInterfaces;
}
}
That you can then call like:
byte[] xmlContent = ... //load the xml from anywhere you like, not just a file as previously.
BACENGQueryXMLbyVTD getNodesInterfaces = new BACENGQueryXMLbyVTD(xmlContent);
for (String nodeName : nodeNames)
{
String query = "/import_data/group/node[#name='" + nodeName + "']/interface/#network_value";
nodeInterfaces = getNodesInterfaces.query(query);
...
}
You might also want to switch to the standard Java XPATH APIs - they're a lot clearer and better documented than VTD-XML.
Ok, I'm really confused by some code I wrote. It's a DataSetter (didn't know a better name for it...), and has methods to change the data in my data file (data.txt). This data has the following format: #key=value (eg. #version=1.0). Now, I tried to run this line of code:
new DataSetter().setValue("version", "1.1");
It just clears the file. That's pretty much all it does. Now, I think it clears the file because it makes a new File, which is completely empty but has the same name. Here's my code:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Scanner;
/**
* This class contains methods to set specific data in the data.txt file. <br>
* The data is rewritten every time a new value is set.
*
* #author Casper van Battum
*
*/
public class DataSetter {
private static final File DATA_FILE = new File("resources/data.txt");
private static final String lineFormat = "#%s=%s";
private FileOutputStream out;
private DataReader reader = new DataReader();
private HashMap<String, String> dataMap = reader.getDataMap();
private Scanner scanner;
public DataSetter() {
try {
out = new FileOutputStream(DATA_FILE, false);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
public void setValue(String key, String newValue) {
openDataFile();
String oldLine = String.format(lineFormat, key, dataMap.get(key));
dataMap.put(key, newValue);
String newLine = String.format(lineFormat, key, newValue);
try {
replace(oldLine, newLine);
} catch (IOException e) {
e.printStackTrace();
}
closeDataFile();
}
private void replace(String oldLine, String newLine) throws IOException {
ArrayList<String> tmpData = new ArrayList<String>();
while (scanner.hasNextLine()) {
String currentLine = scanner.nextLine();
tmpData.add((currentLine == oldLine) ? newLine : currentLine);
}
out.write(new String().getBytes());
String sep = System.getProperty("line.separator");
StringBuffer sb = new StringBuffer();
for (String string : tmpData) {
sb.append(string + sep);
}
FileWriter writer = new FileWriter(DATA_FILE);
String outString = sb.toString();
writer.write(outString);
writer.close();
}
private void openDataFile() {
try {
scanner = new Scanner(DATA_FILE);
} catch (FileNotFoundException ex) {
ex.printStackTrace();
}
}
private void closeDataFile() {
scanner.close();
}
}
So after running the setValue() method, I just have an empty file...
Im really out of idea's on how to solve this...
You are truncating your data file with the
new FileOutputStream(DATA_FILE, false)
so no nothing is written when you go to output your the elements in the tmpData ArrayList read from Scanner.
ArrayList<String> tmpData = new ArrayList<String>();
while (scanner.hasNextLine()) {
String currentLine = scanner.nextLine(); // never gets called
...
}
The typical strategy for updating a text file is to create a temporary file with old file's contents (File#renameTo), write the data to file, then delete the temporary file after closing any open streams to the file being read.
First the revised code which throws javax.swing.text.ChangedCharSetException:
import java.io.*;
import java.net.*;
public class Main
{
public static void main(String[] args) throws IOException, Exception
{
String query = "#pragma";
Socket s = new Socket("google.com",80);
PrintStream p = new PrintStream(s.getOutputStream());
p.print("GET /search?q="+query+" HTTP/1.0\r\n");
p.print("User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)\r\n");
p.print("Connection: close\r\n\r\n");
InputStreamReader in = new InputStreamReader(s.getInputStream());
BufferedReader buffer = new BufferedReader(in);
// String line;
//
// while ((line = buffer.readLine()) != null)
// { System.out.println(line); }
HTMLUtils.ParseLinks (buffer);
in.close();
}
}
import java.io.BufferedReader;
import java.io.IOException;
//import java.io.FileReader;
import java.io.Reader;
import java.util.List;
import java.util.ArrayList;
import javax.swing.text.html.parser.ParserDelegator;
import javax.swing.text.html.HTMLEditorKit.ParserCallback;
import javax.swing.text.html.HTML.Tag;
import javax.swing.text.html.HTML.Attribute;
import javax.swing.text.MutableAttributeSet;
public class HTMLUtils
{
private HTMLUtils() {}
public static List<String> extractLinks(Reader reader) throws IOException
{
final ArrayList<String> list = new ArrayList<String>();
ParserDelegator parserDelegator = new ParserDelegator();
ParserCallback parserCallback = new ParserCallback()
{
public void handleText(final char[] data, final int pos) { }
public void handleStartTag(Tag tag, MutableAttributeSet attribute, int pos)
{
if (tag == Tag.A) {
String address = (String) attribute.getAttribute(Attribute.HREF);
list.add(address);
}
}
public void handleEndTag(Tag t, final int pos) { }
public void handleSimpleTag(Tag t, MutableAttributeSet a, final int pos) { }
public void handleComment(final char[] data, final int pos) { }
public void handleError(final java.lang.String errMsg, final int pos) { }
};
parserDelegator.parse(reader, parserCallback, false);
return list;
}
public static void ParseLinks(BufferedReader buffer) throws Exception{
//FileReader reader = new FileReader("buffer");
List<String> links = HTMLUtils.extractLinks(buffer);
for (String link : links) {
System.out.println(link);
}
}
}
Notice that the user agent is for IE in this example.
Now I Have 3 problems:
How/can I pass the HTMLUtils.ParseLinks method a "raw buffer" instead of an HTML file she's expecting (I can write the buffer to a file but I guess that is unnecessary)
I don't know how to enter inverted commas (" ") inside the query statment in order to get the whole string i.e.: query=" "New York Yankees" "
Is it so complicated to get the User-Agent string from the host machine ???
link text
I have to say that it is imported class that I use and I don't really understand whats going on there. I'll try to understand when it will work [-8
THNX
Have a read of http://code.google.com/apis/ajaxsearch/, it's going to be a lot easier to get the data out of a JSON string than digging through acres of HTML. There's an open source Java class for digesting JSON: http://www.json.org/java/. Transferring the JSON will require a lot less bandwidth too!
If you want to do it in Java, you should consider using XPath to extract all links from the response. Therefore you first have to convert the response to XML. Then you can apply an XPath query like
//a/#href
to extract all href attributes for links. You can modify the query to only include links from the Google results and not from advertisements etc.
Here is another Tutorial to get you started.
Happy coding.
BTW: To avoid mistakes when you create your HTTP request and (even more important) to avoid unnecessary work, you could use a library like Apache Commons HTTPClient. This would reduce your work to:
HttpClient client = new HttpClient();
HttpMethod method = new GetMethod("http://www.google.com/search?q=" + query);
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
String response = new String(method.getResponseBody());