We Keep Coding

Getting Data from multiple a tags in HTML

I am scraping a medical website where I need to extract header wise information regarding a drug e.g Precautions, Contraindications,Dosage, Uses etc. The HTML data looks like below. If I just extract info using the tag p.drug-content I get content under all the headers as one big paragraph. How do I get header wise content where the paragraph for dosage should come under dosage, Precautions under precautions, so on and so forth?
<a name="Warning"></a>
<div class="report-content drug-widget">
<div class="drug-header"><h2 style="color:#000000!important;">What are the warnings and precautions for Abacavir? </h2></div>
<p class="drug-content">
• Caution is advised when used in patients with history of depression or at risk for heart disease<br>• Avoid use with alcohol.<br>• Take along with other anti-HIV drugs and not alone, to prevent resistance.<br>• Continue other precautions to prevent spread of HIV infection.</p></div>
<a name="Prescription"></a>
<div class="report-content drug-widget">
<div class="drug-header"><h2 style="color:#000000!important;">Why is Abacavir Prescribed? (Indications) </h2></div>
<p class="drug-content">Abacavir is an antiviral drug that is effective against the HIV-1 virus. It acts on an enzyme of the virus called reverse transcriptase, which plays an important role in its multiplication. Though abacavir reduces viral load and may slow the progression of the disease, it does not cure the HIV infection. </p></div>
<a name="Dosage"></a>
<div class="report-content drug-widget">
<div class="drug-header"><h2 style="color:#000000!important;">What is the dosage of Abacavir?</h2></div>
<p class="drug-content"> Treatment of HIV-1/AIDS along with other medications. Dose in adults is 600 mg daily, as a single dose or divided into two doses.
</p></div>
Here is my code:
private static void ScrapingDrugInfo() throws IOException{
Connection.Response response = null;
Document doc = null;
List<SideEffectsObject> sideEffectsList = new ArrayList<>();
int i=0;
String[] keywords = {"a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z"};
for (String keyword : keywords){
final String url = "https://www.medindia.net/doctors/drug_information/home.asp?alpha=" + keyword;
response = Jsoup.connect(url)
.userAgent("Mozilla/5.0")
.execute();
doc = response.parse();
Element tds = doc.select("div.related-links.top-gray.col-list.clear-fix").first();
Elements links = tds.select("li[class=list-item]");
for (Element link : links){
final String newURL = "https://www.medindia.net/doctors/drug_information/".concat(link.select("a").attr("href")) ;
response = Jsoup.connect(newURL)
.userAgent("Mozilla/5.0")
.execute();
doc = response.parse();
Elements classification = doc.select("div.clear.b");
System.out.println("Classification::"+classification.text());
Elements drugBrands = doc.select("div.drug-content");
Elements drugBrandsIndian = drugBrands.select("div.links");
System.out.println("Drug Brand Links Indian::"+ drugBrandsIndian.select("a[href]"));
System.out.println("Drug Brand Names Indian::"+ drugBrandsIndian.text());
System.out.println("Drug Brand Names International::"+doc.select("div.drug-content.h3").text());
Elements prescritpionText = doc.select("a[name=Prescription]");
Elements prescriptionData = prescritpionText.select("p.drug-content");
System.out.println("Prescription Data::"+ prescriptionData.text());
Elements contraindications = doc.select("a[name=Contraindications]");
Elements contraindicationsText = contraindications.select("p[class=drug-content]");
System.out.println("Contrainidications Text::" + contraindicationsText.text());
Elements dosage = doc.select("a[name=Dosage]");
Elements dosageText = dosage.select("p[class=drug-content]");
System.out.println("Dosage Text::" + dosageText.text());
}
}

If I understand the question correctly, it sounds like you want to pair up the value of the a tags name attribute with the p content of the following div. You should be able to do that with the following code:
Elements aTags = doc.select("a[name]");
for(Element header : aTags){
System.out.println(header.attr("name"));
// Get the sibling div of a and get it's p content
Element pTag = header.nextElementSibling().select("p.drug-content").first();
System.out.println(pTag.text());
}

Extract text from html file using java

I am working on a text crawler and I need to extract a certain text from several sites. I have used jsoup html parser:
Document doc = Jsoup.connect("http://www.aljazeera.net/programs/behindthenews/2014/11/9/%D8%A3%D8%B3%D8%A8%D8%A7%D8%A8-%D9%88%D8%AF%D9%84%D8%A7%D9%84%D8%A7%D8%AA-%D8%A7%D8%B3%D8%AA%D9%85%D8%B1%D8%A7%D8%B1-%D8%A7%D9%84%D8%B5%D8%AF%D8%A7%D9%85-%D8%A8%D8%AC%D8%A7%D9%85%D8%B9%D8%A7%D8%AA-%D9%85%D8%B5%D8%B1").get();
Elements ps = doc.select("p");
String s = Jsoup.parse(ps.text()).text();
PrintWriter out = new PrintWriter("newFile.txt", "UTF-8");
out.write(s);
out.close();
This algorithm gets me an unwanted text, the wanted text is under this tag <p dir="rtl">
And when I change my selection to Elements ps = doc.select("p dir=rtl");, it returns a compilation error.
Is there any way to make it just select this wanted tag?

the wanted text is under this tag <p dir="rtl">
You need to use the following CSS query:
p /* Select any p element... */
[dir=rtl] /* ... with a 'dir' attribute set to "rtl" */
#NavidShakibapour http://www.aljazeera.net/programs/behindthenews/2014/11/9/أسباب-ودلالات-استمرار-الصدام-بجامعات-مصر
The above url need to be encoded. We'll use the URI#create helper method. On the returned URI instance, we'll invoke the toASCIIString method.
URI uri = URI //
.create("http://www.aljazeera.net/programs/behindthenews/2014/11/9/أسباب-ودلالات-استمرار-الصدام-بجامعات-مصر") //
.toASCIIString();
Here is a working sample code:
SAMPLE CODE
Document doc = Jsoup.connect("http://www.aljazeera.net/").get();
Elements sublinks = doc.select("a[href*=/programs/behindthenews]");
System.out.println("Sublinks found: " + sublinks.size() + "\n*****");
for (Element sublink : sublinks) {
String url = URI.create(sublink.absUrl("href")).toASCIIString();
Document subLinkPage = Jsoup.connect(url).get();
System.out.println(subLinkPage.select("p[dir=rtl]"));
System.out.println("-------------------------");
}
OUTPUT (content stripped)
Sublinks found: 5
*****
<p dir="rtl">وحول هذا الموضوع وصف مدير مكتب الجزيرة في موسكو زاور شوج قرار الرئيس بوتين بالخطوة المفاجئة، باعتبار أن الروس وضعوا سقفا زمنيا مفتوحا لتدخلهم العسكري، يتراوح بين 12 و18 شهرا.</p>
<p dir="rtl">
(...)
-------------------------
<p dir="rtl">وحول هذا الموضوع وصف مدير مكتب الجزيرة في موسكو زاور شوج قرار الرئيس بوتين بالخطوة المفاجئة، باعتبار أن الروس وضعوا سقفا زمنيا مفتوحا لتدخلهم العسكري، يتراوح بين 12 و18 شهرا.</p>
<p dir="rtl">
(...)
-------------------------
(...)

The below code will capture all the <p dir="rtl"> tags:
String uri = URI.create("example.com").toASCIIString();
Document doc = Jsoup.connect(uri).get();
Elements pElements = doc.select("p[dir=rtl]");
StringBuilder sb = new StringBuilder();
for (Element element : pElements) {
sb.append(element.text());
}
PrintWriter out = new PrintWriter("newFile.txt", "UTF-8");
out.write(sb.toString());
out.close();
The only tricky thing is that, you need to use encoded URLs (and not the one with Arabic letters).
Edit 1:
URL encoding can be done in the code.

Put hyperlink into image in excel (Apache POI)

I can set hyperlink for cell in Apache POI, but I don't know how to put the hyperlink into an image (I'm using XSSF)
Here is the function of putting cell hyperlink :
/**
* Helper function for putting hyperlink into specified cell
* #param label
* #param value
* #param col
* #param row
* #param sheet
*/
private static void putImageHyperlink(Cell cell, CellStyle hyperlinkStyle, String value, Workbook wb) {
try {
CreationHelper createHelper = wb.getCreationHelper();
Hyperlink link = createHelper.createHyperlink(Hyperlink.LINK_URL);
link.setAddress(value);
cell.setHyperlink(link);
cell.setCellStyle(hyperlinkStyle);
// Put hyperlink value
cell.setCellValue(value);
} catch (Exception e) {
log.severe("Can't create hyperlink : " + Utils.exceptionToString(e));
}
}
And here is the function for putting image into specified cell :
/**
* Put image into sheet at position [row,col]
* #param sheet
* #param col
* #param row
* #param imgData
* #throws Exception
*/
private static void putImage(Workbook wb, Sheet sheet, int col, int row, byte[] imgData) throws Exception {
try {
Drawing drawing = sheet.createDrawingPatriarch();
int pictureIdx = wb.addPicture(imgData, Workbook.PICTURE_TYPE_PICT);
CreationHelper helper = wb.getCreationHelper();
ClientAnchor anchor4 = helper.createClientAnchor();
//set top-left corner of the picture,
//subsequent call of Picture#resize() will operate relative to it
anchor4.setCol1(col);
anchor4.setRow1(row);
anchor4.setCol2(col+1);
anchor4.setRow2(row+1);
drawing.createPicture(anchor4, pictureIdx);
} catch (Exception ex) {
log.severe("Exception : " + Utils.exceptionToString(ex));
}
}
=> How to put hyperlink into picture created by command drawing.createPicture(anchor4, pictureIdx); ?
Thanks in advance!

When looking at how Excel stores this it seems to be stored differently for Images in the xl\drawings\_rels\drawing1.xml.rels and xl\drawings\drawing1.xml part of the XLSX file:
<Relationship Id="rId1" Target="http://poi.apache.org" TargetMode="External" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/hyperlink"/>
<xdr:cNvPr descr="Picture" id="2" name="Picture 1">
<a:hlinkClick xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" r:id="rId1"/>
</xdr:cNvPr>
POI does not support adding such hyperlinks via it's API yet, but you can use the underlying lowlevel-API as follows to first create the relationship for the hyperlink and then set the relation to the hyperlink in the Picture-object:
PackageRelationship rel = ((XSSFDrawing)patriarch).getPackagePart().addRelationship(
new URI("http://poi.apache.org"),
TargetMode.EXTERNAL, "http://schemas.openxmlformats.org/officeDocument/2006/relationships/hyperlink");
((XSSFDrawing)patriarch).addRelation(rel.getId(),new POIXMLDocumentPart());
CTPictureNonVisual nvPicPr = ((XSSFPicture)picture).getCTPicture().getNvPicPr();
CTHyperlink hLinkClick = nvPicPr.getCNvPr().addNewHlinkClick();
hLinkClick.setId(rel.getId());

Reading from a word document using java

I'm trying to read data from a docx file using java. The data is is tables. Is there a way to iterate through the table cells and extract the cell data?

You have two choices:
Write a library that can open, read, and manipulate Word documents, for some definitions of "open", "read", and "manipulate", and some definition of "Word document".
Find a third-party library that can do some or all of what is defined in (1).
Search "java docx library" in your favourite web search and see what comes up.

This should help you.
http://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/poi/xwpf/usermodel/SimpleTable.java
The above was achieved using Apache POI
Here is the code from the link:
package org.apache.poi.xwpf.usermodel;
import java.io.FileOutputStream;
import java.math.BigInteger;
import java.util.List;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.CTHeight;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.CTShd;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.CTString;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.CTTblPr;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.CTTcPr;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.CTTrPr;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.CTVerticalJc;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.STShd;
import org.openxmlformats.schemas.wordprocessingml.x2006.main.STVerticalJc;
/**
* This program creates a simple WordprocessingML table using POI XWPF API, and
* a more complex, styled table using both XWPF and ooxml-schema. It's possible
* that not all referenced wordprocessingml classes are defined in
* poi-ooxml-schemas-3.8-beta4. If this is the case, you'll need to use the full
* ooxml-schemas.jar library.
*
* #author gisella bronzetti (original)
* #author Gregg Morris (styled table)
*/
public class SimpleTable {
public static void main(String[] args) throws Exception {
try {
createSimpleTable();
}
catch(Exception e) {
System.out.println("Error trying to create simple table.");
throw(e);
}
try {
createStyledTable();
}
catch(Exception e) {
System.out.println("Error trying to create styled table.");
throw(e);
}
}
public static void createSimpleTable() throws Exception {
XWPFDocument doc = new XWPFDocument();
XWPFTable table = doc.createTable(3, 3);
table.getRow(1).getCell(1).setText("EXAMPLE OF TABLE");
// table cells have a list of paragraphs; there is an initial
// paragraph created when the cell is created. If you create a
// paragraph in the document to put in the cell, it will also
// appear in the document following the table, which is probably
// not the desired result.
XWPFParagraph p1 = table.getRow(0).getCell(0).getParagraphs().get(0);
XWPFRun r1 = p1.createRun();
r1.setBold(true);
r1.setText("The quick brown fox");
r1.setItalic(true);
r1.setFontFamily("Courier");
r1.setUnderline(UnderlinePatterns.DOT_DOT_DASH);
r1.setTextPosition(100);
table.getRow(2).getCell(2).setText("only text");
FileOutputStream out = new FileOutputStream("simpleTable.docx");
doc.write(out);
out.close();
}
/**
* Create a table with some row and column styling. I "manually" add the
* style name to the table, but don't check to see if the style actually
* exists in the document. Since I'm creating it from scratch, it obviously
* won't exist. When opened in MS Word, the table style becomes "Normal".
* I manually set alternating row colors. This could be done using Themes,
* but that's left as an exercise for the reader. The cells in the last
* column of the table have 10pt. "Courier" font.
* I make no claims that this is the "right" way to do it, but it worked
* for me. Given the scarcity of XWPF examples, I thought this may prove
* instructive and give you ideas for your own solutions.
* #throws Exception
*/
public static void createStyledTable() throws Exception {
// Create a new document from scratch
XWPFDocument doc = new XWPFDocument();
// -- OR --
// open an existing empty document with styles already defined
//XWPFDocument doc = new XWPFDocument(new FileInputStream("base_document.docx"));
// Create a new table with 6 rows and 3 columns
int nRows = 6;
int nCols = 3;
XWPFTable table = doc.createTable(nRows, nCols);
// Set the table style. If the style is not defined, the table style
// will become "Normal".
CTTblPr tblPr = table.getCTTbl().getTblPr();
CTString styleStr = tblPr.addNewTblStyle();
styleStr.setVal("StyledTable");
// Get a list of the rows in the table
List<XWPFTableRow> rows = table.getRows();
int rowCt = 0;
int colCt = 0;
for (XWPFTableRow row : rows) {
// get table row properties (trPr)
CTTrPr trPr = row.getCtRow().addNewTrPr();
// set row height; units = twentieth of a point, 360 = 0.25"
CTHeight ht = trPr.addNewTrHeight();
ht.setVal(BigInteger.valueOf(360));
// get the cells in this row
List<XWPFTableCell> cells = row.getTableCells();
// add content to each cell
for (XWPFTableCell cell : cells) {
// get a table cell properties element (tcPr)
CTTcPr tcpr = cell.getCTTc().addNewTcPr();
// set vertical alignment to "center"
CTVerticalJc va = tcpr.addNewVAlign();
va.setVal(STVerticalJc.CENTER);
// create cell color element
CTShd ctshd = tcpr.addNewShd();
ctshd.setColor("auto");
ctshd.setVal(STShd.CLEAR);
if (rowCt == 0) {
// header row
ctshd.setFill("A7BFDE");
}
else if (rowCt % 2 == 0) {
// even row
ctshd.setFill("D3DFEE");
}
else {
// odd row
ctshd.setFill("EDF2F8");
}
// get 1st paragraph in cell's paragraph list
XWPFParagraph para = cell.getParagraphs().get(0);
// create a run to contain the content
XWPFRun rh = para.createRun();
// style cell as desired
if (colCt == nCols - 1) {
// last column is 10pt Courier
rh.setFontSize(10);
rh.setFontFamily("Courier");
}
if (rowCt == 0) {
// header row
rh.setText("header row, col " + colCt);
rh.setBold(true);
para.setAlignment(ParagraphAlignment.CENTER);
}
else if (rowCt % 2 == 0) {
// even row
rh.setText("row " + rowCt + ", col " + colCt);
para.setAlignment(ParagraphAlignment.LEFT);
}
else {
// odd row
rh.setText("row " + rowCt + ", col " + colCt);
para.setAlignment(ParagraphAlignment.LEFT);
}
colCt++;
} // for cell
colCt = 0;
rowCt++;
} // for row
// write the file
FileOutputStream out = new FileOutputStream("styledTable.docx");
doc.write(out);
out.close();
}
}

Table creation on separate method using iText in Java

I'm trying to create an PDF document using iText. I followed to THIS nice tutorial and tried to create single page pdf document which has a table. In the tutorial the author keeps table creation of table on separate method such as addMetaData, addTitlePage and addContent. I also would keep them separately, but I'm new to iText and currently I'm stuck. The current code is:
public static void main(String args[]) {
try {
Document document = new Document(PageSize.A4);
PdfWriter.getInstance(document, new FileOutputStream(FILE));
document.open();
addMetaData(document);
addTitlePage(document);
addContent(document);
document.close();
} catch (Exception e) {
e.printStackTrace();
}
}
private static void addTitlePage(Document document) throws DocumentException {
Paragraph preface = new Paragraph();
// Add one empty line
addEmptyLine(preface, 1);
// Header of the document
preface.add(new Paragraph("Title here", capFont));
addEmptyLine(preface, 1);
// Report generated by: _name, _date
preface.add(new Paragraph("Report generated by: " + System.getProperty("user.name") + ", " + new Date(), //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
normFont));
addEmptyLine(preface, 2);
preface.add(new Paragraph("This document describes some kind of price list which is unknown to me.", normFont));
document.add(preface);
}
private static void addContent(Document document) throws DocumentException {
Paragraph content = new Paragraph();
// Add one empty line
addEmptyLine(content, 1);
// Content of the document
content.add(new Paragraph(createTable(subPart), normFont)); // not working line
addEmptyLine(content, 5);
content.add(new Paragraph("This document is a preliminary version and not subject to the license agreement.", redFont));
document.add(content);
}
private static void createTable(Section subPart) throws BadElementException {
PdfPTable table = new PdfPTable(3);
table.setHorizontalAlignment(Element.ALIGN_CENTER);
// Data
table.addCell("1");
table.addCell("2");
table.addCell("3");
subPart.add(table);
}
Any help would be appreciated.

So, after one day literature reading and api homepage visiting I came to my solution:
Instead of: content.add(new Paragraph(createTable(subPart), normFont));
I have now: createTable(content);
and of course I changed the type of variable in createTable method to the paragraph in order to get it working.

package src.AutosysPolicyWriter.Utility;
import java.util.StringTokenizer;
import com.lowagie.text.Cell;
import com.lowagie.text.Document;
import com.lowagie.text.Element;
import com.lowagie.text.Font;
import com.lowagie.text.Phrase;
import com.lowagie.text.Rectangle;
import com.lowagie.text.Table;
/**
* Modification/itext 1.4 - <Modified : September 26, 2013>
* #author Oliver Lundag
* #date 2013-09-06
* Can handle table creation in the PDF
*/
public class PdfTableUtility {
/***
*
* #param titleFont - font of the title of the table
* #param fontHeader - font of the headers
* #param fontData - font of the data
* #param thisReport - Document
* #param tableTitle - Name of the table
* #param headerStrings - headers
* #param data - data
*
* How to use this function;
*
* Example:
*
* // Initialize PdfTableUtility object
* 1. PdfTableUtility tableUtility = new PdfTableUtility();
*
* // Declare the value for headers.
* // Note: Number of columns will depends on how many headers has been declared
* 2. String[] headers = {"Customer Name","Age","Plan","Amount"};
*
* // Declare the data that will be put inside
* // Note that the arrangement of the strings are the actual display in the pdf
* // ; - separator
* 3. String data = "1.Oliver Lundag;26;Plan A;250,000;"+
* "2.Oliver Lundag;26;Plan A;250,000;"+
* "3.Oliver Lundag;26;Plan A;250,000";
*
* //call the function with specified arguments
* //arguments will depend on developers perspective
* 4. tableUtility.displaytable(FontBold11, FontBold9, FontNormal9, thisReport, "Information", 4, headers, data);
*/
public void displaytable(Font titleFont, Font fontHeader, Font fontData, Document thisReport,String tableTitle, String[] headerStrings, String data) {
try{
//START-(Modification/itext 1.4) SR-CS-13035 - OLUND <Modified : September 26, 2013> - title
//1.create table
Table title = new Table(1);
title.setLastHeaderRow(1);
title.setOffset(12f);
title.setSpaceInsideCell(1f);
title.setBorder(Rectangle.NO_BORDER);
//2.create table
Cell celltitle = new Cell(new Phrase(tableTitle, titleFont));
celltitle.setLeading(12);
celltitle.setBorder(Rectangle.NO_BORDER);
celltitle.setHorizontalAlignment(Element.ALIGN_CENTER);
title.addCell(celltitle);
//3.add the title in document
thisReport.add(title);
//END -(Modification/itext 1.4) SR-CS-13035 - OLUND <Modified : September 26, 2013> - title
//START-(Modification/itext 1.4) SR-CS-13035 - OLUND <Modified : September 26, 2013> - data
//1. get the max number of columns
int numColumns = headerStrings.length;
//2. create a table
Table table = new Table(numColumns);
table.setOffset(12f);
table.setLastHeaderRow(1);
table.setSpaceInsideCell(1f);
table.setTableFitsPage(true);
table.setAutoFillEmptyCells(true);
//3.get headers and add it into cells
for (String header : headerStrings) {
Cell headerCell = new Cell(new Phrase(header, fontHeader));
headerCell.setLeading(12);
headerCell.setHorizontalAlignment(Element.ALIGN_CENTER);
table.addCell(headerCell);
}
//4.get data and add it into cells
String strToken;
if (data.length() > 0) {
StringTokenizer stStr = new StringTokenizer(data,";",false);
while(stStr.hasMoreTokens()){
strToken = stStr.nextToken().toString();
Cell cell = new Cell(new Phrase(strToken,fontData));
cell.setHorizontalAlignment(Element.ALIGN_CENTER);
cell.setLeading(12);
table.addCell(cell);
}
}
//5. add the table in the document
thisReport.add(table);
//END - (Modification/itext 1.4) SR-CS-13035 - OLUND <Modified : September 26, 2013> - data
}catch (Exception e) {
e.printStackTrace();
}
}
}