I want to download an HTML page, extract some used full text out of this HTML and convert the HTML to PDF then store the useful text and PDF in a noSQL solution.
What is the most efficient way to pass the HTML to the modules which extract useful text and the module which creates the PDF. I don't want to download the same HTML twice.
One way to store the HTML is to download the HTML to a local disk under a unique named folder and pass the path to other modules so that they can process the HTML.
This approach doesn't looks that good to me, as there is implementation overhead.
I would love to see the entire HTML as a single variable so I can give it to other modules so they can traverse the HTML without loading it. One idea that crossed my mind is to download and zip the HTML and related code/pics then store the binary in a byte[].
I haven't used these before but a quick Type search on eclipse with the text html gave me this:
Class HTMLDocument
From the docs :
A document that models HTML. The purpose of this model is to support both browsing and editing
Related
How to extract PDF file content in java completely as Text and render as HTML?
Not like extracting just text separately or just images separately, requirement is to display contents of PDF file (as like original file-means including images and tables right at place where it was in original file) as HTML content.
Some how same like the sample in the answer here Convert Word to HTML with Apache POI which extracts contents of MS Doc file into HTML using Apache POI.
Extracting data from a PDF file is fairly simple. There are multiple libraries out there that do it correctly. Extracting data, and preserving its layout, on the other hand (the workflow the OP describes) is a very difficult process. The reason behind it is simple - most* PDF files, don't really have any elements that define structure. When a PDF file, for example, displays a table, it's very easy for humans to see it, and understand this is indeed a table with some data in it. However, in the PDF file itself, this is a collection of vector lines, and some text runs in between. The PDF itself, or the PDF viewer, are not aware that this is a table. Therefore when this data is converted to HTML, we don't know that we need to draw a table, but instead see this as vector art. This is just one example of why this is difficult. There are many others that can be used to illustrate this point.
On the other hand, such a thing exists as "Tagged PDF" (section 10.7). It's a PDF where structure elements are actually defined, and extraction is fairly easy. However tagged PDF files are not as common as we would like, and in most cases you won't be guaranteed to work with one.
There are some tools on the market that use sophisticated logic to infer the structure of an untagged document. Some of them do a better job than others at this. I've worked with Adobe Acrobat, which does a decent job at creating an HTML file. There is also an offering from Datalogics (I work for Datalogics) called PDF Alchemist which converts PDF to HTML. Both of them are commercial solutions.
If you are looking for a free solution, PDFBox does a good job at extracting content from a PDF document. However, it doesn't have the ability to create an HTML file, and this is something that will have to be implemented outside of the library. I'm not aware of any free PDF to HTML solutions that do a good enough job, and I would be willing to recommend.
I am working on a code, that requires uploading of any kind of document from client's machine to the server, and extracting images out of it. For almost all docs Tika is helpful, but in case of an html page, the images are referenced to the local machine's path. So how do I upload the html page along with the images it contains?
I'm using Java Servlets and JSP as platform.
This is impossible to solve server-side, you have to implement a client-side (Javascript? Java applet? Flash (yuck!)?) solution. The HTML document is just a text, it does not contain the images - it just references them. So you have to parse the document, get the images, upload them independently, and then - server-side - process the document and adjust the image references (the values of src attributes).
Pretty complex, isn't it?
Does anybody know some open source tools to parse the html pages, filter the Ads,JS and etc to get title, text. Front end of my application is based on LAMP. So I needs to parse the html pages and storage them into Mysql. And populate front pages with these data.
I know some tools: Heritrix, Nutch. But it seems that they are crawlers.
Thanks.
Joseph
It depends on what you mean by "text" from the webpage. I did a similar thing by grabbing a webpage using the apache HttpClient libraries and then dom4j to look for a particular tag to extract text from. But you do in effect need the same type of crawler that search engines like google use. You are emulating the basic steps that they do when they crawl a website. Extracting the information. It would be helpful if you went into a little more detail on what kind of information you want to retrieve from the pages.
I generate a html file using log4j WriterAppender file. I also takesnapshots of my screen using webdriver. Now I wish to append them together.
Any idea how to do that?
Thanks!
Apologies for not being clear and daft. My situation is that I have got a html file which is generated dynamically by my logger class and then there are some .png file which are also being created dynamically. Now I want them to appear together in one file. Am I clear now? Please ask for more information if needed
It's possible to embed graphics data in a couple of ways. Most modern browsers accept the data: url notation. An image can be embedded straight into a url.
I took an example from this site. Cut and paste the whole line into the url bar:
data:image/gif;base64,R0lGODlhEAAOALMAAOazToeHh0tLS/7LZv/0jvb29t/f3//Ub//ge8WSLf/rhf/3kdbW1mxsbP//mf///yH5BAAAAAAALAAAAAAQAA4AAARe8L1Ekyky67QZ1hLnjM5UUde0ECwLJoExKcppV0aCcGCmTIHEIUEqjgaORCMxIC6e0CcguWw6aFjsVMkkIr7g77ZKPJjPZqIyd7sJAgVGoEGv2xsBxqNgYPj/gAwXEQA7
You should see a folder graphic. Some older browsers don't accept this, and some such as IE8 restrict content in various ways, to static content for security reasons.
The second way of doing the same is for the server to serve multi-part MIME. Basically a server would shove out a multi-part mime document consisting of the HTML body and then any inline images base64 encoded as separate parts. This is more suitable for email HTML although it might work through a web browser.
It's not quite clear what you're asking here, but let's assume that you want to manually add an image to the log output HTML file.
If you want to include an image in your HTML file, just save the snapshot PNG file in a place relative to where the HTML is generated, then include it using standard HTML syntax:
<img src="images/snapshot.png" alt="snapshot description">
Update: the requirement is to add dynamically generated PNG files to a dynamically created HTML log file.
If one process is creating both the PNG and the log output, you should be fine - just keep note of the appropriate PNG filename and include it in the logger output in an IMG tag (as described above).
If they are generated by separate processes, this may be more difficult; you would need to either stick to a known naming convention, have the process generating the log query the filesystem to determine the appropriate PNG file to include, or build some sort of message-passing between the two processes.
Please stop posting the same comment to each and any of the different answers given to you, when all of those answers basically tell you that the notion of concatenating two different file formats into a single file is not meaningful.
Let me repeat that again for clarity: Copying a PNG file into a HTML document makes no sense.
You either save the PNG in a directory where it's accessible in the HTML document and add an img tag so it can be referenced (see the answer by stark), which would be the recommended way in terms of portability and usage of the files as they were intended to be used.
If you really, really want to end up with a single file for whatever reasons, there are bascially two options: You follow the advice of locka and encode the PNG image with Base64 and insert an img tag with a data URI at a meaningful position. This probably involves parsing the HTML "a little" to come up with a good place to insert it.
The other option is to not create HTML, but MHTML files. MHTML is a file format that allows saving HTML source code and resources like images into a single file. MHTML is supported by the most popular browsers nowadays, you may find info on the file format here: http://people.dsv.su.se/~jpalme/ietf/mhtml.html
In the code where you are generating the html you should just include the img using the img html tag
If you want the picture to appear in the html, add the tag
<img src=./img.png /> to your html.
If you want the 2 files in one, you'll need to zip them into an archive or something?
It makes no sense to append a HTML file to a PNG file, or vice-versa. Neither file format allows this, so if you do this you will end up with a "corrupt" document that a typical web browser or image viewer won't understand.
"I want them to appear together in one file".
That's still pretty vague, I'm afraid.
Assuming that you want the image to appear embedded in the HTML document when you open the HTML document in a browser, the simple solution is create separate HTML and PNG files, and have the HTML file link to the PNG file using an <img> element.
If you want, you can bundle up the files (and others) as a ZIP or TAR file, so that you can deliver everything as a single file. However, a ZIP/TAR file typically needs to be extracted before the document can be viewed. (A typical web browser won't "display" a ZIP file. Rather it will open it in some kind of archive extractor or directory browser, allowing the user to access the individual files.)
It might also be possible to embed an image file in a HTML file by base64 encoding the image, and using embedded javascript to decode the image and then insert it into the DOM ... But this is probably waaay to complicated.
I need to convert a bundle of static HTML documents into a single PDF file programmatically on the server side on a Java/J2EE platform using a batch process preferably. The pdf files would be distributed to site users for offline browsing of the web pages.
The major points of the requirements are:
The banner at the top should not be present in the final pdf document.
The navigation bar on the left should be transformed into pdf bookmarks from html hyperlinks.
All hyperlinked contents (html/pdf/doc/docx etc.) present in the web pages should be part of the final pdf document with pdf bookmarks.
Is there any standard open source way of doing this?
Try Apache FOP. I just used it to convert XML to PDF and I think you can do the same with HTML/DOM. The website has a whole section on running FOP in a Java application and there's example code for DOM to PDF.
You can try iText - but I am not sure whether it handles all that you require.
Moreover, it is always better if you explore many options and then decide what you can and cannot do. In many cases there won't be any library/API that will out of the box support all that you ask for.
You can try www.alt-soft.com Xml2PDF for this