We Keep Coding

how to read a pdf file online and save on local machine using java

Hi I was trying to read a PDF file online but after reading and writing on local. after viewing the document I am getting an error that content is not supported .
URL url1 =
new URL("http://www.gnostice.com/downloads/Gnostice_PathQuest.pdf");
byte[] ba1 = new byte[1024];
int baLength;
FileOutputStream fos1 = new FileOutputStream("/mnt/linuxabc/research_paper/Gnostice_PathQuest.pdf");
try {
URLConnection urlConn = url1.openConnection();
/* if (!urlConn.getContentType().equalsIgnoreCase("application/pdf")) {
System.out.println("FAILED.\n[Sorry. This is not a PDF.]");
} else {*/
try {
InputStream is1 = url1.openStream();
while ((baLength = is1.read(ba1)) != -1) {
fos1.write(ba1, 0, baLength);
}
fos1.flush();
fos1.close();
is1.close();
} catch (ConnectException ce) {
System.out.println("FAILED.\n[" + ce.getMessage() + "]\n");
}
// }

Your Pdf Link actually redirects to https://www.gnostice.com/downloads.asp, so there is no pdf directly behind the link.
Try with another link: check first in a browser of your choice that invoking the pdf's url render a real pdf in the browser.
The code below is practically the same as yours except for the pdf's url and the output's path, and I am also adding exception throws to the main method's signature and simply printing the content type.
It works as expected:
public class PdfFileReader {
public static void main(String[] args) throws IOException {
URL pdfUrl = new URL("http://www.crdp-strasbourg.fr/je_lis_libre/livres/Anonyme_LesMilleEtUneNuits1.pdf");
byte[] ba1 = new byte[1024];
int baLength;
try (FileOutputStream fos1 = new FileOutputStream("c:\\mybook.pdf")) {
URLConnection urlConn = pdfUrl.openConnection();
System.out.println("The content type is: " + urlConn.getContentType());
try {
InputStream is1 = pdfUrl.openStream();
while ((baLength = is1.read(ba1)) != -1) {
fos1.write(ba1, 0, baLength);
}
fos1.flush();
fos1.close();
is1.close();
} catch (ConnectException ce) {
System.out.println("FAILED.\n[" + ce.getMessage() + "]\n");
}
}
}
}
Output:
The content type is: application/pdf

private static String readPdf() throws MalformedURLException, IOException {
URL url = new URL("https://colaboracion.dnp.gov.co/CDT/Sinergia/Documentos/Informe%20al%20Congreso%20Presidencia%202017_Baja_f.pdf");
BufferedReader read = new BufferedReader(
new InputStreamReader(url.openStream()));
String i;
StringBuilder stringBuilder = new StringBuilder();
while ((i = read.readLine()) != null) {
stringBuilder.append(i);
}
read.close();
return stringBuilder.toString();
}

How to convert HTML String to PDF using ConvertAPI (without a physical file)

Previously I was using http://do.convertapi.com/Web2Pdf to convert HTML String to PDF using a simple GET request (not POST) using Java. The entire content was passed using curl parameter.
However, that API seems to have stopped working recently. I'm trying to port over to https://v2.convertapi.com/web/to/pdf but I cannot find a sample to do the same using the new API either with GET or POST.
Can someone provide an example to make a GET or POST request using Java?
UPDATE: I have managed to make it work.
private static final String WEB2PDF_API_URL = "https://v2.convertapi.com/html/to/pdf";
private static final String WEB2PDF_SECRET = "secret-here";
String htmlContent = "valid HTML content here";
URL apiUrl = new URL(WEB2PDF_API_URL + "?secret=" + WEB2PDF_SECRET + "&download= attachment&PageOrientation=landscape&MarginLeft=0&MarginRight=0&MarginTop=0&MarginBottom=0");
HttpURLConnection connection = null;
ByteArrayOutputStream buffer = null;
connection = (HttpURLConnection) apiUrl.openConnection();
connection.setRequestProperty("Content-Disposition", "attachment; filename=\"data.html\"");
connection.setRequestProperty("Content-Type", "application/octet-stream");
connection.setRequestMethod("POST");
connection.setConnectTimeout(60000);
connection.setReadTimeout(60000);
connection.setDoOutput(true);
/* write request */
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
writer.write(htmlContent);
writer.flush();
writer.close();
/* read response */
String responseMessage = connection.getResponseMessage();
logger.info("responseMessage: " + responseMessage);
int statusCode = connection.getResponseCode();
logger.info("statusCode: " + statusCode);
if (statusCode == HttpURLConnection.HTTP_OK) {
logger.info("HTTP status code OK");
// parse output
InputStream is = connection.getInputStream();
buffer = new ByteArrayOutputStream();
int nRead;
byte[] data = new byte[16384];
while ((nRead = is.read(data, 0, data.length)) != -1) {
buffer.write(data, 0, nRead);
}
buffer.flush();
byte[] attachmentData = buffer.toByteArray();
Multipart content = new MimeMultipart();
...
MimeBodyPart attachment = new MimeBodyPart();
InputStream attachmentDataStream = new ByteArrayInputStream(attachmentData);
attachment.setFileName("filename-" + Long.toHexString(Double.doubleToLongBits(Math.random())) + ".pdf");
attachment.setContent(attachmentDataStream, "application/pdf");
content.addBodyPart(attachment);
...
}

You can easily push HTML string as file. I do not have JAVA example but the C# demo will give you right path.
using System;
using System.IO;
using System.Net.Http;
using System.Text;
class MainClass {
public static void Main (string[] args) {
var url = new Uri("https://v2.convertapi.com/html/to/pdf?download=attachment&secret=<YourSecret>");
var htmlString = "<!doctype html><html lang=en><head><meta charset=utf-8><title>ConvertAPI test</title></head><body>This page is generated from HTML string.</body></html>";
var content = new StringContent(htmlString, Encoding.UTF8, "application/octet-stream");
content.Headers.Add("Content-Disposition", "attachment; filename=\"data.html\"");
using (var resultFile = File.OpenWrite(#"C:\Path\to\result\file.pdf"))
{
new HttpClient().PostAsync(url, content).Result.Content.CopyToAsync(resultFile).Wait();
}
}
}

java sending images over http get doesn't work

I'm trying to send png to server using http get request. I'm using code below to to so.
String encoding (on client):
byte[] bytes = Files.readAllBytes(Paths.get(chosenFile.getPath()));
String image = new String(bytes, "ISO-8859-1");
Then I send http get request to server with String image.
Server receive it but I get image that is not the same as one I sent (I can't open one I received). I'm using URLEncoder.encode(image, "ISO-8859-1") to encode url for http get request. When I use URLEncoder.encode(image, "UTF-8"), the same happens.
Why this doesn't work?
Is there any better way of doing this kind of stuff?
UPD #0
Code for sending an image:
private void jMenuItem5ActionPerformed(java.awt.event.ActionEvent evt) {
JFileChooser chooser= new JFileChooser();
int choice = chooser.showOpenDialog(this);
if (choice != JFileChooser.APPROVE_OPTION) return;
File chosenFile = chooser.getSelectedFile();
try {
byte[] bytes = Files.readAllBytes(Paths.get(chosenFile.getPath()));
String image = new String(bytes, "ISO-8859-1");
if(image != null){
boolean allsent = false;
long k = 0;
String query = "0";
while(!allsent){
String s = image;
String send = "";
long q;
if(k+400>image.length()){
q=image.length();
allsent=true;
}
else q = k+400;
for(long i=k;i<q;i++)
send+=s.charAt((int) i);
System.out.println(send);
String response = new HTTP().GET(Constants.ADDRESS_ADDIMAGE
+ "?" + Constants.USERNAME + "=" + URLEncoder.encode(user, "UTF-8")
+ "&" + Constants.IMAGE + "=" + URLEncoder.encode(send, "ISO-8859-1")
+ "&" + Constants.QUERY + "=" + URLEncoder.encode(query, "UTF-8"));
k+=400;
query="1";
}
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
Note:
HTTP().GET() invokes standard http get.
UPD #1
Server code
#GET
#Path("/addimage")
#Produces(MediaType.APPLICATION_JSON)
public String addImage(#QueryParam("username") String uname, #QueryParam("image") String image, #QueryParam("query") String query) {
if(query.equals("0")){
String s = image;
JDBC.addImage("ABase", "MarketLogin", "image", uname, s);
}
else{
String s = JDBC.selectDB("ABase", "MarketLogin", "image", uname, "username") + image;
JDBC.addImage("ABase", "MarketLogin", "image", uname, s);
}
return "1";
}
Note:
JDBC is class for updating mysql DB. Server is expecting String encoded by ISO-8859-1.

How about using a HttpURLConnection and sending bytes instead of strings.
HttpURLConnection connection;
ByteArrayOutputStream baos;
try {
URL url = new URL("http://www.somewebsite.com");
connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "image/jpeg");
connection.setRequestMethod("POST");
baos = new ByteArrayOutputStream();
baos.write(bytes); // your bytes here
baos.writeTo(connection.getOutputStream());
}
finally {
if(baos != null)
baos.close();
if(osw != null)
osw.close();
if(connection != null)
connection.disconnect();
}

Download the entire webpage

There are ways to download an entire webpage, using HTMLEditorKit. However, I need to download an entire webpage which needs scrolling in order to load its entire content. This technology is achieved most commonly through JavaScript bundled with Ajax.
Q.: Is there a way to trick the destined webpage, using only Java code, in order to download its full content?
Q.2: If this is not possible only with Java, then is it possible in combination with JavaScript?
Simple notice, what I wrote:
public class PageDownload {
public static void main(String[] args) throws Exception {
String webUrl = "...";
URL url = new URL(webUrl);
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
HTMLEditorKit htmlKit = new HTMLEditorKit();
HTMLDocument htmlDoc = (HTMLDocument) htmlKit.createDefaultDocument();
HTMLEditorKit.Parser parser = new ParserDelegator();
HTMLEditorKit.ParserCallback callback = htmlDoc.getReader(0);
parser.parse(br, callback, true);
for (HTMLDocument.Iterator iterator = htmlDoc.getIterator(HTML.Tag.IMG);
iterator.isValid(); iterator.next()) {
AttributeSet attributes = iterator.getAttributes();
String imgSrc = (String) attributes.getAttribute(HTML.Attribute.SRC);
if (imgSrc != null && (imgSrc.endsWith(".jpg") || (imgSrc.endsWith(".jpeg"))
|| (imgSrc.endsWith(".png")) || (imgSrc.endsWith(".ico"))
|| (imgSrc.endsWith(".bmp")))) {
try {
downloadImage(webUrl, imgSrc);
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
}
}
}
private static void downloadImage(String url, String imgSrc) throws IOException {
BufferedImage image = null;
try {
if (!(imgSrc.startsWith("http"))) {
url = url + imgSrc;
} else {
url = imgSrc;
}
imgSrc = imgSrc.substring(imgSrc.lastIndexOf("/") + 1);
String imageFormat = null;
imageFormat = imgSrc.substring(imgSrc.lastIndexOf(".") + 1);
String imgPath = null;
imgPath = "..." + imgSrc + "";
URL imageUrl = new URL(url);
image = ImageIO.read(imageUrl);
if (image != null) {
File file = new File(imgPath);
ImageIO.write(image, imageFormat, file);
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}

Use HtmlUnit library to get all text and images/css files.
HTMLUnit [link] htmlunit.sourceforge.net
1) To download text content use code on below link s
all Text content [link] How to get a HTML page using HtmlUnit
Specific tag such as span [link] how to get text between a specific span with HtmlUnit
2) To get images/files use below [link] How can I tell HtmlUnit's WebClient to download images and css?

Yes you can trick a a webpage to download on your locals by Java code. You can not Download HTMl Static content by Java Script. JavaScript is not providing you to create a files as Java Provides.
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
public class HttpDownloadUtility {
private static final int BUFFER_SIZE = 4096;
/**
* Downloads a file from a URL
* #param fileURL HTTP URL of the file to be downloaded
* #param saveDir path of the directory to save the file
* #throws IOException
*/
public static void downloadFile(String fileURL, String saveDir)
throws IOException {
URL url = new URL(fileURL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
int responseCode = httpConn.getResponseCode();
// always check HTTP response code first
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = httpConn.getHeaderField("Content-Disposition");
String contentType = httpConn.getContentType();
int contentLength = httpConn.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10,
disposition.length() - 1);
}
} else {
// extracts file name from URL
fileName = fileURL.substring(fileURL.lastIndexOf("/") + 1,
fileURL.length());
}
System.out.println("Content-Type = " + contentType);
System.out.println("Content-Disposition = " + disposition);
System.out.println("Content-Length = " + contentLength);
System.out.println("fileName = " + fileName);
// opens input stream from the HTTP connection
InputStream inputStream = httpConn.getInputStream();
String saveFilePath = saveDir + File.separator + fileName;
// opens an output stream to save into file
FileOutputStream outputStream = new FileOutputStream(saveFilePath);
int bytesRead = -1;
byte[] buffer = new byte[BUFFER_SIZE];
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
}
outputStream.close();
inputStream.close();
System.out.println("File downloaded");
} else {
System.out.println("No file to download. Server replied HTTP code: " + responseCode);
}
httpConn.disconnect();
}
}

You can achieve this with Selenium Webdriver java classes...
https://code.google.com/p/selenium/wiki/GettingStarted
Generally, webdriver is used for testing, but it is able to emulate a user scrolling down the page, until the page stops changing, and then you can use java code to save the content to a file.

You can do it using IDM's grabber.
This should help:
https://www.internetdownloadmanager.com/support/idm-grabber/grabber_wizard.html

Upload video to YouTube Data API v3 without using the Java Client Library

I'm developing a service to upload videos to YouTube and I got it working on V2, but I'm migrating to V3 but I can't find REST endpoints to hit.
The Resumable Upload is well docummented here but the Direct Upload is only documented as a Python script reference.
My process can't support Resumable Uploads and I can't use the Java Client Library because I don't have the files stored locally. For V2 I used a different approach that allowed me to do piping.
Is there a way to do this on V3???
String lineEnd = "\r\n";
String twoHyphens = "--";
String access_token = "A1S2D3F4G5H6J7K8L9";
String gameClipID = "gameClipID";
String sourceServerURL = "sourceServerURL";
String boundary = "Boundary" + gameClipID;
String closingBoundary = lineEnd + twoHyphens + boundary + twoHyphens;
URL youTubeURL = new URL("http://uploads.gdata.youtube.com/feeds/api/users/default/uploads");
HttpURLConnection youTubeConnection = (HttpURLConnection)youTubeURL.openConnection();
youTubeConnection.setRequestMethod("POST");
youTubeConnection.setDoOutput(true);
youTubeConnection.setDoInput(true);
// Set Headers
youTubeConnection.setRequestProperty("Authorization" , "GoogleLogin auth=" + auth);
youTubeConnection.setRequestProperty("GData-Version", "2");
youTubeConnection.setRequestProperty("X-GData-Key", "key=" + developerKey);
youTubeConnection.setRequestProperty("Slug", gameClipID);
youTubeConnection.setRequestProperty("Accept", "*/*");
youTubeConnection.setRequestProperty("Accept-Charset", "UTF-8");
youTubeConnection.setRequestProperty("Content-Type", "multipart/related;boundary=" + boundary);
String content = prepareContent(boundary, "Title", "Description", keywords);
Long totalLength = (long)content.length() + (long)fileSize + (long)closingBoundary.length();
youTubeConnection.setRequestProperty("Content-Length", totalLength.toString());
youTubeConnection.setRequestProperty("Connection", "close");
URL sourceURL = new URL(sourceServerURL);
HttpURLConnection sourceConnection = (HttpURLConnection) sourceURL.openConnection();
sourceConnection.setAllowUserInteraction(false);
sourceConnection.setRequestProperty("Authorization", access_token);
sourceConnection.connect();
BufferedInputStream bis = new BufferedInputStream(sourceConnection.getInputStream());
BufferedOutputStream request = new BufferedOutputStream(youTubeConnection.getOutputStream());
request.write(content.getBytes("UTF-8"));
boolean eof = false;
byte[] input = new byte[4096];
while (!eof) {
int length = bis.read(input);
if (length == -1) {
eof = true;
} else {
request.write(input, 0, length);
}
}
request.write(closingBoundary.getBytes("UTF-8"));
request.flush();
request.close();
bis.close();
sourceConnection.disconnect();
InputStream responseStream = new BufferedInputStream(youTubeConnection.getInputStream());
BufferedReader responseStreamReader = new BufferedReader(new InputStreamReader(responseStream));
String line;
StringBuilder stringBuilder = new StringBuilder();
while ((line = responseStreamReader.readLine()) != null) {
stringBuilder.append(line);
}
responseStreamReader.close();
String response = stringBuilder.toString();
responseStream.close();
youTubeConnection.disconnect();
return YouTubeClient.parseXMLResponse(response);

Direct upload equivalent of the documentation, would be just setting uploadType=direct.
You can set directUploadEnabled to true in this example and see the Post request to make sure.
https://github.com/youtube/api-samples/blob/master/java/src/main/java/com/google/api/services/samples/youtube/cmdline/data/UploadVideo.java#L136
Also here's the source code of the directUpload so you can build this manually.
https://code.google.com/p/google-api-java-client/source/browse/google-api-client/src/main/java/com/google/api/client/googleapis/media/MediaHttpUploader.java#345