We Keep Coding

Processing large input Stream to List String in Java

Hey I am having a file nearly 110MB size at apache. I am reading that file into input stream and then converting that input stream to List of String based on all suggestion i find on stack overflow. But still i am facing out of memory issue.
Below is my code.
private List<String> readFromHttp(String url, PlainDiff diff) throws Exception {
HttpUrlConnection con = new HttpUrlConnection();
con.setGetUrl(url);
List<String> lines = new ArrayList<String>();
final String PREFIX = "stream2file";
final String SUFFIX = ".tmp";
final File tempFile = File.createTempFile(PREFIX, SUFFIX);
tempFile.deleteOnExit();
StringBuilder sb = new StringBuilder();
try {
InputStream data = con.sendGetInputStream();
if(data==null)
throw new UserAuthException("diff is not available at the location");
else {
try (FileOutputStream out = new FileOutputStream(tempFile)) {
IOUtils.copy(data, out);
LineIterator it = FileUtils.lineIterator(tempFile, "UTF-8");
try {
while (it.hasNext()) {
String line = it.nextLine();
lines.add(line);
sb.append(line);
}
} finally {
LineIterator.closeQuietly(it);
}
}
data.close();
diff.setLineAsString(sb.toString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
//System.out.println(lines);
return lines;
}
public InputStream sendGetInputStream() throws IOException {
String encoding = Base64.getEncoder().encodeToString(("abc:$xyz$").getBytes("UTF-8"));
URL obj = new URL(getGetUrl());
// Setup the connection
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// Set the parameters from the headers
con.setRequestMethod("GET");
con.setDoOutput(true);
con.setRequestProperty ("Authorization", "Basic " + encoding);
InputStream is;
int responseCode = con.getResponseCode();
logger.info("GET Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK) {
is = con.getInputStream();
}
else {
is = null;
}
return is;
}
Is something in memory i am doing that is consuming lot of heap? Is there a better way to do it?

Your code has multiple issues. I am not going to solve each and every issue but point that out so that you can review your code and learn to write better code.
In method readFromHttp(..):
There is no need to create a new file by IOUtils.copy(data, out);
No use of String Builder StringBuilder sb = new StringBuilder();
No use of line iterator LineIterator
And there are multiple other memory-related issues but for the time being correct these points and test with the below-mentioned code.
Change your reading lines from file to very simple way after correcting the above mistakes:
try(BufferedReader reader = new BufferedReader(new InputStreamReader(data, StandardCharsets.UTF_8))) {
for (String line; (line = reader.readLine()) != null;) {
lines.add(line);
}
}

how to read a pdf file online and save on local machine using java

Hi I was trying to read a PDF file online but after reading and writing on local. after viewing the document I am getting an error that content is not supported .
URL url1 =
new URL("http://www.gnostice.com/downloads/Gnostice_PathQuest.pdf");
byte[] ba1 = new byte[1024];
int baLength;
FileOutputStream fos1 = new FileOutputStream("/mnt/linuxabc/research_paper/Gnostice_PathQuest.pdf");
try {
URLConnection urlConn = url1.openConnection();
/* if (!urlConn.getContentType().equalsIgnoreCase("application/pdf")) {
System.out.println("FAILED.\n[Sorry. This is not a PDF.]");
} else {*/
try {
InputStream is1 = url1.openStream();
while ((baLength = is1.read(ba1)) != -1) {
fos1.write(ba1, 0, baLength);
}
fos1.flush();
fos1.close();
is1.close();
} catch (ConnectException ce) {
System.out.println("FAILED.\n[" + ce.getMessage() + "]\n");
}
// }

Your Pdf Link actually redirects to https://www.gnostice.com/downloads.asp, so there is no pdf directly behind the link.
Try with another link: check first in a browser of your choice that invoking the pdf's url render a real pdf in the browser.
The code below is practically the same as yours except for the pdf's url and the output's path, and I am also adding exception throws to the main method's signature and simply printing the content type.
It works as expected:
public class PdfFileReader {
public static void main(String[] args) throws IOException {
URL pdfUrl = new URL("http://www.crdp-strasbourg.fr/je_lis_libre/livres/Anonyme_LesMilleEtUneNuits1.pdf");
byte[] ba1 = new byte[1024];
int baLength;
try (FileOutputStream fos1 = new FileOutputStream("c:\\mybook.pdf")) {
URLConnection urlConn = pdfUrl.openConnection();
System.out.println("The content type is: " + urlConn.getContentType());
try {
InputStream is1 = pdfUrl.openStream();
while ((baLength = is1.read(ba1)) != -1) {
fos1.write(ba1, 0, baLength);
}
fos1.flush();
fos1.close();
is1.close();
} catch (ConnectException ce) {
System.out.println("FAILED.\n[" + ce.getMessage() + "]\n");
}
}
}
}
Output:
The content type is: application/pdf

private static String readPdf() throws MalformedURLException, IOException {
URL url = new URL("https://colaboracion.dnp.gov.co/CDT/Sinergia/Documentos/Informe%20al%20Congreso%20Presidencia%202017_Baja_f.pdf");
BufferedReader read = new BufferedReader(
new InputStreamReader(url.openStream()));
String i;
StringBuilder stringBuilder = new StringBuilder();
while ((i = read.readLine()) != null) {
stringBuilder.append(i);
}
read.close();
return stringBuilder.toString();
}

Java application open and play media error on Debian

I am currently working on an application and I wrote it with Java. It is downloading some media files to local computer and open it with a Java method called Desktop.getDesktop().open(file); It is working good on windows but it is not working on debian.
Here is used download from url method:
public String DownloadFromUrlAndGetPath(String DownloadUrl) {
String fileName="";
try {
URL url = new URL(DownloadUrl);
URLConnection ucon = url.openConnection();
String raw = ucon.getHeaderField("Content-Disposition");
// raw = "attachment; filename=abc.mp3"
if(raw != null && raw.indexOf("=") != -1) {
fileName = raw.split("=")[1]; //getting value after '='
fileName = fileName.replace("\"", "").trim();
} else {
return "";
}
File file = new File(Paths.get(System.getProperty("user.home"), "MyFolderToSaveFiles") +"/"+ fileName);
InputStream is = ucon.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
ByteArrayBuffer baf = new ByteArrayBuffer(5000);
int current = 0;
while ((current = bis.read()) != -1) {
try {
baf.append((int)((byte)current));
continue;
}
catch (Exception var12_13) {
}
}
FileOutputStream fos = new FileOutputStream(file);
fos.write(baf.toByteArray());
fos.flush();
fos.close();
}
catch (IOException e) {
e.getMessage();
}
}
return Paths.get(System.getProperty("user.home"), "MyFolderToSaveFiles") +"/"+ fileName;
Then I want to open that file like that:
File f = new File(url);
Desktop.getDesktop().open(f);
And the error says;
Any suggestion ? Thanks

I solved that with using this , so when I open file I am using xdg-open..

How to get VIEWSTATE before posting using HttpURLConnection (android/java)?

I am writing an Android application, and have been looking for a way to get the _VIEWSTATE from the server I want to post to so I can put it in my Http post content. A few people recommended regex, but then some other pros were strongly opposed to parsing HTML with regex. So, how to parse the _VIEWSTATE ? I am using HttpURLConnection/HttpsURLConnection in an AsyncTask. Also, don't I need to put the InputStream reader first, to get the _VIEWSTATE first? All the android examples put the input stream after the output stream. Here is what my code looks like so far (posting to one site that has three pages that have to be "clicked through"):
In my Activity, I call the Async task like this:
//execute AsyncTask for all three reports
submit_report.execute(report1, report2, report3);
My Async task doInBackground method:
class UploadReportTask extends AsyncTask<HashMap<String,String>, ProgressBar, Void> {
//this is called on task.execute
protected Void doInBackground(HashMap<String,String>...maps) {
System.out.println("Report is being uploaded");
URL url = null;
try {
url = new URL(getString(R.string.url_dnc));
} catch (MalformedURLException e) {
e.printStackTrace();
}
HttpURLConnection urlConnection = null;
try {
urlConnection = (HttpURLConnection) url.openConnection();
} catch (IOException e) {
e.printStackTrace();
}
try {
urlConnection.setDoOutput(true);
urlConnection.setRequestProperty("Accept-Charset", utf);
urlConnection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + utf);
urlConnection.setChunkedStreamingMode(0);
//For each map in maps, encode the map,
//get the headers, add the headers to the map, convert to bytes,
//then post the bytes,
//get response.
for (HashMap<String,String> map : maps){
byte[] payload = makePayload(map);
OutputStream out = new BufferedOutputStream(urlConnection.getOutputStream());
//urlConn.connect //I think this happens here
out.write(payload);
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
int length = in.read();
String result, line = reader.readLine();
result = line;
while (length != -1){
result+=line;
}
System.out.println(result);
out.flush();
out.close();
}
} catch (IOException e) {
e.printStackTrace();
}
finally{
urlConnection.disconnect();
}
return null;
}

protected String parseViewstate(String response){
int i = 0;
String viewstate = "";
while (true){
int found = response.indexOf("\"__VIEWSTATE\"", i);
if (found == -1) break;
int start = found + 38; //check numbers from start of "__V"
int end = (response.indexOf("/>", start)) -2;
viewstate = response.substring(start, end);
i = end + 1;
}return viewstate;
}

ready() in Java testing url too fast

When debugging I can confirm 6 out of 9 url connections work. One is dead, and others are sending a exceptions. Question 1:I want it to show either alive or dead, nothing else. Question 2 : when running the test, it tests too fast and it shows all are "dead" links when running (even though I know 6 work). Any suggestions?
try{
//Read File
FileReader file = new FileReader("c:\\blocked\\domains1.txt");
BufferedReader buff = new BufferedReader(file);
BufferedReader rd;
OutputStreamWriter wr;
while (true)
{
String line = buff.readLine();
if (line == null)
break;
else
try
{// Test URL Connection
URL url = new URL("http://www."+line);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
wr = new OutputStreamWriter(conn.getOutputStream());
wr.flush();
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
if( rd.ready())
{
//write to output file
System.out.println("Good URL: " + line);
urlAlive ++;
//Write to file
Path filePath = Paths.get("c:\\RevisedUrls\\revised.txt");
try
{
BufferedWriter myWriter = new BufferedWriter(new FileWriter(filePath.toFile()));
// write the user supplied data to the file plus a line separator.
myWriter.write(line + System.getProperty("line.separator"));
myWriter.close(); // immediately close the file when finished with it.
}
catch(Exception e)
{
System.out.println(e);
}
}
else// increment dead url
urlDead++;
}
catch (Exception e)
{
System.out.println(e.toString());
}
}
buff.close();
}
finally
{
int totalSites = urlAlive + urlDead;
JOptionPane.showMessageDialog(null,"Total Number of Alive Sites: " + urlAlive);
JOptionPane.showMessageDialog(null,"Total Number of Dead Sites: " + urlDead);
JOptionPane.showMessageDialog(null,"Total Number of Sites: " + totalSites);
}
}
}