We Keep Coding

reading remote csv file without downloading

I have requirement to read remote big csv file line by line (basically streaming). After each read I want to persist record in db. Currently I am achieving it through below code but I am not sure if it download complete file and keep it in jvm memory. I assume it is not. Can I write this code in better way using some java 8 stream features
URL url = new URL(baseurl);
HttpURLConnection urlConnection = url.openConnection();
if(connection.getResponseCode() == 200)
{
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String current;
while((current = in.readLine()) != null)
{
persist(current);
}
}

First you should use a try-with-resources statement to automatically close your streams when reading is done.
Next BufferedReader has a method BufferedReader::lines which returns a Stream<String>.
Then your code should look like this:
URL url = new URL(baseurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if (connection.getResponseCode() == 200) {
try (InputStreamReader streamReader = new InputStreamReader(connection.getInputStream());
BufferedReader br = new BufferedReader(streamReader);
Stream<String> lines = br.lines()) {
lines.forEach(s -> persist(s)); //should be a method reference
}
}
Now it's up to you to decide if the code is better and your assumption is right that you don't keep the whole file in the JVM.

Base64 image data is not getting in android

I am using HTTPURLConnection to connect to server and my response contains Base64 image data. When trying to read the response using getInputStream its not reading the complete response, breaks in between. My response contains list of objects in JSON format and each object contains BASE64 image data. Reading breaks while trying to read the first image data from the first object. Though its not showing any error it displays till half of the image data.How do i get the full response?? Here is my code
InputStream is = httpURLConnection.getInputStream();
byte[] b = new byte[1024];
StringBuffer buffer=new StringBuffer();
while ( is.read(b) != -1){
buffer.append(new String(b));
System.out.println("Read= "+is.read());
}
System.out.println(buffer);

Have you tried the example code from Google?
URL url = new URL("http://www.android.com/");
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
try {
InputStream in = new BufferedInputStream(urlConnection.getInputStream());
readStream(in);
} finally {
urlConnection.disconnect();
}
where readStream is your own method.
Reading from a BufferedInputStream is easier and faster.

HttpURLConnection response is incorrect

When using this code below to make a get request:
private String get(String inurl, Map headers, boolean followredirects) throws MalformedURLException, IOException {
URL url = new URL(inurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setInstanceFollowRedirects(followredirects);
// Add headers to request.
Iterator entries = headers.entrySet().iterator();
while (entries.hasNext()) {
Entry thisEntry = (Entry) entries.next();
Object key = thisEntry.getKey();
Object value = thisEntry.getValue();
connection.addRequestProperty((String)key, (String)value);
}
// Attempt to parse
InputStream stream = connection.getInputStream();
InputStreamReader isReader = new InputStreamReader(stream );
BufferedReader br = new BufferedReader(isReader );
System.out.println(br.readLine());
// Disconnect
connection.disconnect();
return connection.getHeaderField("Location");
}
The resulting response is completely nonsensical (e.g ���:ks�6��﯐9�rђ� e��u�n�qש�v���"uI*�W��s)
However I can see in Wireshark that the response is HTML/XML and nothing like the string above. I've tried a myriad of different methods for parsing the InputStream but I get the same result each time.
Please note: this only happens when it's HTML/XML, plain HTML works.
Why is the response coming back in this format?
Thanks in advance!
=== SOLVED ===
Gah, got it!
The server is compressing the response when it contains XML, so I needed to use GZIPInputStream instead of InputSream.
GZIPInputStream stream = new GZIPInputStream(connection.getInputStream());
Thanks anyway!

use an UTF-8 encoding in input stream like below
InputStreamReader isReader = new InputStreamReader(stream, "UTF-8");

Read Status code

conn = (HttpURLConnection) connectURL.openConnection();
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setUseCaches(false);
conn.setRequestMethod("POST");
conn.connect();
int code = conn.getResponseCode();
I have successfully established a connection. I am trying to pass the information over the internet.When the url is opened via browser I am getting response as
{"status":"0","responseCode":"1001","response":"Wrong Settings."}
For correct status is returned as 1.
Is there any method where I can get the status only.I have been trying the following methods but every time I am getting code (below is code snippet) as -1 irrespect of status code when I am verifying manually via browser

This is a JSON text. You will need to use a JSON library.

int code = conn.getResponseCode();
this method returns http status code, for http status codes see
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
while the response code you want to retrieve is actually the response string returned by the server.
To read this use:
try {
InputStream in = new BufferedInputStream(conn.getInputStream());
readStream(in);//method to read characters from stream.
finally {
urlConnection.disconnect();
}
}

You can add below code for get Response string from your connection.
OutputStream connectionOutput = null;
connectionOutput=connection.getOutputStream();
connectionOutput.write(requestJson.toString().getBytes());
connectionOutput.flush();
connectionOutput.close();
inputStream = new BufferedInputStream(connection.getInputStream());
ByteArrayOutputStream dataCache = new ByteArrayOutputStream();
// Fully read data
byte[] buff = new byte[1024];
int len;
while ((len = inputStream.read(buff)) >= 0) {
dataCache.write(buff, 0, len);
}
// Close streams
dataCache.close();
Now get Response string of json like below.
String jsonString = new String(dataCache.toByteArray()).trim();
JSONObject mJsonobject=new JSONObject(jsonString);
You can now parse your key from this mJsonobject Object.

How can I send POST data through url.openStream()?

i'm looking for tutorial or quick example, how i can send POST data throw openStream.
My code is:
URL url = new URL("http://localhost:8080/test");
InputStream response = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response, "UTF-8"));
Could you help me ?

URL url = new URL(urlSpec);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(method);
connection.setDoOutput(true);
connection.setDoInput(true);
// important: get output stream before input stream
OutputStream out = connection.getOutputStream();
out.write(content);
out.close();
// now you can get input stream and read.
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
writer.println(line);
}

Use Apache HTTP Compoennts http://hc.apache.org/httpcomponents-client-ga/
tutorial: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
Look for HttpPost - there are some examples of sending dynamic data, text, files and form data.

Apache HTTP Components in particular, the Client would be the best way to go.
It absracts a lot of that nasty coding you would normally have to do by hand