We Keep Coding

reading remote csv file without downloading

I have requirement to read remote big csv file line by line (basically streaming). After each read I want to persist record in db. Currently I am achieving it through below code but I am not sure if it download complete file and keep it in jvm memory. I assume it is not. Can I write this code in better way using some java 8 stream features
URL url = new URL(baseurl);
HttpURLConnection urlConnection = url.openConnection();
if(connection.getResponseCode() == 200)
{
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String current;
while((current = in.readLine()) != null)
{
persist(current);
}
}

First you should use a try-with-resources statement to automatically close your streams when reading is done.
Next BufferedReader has a method BufferedReader::lines which returns a Stream<String>.
Then your code should look like this:
URL url = new URL(baseurl);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
if (connection.getResponseCode() == 200) {
try (InputStreamReader streamReader = new InputStreamReader(connection.getInputStream());
BufferedReader br = new BufferedReader(streamReader);
Stream<String> lines = br.lines()) {
lines.forEach(s -> persist(s)); //should be a method reference
}
}
Now it's up to you to decide if the code is better and your assumption is right that you don't keep the whole file in the JVM.

How to read json data from HttpURLConnection

I am using HttpURLConnection from application1 to get json data from applicaton2. 'applicaton2' sets json data in Rest response object. How can i read that json data after getting response in application1.
Sample code:
Application1:
url = "url to application2";
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.connect();
Application2":
List<ModelA> lListModelAs = Entities data;
GenericEntity<List<ModelA>> lEntities = new GenericEntity<List<ModelA>>(lListModelAs) {};
lResponse = Response.ok(lEntities ).build();
I need to read above json data from urlConnection from response.
Any hints? Thanks in advance.

After setting up your HttpURLConnection.
BufferedReader reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
StringBuilder response = new StringBuilder();
String line;
JsonObject obj = new JsonParser().parse(reader).getAsJsonObject();
boolean status = contentObj.get("status").getAsBoolean();
String Message = contentObj.get("msg").getAsString();
String Regno = contentObj.get("regno").getAsString();
String User_Id = contentObj.get("userid").getAsString();
String SessionCode = contentObj.get("sesscode").getAsString();
You can download the gson jar here enter link description here

Use dedicated library for json serialization/deserialization, Jackson for example. It will allow you to read json content directly from InputStream into POJOs that maps the response. It will be something like that:
MyRestResponse response=objectMapper.readValue(urlConnection.getInput(),MyRestResponse.class);
Looking good isnt it??
Here you have Jackson project GitHub page with usage examples.
https://github.com/FasterXML/jackson

You can use gson library
https://github.com/google/gson for parsing your data
Gson gson = new Gson();
YourClass objOfYourClass = gson.fromJson(urlConnection.getInputStream(), YourClass.class);

how to do an online xml request in java to a url that require authentication

I want to do an online xml request in java but the server responds with 401 error that means that there is an authentication that is need to access the server. I have the certfile.cer that i can use to do the authentication but i dont know how to load it in java.How can I achieve this in java? Here is part of my code.
StringBuilder answer = new StringBuilder();
URL url = new URL("www.myurl.com");
URLConnection conn = url.openConnection();
OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream());
writer.write(xml);
writer.flush();
String line;
while ((line = reader.readLine()) != null)
{
answer.append(line);
}

How to decode JSON from Java REST API request?

I'm sending data to an API from Java using POST.
What I'm trying to do is send a particular variable to the API in the POST request, and then use the value of it. But currently the value is empty. The API is definitely being called.
My Java looks like this:
String line;
StringBuffer jsonString = new StringBuffer();
try {
URL url = new URL("https://www.x.com/api.php");
String payload = "{\"variable1\":\"value1\",\"variable2\":\"value2\"}";
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setRequestProperty("Accept", "application/json");
connection.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream(), "UTF-8");
writer.write(payload);
writer.close();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
br.close();
connection.disconnect();
}
This is based on: How to send Request payload to REST API in java?
Currently the value isn't being read correctly. Am I sending it correctly in Java? Do I have to do something to decode it?

The $_POST variable is not set for all HTTP POST requests, but only for specific types, e.g application/x-www-form-urlencoded.
Since you are posting a request containing JSON entity (application/json), you need to access it as follows.
$json = file_get_contents('php://input');
$entity= json_decode($json, TRUE);

You can try to use the following code instead of your String variable payload:
List<NameValuePair> payload = new ArrayList<NameValuePair>();
payload.add(new BasicNameValuePair("variable1", "value1");
That worked for me

How can I send POST data through url.openStream()?

i'm looking for tutorial or quick example, how i can send POST data throw openStream.
My code is:
URL url = new URL("http://localhost:8080/test");
InputStream response = url.openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(response, "UTF-8"));
Could you help me ?

URL url = new URL(urlSpec);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod(method);
connection.setDoOutput(true);
connection.setDoInput(true);
// important: get output stream before input stream
OutputStream out = connection.getOutputStream();
out.write(content);
out.close();
// now you can get input stream and read.
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null) {
writer.println(line);
}

Use Apache HTTP Compoennts http://hc.apache.org/httpcomponents-client-ga/
tutorial: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html
Look for HttpPost - there are some examples of sending dynamic data, text, files and form data.

Apache HTTP Components in particular, the Client would be the best way to go.
It absracts a lot of that nasty coding you would normally have to do by hand