We Keep Coding

Processing large input Stream to List String in Java

Hey I am having a file nearly 110MB size at apache. I am reading that file into input stream and then converting that input stream to List of String based on all suggestion i find on stack overflow. But still i am facing out of memory issue.
Below is my code.
private List<String> readFromHttp(String url, PlainDiff diff) throws Exception {
HttpUrlConnection con = new HttpUrlConnection();
con.setGetUrl(url);
List<String> lines = new ArrayList<String>();
final String PREFIX = "stream2file";
final String SUFFIX = ".tmp";
final File tempFile = File.createTempFile(PREFIX, SUFFIX);
tempFile.deleteOnExit();
StringBuilder sb = new StringBuilder();
try {
InputStream data = con.sendGetInputStream();
if(data==null)
throw new UserAuthException("diff is not available at the location");
else {
try (FileOutputStream out = new FileOutputStream(tempFile)) {
IOUtils.copy(data, out);
LineIterator it = FileUtils.lineIterator(tempFile, "UTF-8");
try {
while (it.hasNext()) {
String line = it.nextLine();
lines.add(line);
sb.append(line);
}
} finally {
LineIterator.closeQuietly(it);
}
}
data.close();
diff.setLineAsString(sb.toString());
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
//System.out.println(lines);
return lines;
}
public InputStream sendGetInputStream() throws IOException {
String encoding = Base64.getEncoder().encodeToString(("abc:$xyz$").getBytes("UTF-8"));
URL obj = new URL(getGetUrl());
// Setup the connection
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
// Set the parameters from the headers
con.setRequestMethod("GET");
con.setDoOutput(true);
con.setRequestProperty ("Authorization", "Basic " + encoding);
InputStream is;
int responseCode = con.getResponseCode();
logger.info("GET Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK) {
is = con.getInputStream();
}
else {
is = null;
}
return is;
}
Is something in memory i am doing that is consuming lot of heap? Is there a better way to do it?

Your code has multiple issues. I am not going to solve each and every issue but point that out so that you can review your code and learn to write better code.
In method readFromHttp(..):
There is no need to create a new file by IOUtils.copy(data, out);
No use of String Builder StringBuilder sb = new StringBuilder();
No use of line iterator LineIterator
And there are multiple other memory-related issues but for the time being correct these points and test with the below-mentioned code.
Change your reading lines from file to very simple way after correcting the above mistakes:
try(BufferedReader reader = new BufferedReader(new InputStreamReader(data, StandardCharsets.UTF_8))) {
for (String line; (line = reader.readLine()) != null;) {
lines.add(line);
}
}

Android - Size in chars of an http response

I am not an pro developing android. I wanted to download a JSON object from my server, but only code I could find was this:
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
// Only display the first 500 characters of the retrieved
// web page content.
int len = 500;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d("ServerConnection", "The response is: " + response);
is = conn.getInputStream();;
//is.
// Convert the InputStream into a string
String contentAsString = readIt(is, len);
return contentAsString;
// Makes sure that the InputStream is closed after the app is
// finished using it.
} catch (MalformedURLException e) {
//
return "error";
} catch (IOException e) {
//
return "error";
} finally {
if (is != null) {
is.close();
}
}
}
And it works fine, I cant understand. But it has a int len = 500, and my returned json is cropped to 500 chars. I tried changing to a great number, but it puts spaces at the end. How can I know the size in chars of the String contained by the InputSteam?
Thanks

You can check the Content-Length header value of your response:
Map<String, List<String>> headers = connection.getHeaderFields();
for (Entry<String, List<String>> header : headers.entrySet()) {
if(header.getKey().equals("Content-Legth")){
len=Integer.parseInt(header.getValue());
}
}
or you can your response in a buffered reader like this:
InputStream is = connection.getInputStream();
InputStreamReader reader = new InputStreamReader(is);
StringBuilder builder = new StringBuilder();
int c = 0;
while ((c = reader.read()) != -1) {
builder.append((char) c);
}

Yout can use Apache Commons IO IOUtils.toString to convert InputStream to String or use Gson to read object from input stream directly:
return gson.fromJson(new InputStreamReader(inputStream), YourType.class);

URLConnection and POST method android

I am trying to make a post request using the reference of this documentation. But the problem is that the PHP developer at the other end is not able to receive the value of the parameter hence is not able to send a proper response. Am I missing something out here.
// Edits ;
I am making a HTTP Post request. As you can seen the code below. I am writing the arguments and parameters (location_id=3) to the outputstream. I have also pasted the code for PHP which i have been using. Now the problem is:
The parameter value ( which is 3 ) is not received at the PHP code so I am getting a response which is surrounded by the else block. So I just want to know if there is an error in the android code or the PHP code
#Override
protected Boolean doInBackground(String... params) {
Log.d(Constants.LOG_TAG,Constants.FETCH_ALL_THEMES_ASYNC_TASK);
Log.d(Constants.LOG_TAG," The url to be fetched "+params[0]);
try {
url = new URL(params[0]);
urlConnection = (HttpURLConnection) url.openConnection();
// /* optional request header */
// urlConnection.setRequestProperty("Content-Type", "application/json");
//
// /* optional request header */
// urlConnection.setRequestProperty("Accept", "application/json");
/* for Get request */
urlConnection.setChunkedStreamingMode(0);
urlConnection.setDoOutput(true);
urlConnection.setDoInput(true);
urlConnection.setRequestMethod("POST");
List<BasicNameValuePair> nameValuePairs = new ArrayList<BasicNameValuePair>();
nameValuePairs.add(new BasicNameValuePair("location_id",params[1]));
outputStream = urlConnection.getOutputStream();
BufferedWriter bufferedWriter = new BufferedWriter(new OutputStreamWriter(outputStream));
bufferedWriter.write(writeToOutputStream(nameValuePairs));
int statusCode = urlConnection.getResponseCode();
/* 200 represents HTTP OK */
if (statusCode == 200) {
inputStream = new BufferedInputStream(urlConnection.getInputStream());
response = convertInputStreamToString(inputStream);
Log.d(Constants.LOG_TAG, " The response is " + response);
return true;
}
else {
return false;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if(inputStream != null){
inputStream.close();
}
if(outputStream != null){
outputStream.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
return false;
}
// Here is the code for writeToOutputStream
public String writeToOutputStream(List<BasicNameValuePair> keyValuePair)throws UnsupportedEncodingException{
String result="";
boolean firstTime = true;
for(BasicNameValuePair pair : keyValuePair){
if(firstTime){
firstTime = false;
}
else{
result = result.concat("&");
}
result = result + URLEncoder.encode(pair.getKey(), "UTF-8");
result = result + "=";
result = result+ URLEncoder.encode(pair.getValue(),"UTF-8");
}
Log.d(Constants.LOG_TAG," The result is "+result);
return result;
}
// Here is the code for convertInputStream to String
public String convertInputStreamToString(InputStream is) throws IOException {
String line="";
String result="";
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(is));
while((line = bufferedReader.readLine()) != null){
Log.d(Constants.LOG_TAG," The line value is "+line);
result += line;
}
/* Close Stream */
if(null!=inputStream){
inputStream.close();
}
return result;
}
Here is the PHP CODE
<?php
include 'config.php';
header ('Content-Type:application/json');
if(isset($_POST['location_id']))
{
$id=$_POST['location_id'];
$selectThemeQuery = mysql_query("select theme_id from location_theme where location_id='$id'",$conn) or die (mysql_error());
$noRows = mysql_num_rows($selectThemeQuery);
//echo "HI";
if($noRows > 0)
{
$result = array();
while($row = mysql_fetch_assoc($selectThemeQuery))
{
$themeid = $row['theme_id'];
//echo "HI";
$selectNameQuery = mysql_query("select theme_name,theme_image from theme where theme_id='$themeid'",$conn) or die(mysql_error());
$numRows = mysql_num_rows($selectNameQuery);
if($numRows > 0)
{
while($rows = mysql_fetch_assoc($selectNameQuery))
{
$name = $rows['theme_name'];
$image = $rows['theme_image'];
$result[] = array('theme_id'=>$themeid,'theme_name'=>$name, 'theme_image'=>$image);
}
}
}
//echo json_encode($result);
echo json_encode("Hi");
}
else
{
$data2[] = array('Notice'=>false);
echo json_encode($data2);
}
}
else
{
echo "Not Proper Data";
}
?>

Remove:
urlConnection.setChunkedStreamingMode(0);
You use a buffered writer so it can only buffer instead of write.
To force all been written:
bufferedWriter.write(writeToOutputStream(nameValuePairs));
bufferedWriter.flush();
And then ask for a response code. And don't call a response code a status code.
writeToOutputStream() ??? What a terrible name. That function does not write to an output stream. It justs makes a text string.

For Android, I would suggest using a library like Spring-Android.
Spring-Android contains a RestTemplate class, which is a quite effective REST-Client. For example, a simple POST request would be...
myRestTemplate.exchange("http://www.example.net", HttpMethod.POST, new HttpEntity( ...some JSON string ...), String.class );
To create the JSON String, I suggest a library like Jackson, which should work fine on Android, see for example here. Not sure if Jackson integrates as fine in Spring-Android as it does in Spring-Web, but in any case, using it to create the Json Strings manually should work just fine.

for post method
create a string builder first
StringBuilder strbul=new StringBuilder();
then append the data like
strbul.append("name=sangeet&state=kerala");
then write to output stream like
httpurlconnection.getOutput().write(strbul.toString().getBytes("UTF-8"));
php script will recieve that data on
$_POST['name'] and $_POST['state']

Grabbing JSON works from one link, not from another

I'm doing a simple JSON grab from two links with the same code. I'm doing it two separate times, so the cause of my issue isn't because they're running into each other or something.
Here is my code:
#Override
protected String doInBackground(Object... params) {
try {
URL weatherUrl = new URL("my url goes here");
HttpURLConnection connection = (HttpURLConnection) weatherUrl
.openConnection();
connection.connect();
responseCode = connection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
InputStream inputStream = connection.getInputStream();
Reader reader = new InputStreamReader(inputStream);
int contentLength = connection.getContentLength();
char[] charArray = new char[contentLength];
reader.read(charArray);
String responseData = new String(charArray);
Log.v("test", responseData);
When I try this with:
http://www.google.com/calendar/feeds/developer-calendar#google.com/public/full?alt=json
I get an error of having an array lenth of -1
For this link:
http://api.openweathermap.org/data/2.5/weather?id=5815135
It returns fine and I get a log of all of the JSON. Does anyone have any idea why?
Note: I tried stepping through my code in debug mode, but I couldn't catch anything. I also downloaded a Google chrome extension for parsing json in the browser and both urls look completely valid. I'm out of ideas.

Log this: int contentLength = connection.getContentLength();
I don't see the google url returning a content-length header.
If you just want String output from a url, you can use Scanner and URL like so:
Scanner s = new Scanner(new URL("http://www.google.com").openStream(), "UTF-8").useDelimiter("\\A");
out = s.next();
s.close();
(don't forget try/finally block and exception handling)
The longer way (which allows for progress reporting and such):
String convertStreamToString(InputStream is) throws UnsupportedEncodingException {
BufferedReader reader = new BufferedReader(new
InputStreamReader(is, "UTF-8"));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null)
sb.append(line + "\n");
} catch (IOException e) {
// Handle exception
} finally {
try {
is.close();
} catch (IOException e) {
// Handle exception
}
}
return sb.toString();
}
}
and then call String response = convertStreamToString( inputStream );

upload image from android phone to the server

I am new to android trying to work on uploading image in to server.
Got some sample code from the Internet. But it is showing some error in the line which am not able to handle it. Can any one help me in this case. And the link which is fetch the code is http://blog.sptechnolab.com/2011/03/09/android/android-upload-image-to-server/
and the error is getting is
"The method encodeBytes(byte[]) is undefined for the type Base64"
and the corresponding
i have even downloaded base64.java file in the project

There is no encodeBytes in the API. Use encodeToString.

You can use these methods instead
public static String encodeToString (byte[] input, int offset, int len, int flags)
Since: API Level 8
Base64-encode the given data and return a newly allocated String with the result.
Parameters
input : the data to encode
offset : the position within the input array at which to start
len : The number of bytes of input to encode
flags : controls certain features of the encoded output.
Passing DEFAULT results in output that adheres to RFC 2045.
public static String encodeToString (byte[] input, int flags)
Since: API Level 8
Base64-encode the given data and return a newly allocated String with the result.
Parameters
input : the data to encode
flags :controls certain features of the encoded output.
Passing DEFAULT results in output that adheres to RFC 2045.

Whaaaa!
You MUST indent your code!
For every { you open you should space to the right so you'll see better what your code is doing:
This is good:
try {
something();
} catch (Exception e) {
weMessedUp();
if (e == i)
{
lol();
}
}
This is bad:
try {
something();
} catch (Exception e) {
weMessedUp();
if (e == i)
{
lol();
}
}
It's only for reading, your programs will be faster to understand if in one week you want to modify something.
In eclipse to indent, do ctrl + a to select your whole code, then ctrl + i to indent.
This doesn't answer you question but will help others to answer and you to improve your skills.

You can simply open the file as a bytestream and send it as a stream to your httpconnection?
Opening a file as a stream like so:
File inFile = new File(fileName);
BufferedReader br = new BufferedReader(
new InputStreamReader(
new FileInputStream(inFile)
)
);
URL url = new URL("http://www.google.com");
URLConnection connection = url.openConnection();
connection.setDoOutput(true);
OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
while ((decodedString = br.readLine()) != null) {
out.write(decodedString);
}
out.close();
This is the implementation for reading a file line by line. Not sure if it will work given the encoding of an image without line breaks, but you should be able to reengineer to stream byte-by-byte without much trouble.

public class UploadImage extends Activity {
InputStream inputStream;
#Override
public void onCreate(Bundle icicle) {
super.onCreate(icicle);
setContentView(R.layout.main);
Bitmap bitmap = BitmapFactory.decodeResource(getResources(),R.drawable.icon); ByteArrayOutputStream <span id="IL_AD5" class="IL_AD">stream</span> = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 90, stream); //compress to which format you want.
byte [] byte_arr = stream.toByteArray();
String image_str = Base64.encodeBytes(byte_arr);
ArrayList<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
nameValuePairs.add(new BasicNameValuePair("image",image_str));
try{
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://10.0.2.2/Upload_image_ANDROID/upload_image.php");
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs));
HttpResponse response = httpclient.execute(httppost);
String the_string_response = convertResponseToString(response);
Toast.makeText(UploadImage.this, "Response " + the_string_response, Toast.LENGTH_LONG).show();
}catch(Exception e){
Toast.makeText(UploadImage.this, "ERROR " + e.getMessage(), Toast.LENGTH_LONG).show();
System.out.println("Error in http connection "+e.toString());
}
}
public String convertResponseToString(HttpResponse response) throws IllegalStateException, IOException{
String res = "";
StringBuffer buffer = new StringBuffer();
inputStream = response.getEntity().getContent();
int contentLength = (int) response.getEntity().getContentLength(); //getting content length…..
Toast.makeText(UploadImage.this, "contentLength : " + contentLength, Toast.LENGTH_LONG).show();
if (contentLength < 0){
}
else{
byte[] data = new byte[512];
int len = 0;
try
{
while (-1 != (len = inputStream.read(data)) )
{
buffer.append(new String(data, 0, len)); //converting to string and appending to stringbuffer…..
}
}
catch (IOException e)
{
e.printStackTrace();
}
try
{
inputStream.close(); // closing the stream…..
}
catch (IOException e)
{
e.printStackTrace();
}
res = buffer.toString(); // converting stringbuffer to string…..
Toast.makeText(UploadImage.this, "Result : " + res, Toast.LENGTH_LONG).show();
//System.out.println("Response => " + EntityUtils.toString(response.getEntity()));
}
return res;