We Keep Coding

How to deal with NumberFormatException when reading from a csv file [duplicate]

This question already has answers here:
How can I prevent java.lang.NumberFormatException: For input string: "N/A"?
(6 answers)
Closed 9 months ago.
My task is to read values from a csv file, and import each line of information from this file into an object array. I think my issue is the blank data elements in my csv file which doesn't work for my parsing from string to int, but I have found no way to deal with this. Here is my code:
`fileStream = new FileInputStream(pFileName);
rdr = new InputStreamReader(fileStream);
bufRdr = new BufferedReader(rdr);
lineNum = 0;`
while (line != null) {
lineNum++;
String[] Values = new String[13];
Values = line.split(",");
int cumulPos = Integer.parseInt(Values[6]);
int cumulDec = Integer.parseInt(Values[7]);
int cumuRec = Integer.parseInt(Values[8]);
int curPos = Integer.parseInt(Values[9]);
int hosp = Integer.parseInt(Values[10]);
int intenCar = Integer.parseInt(Values[11]);
double latitude = Double.parseDouble(Values[4]);
double longitude = Double.parseDouble(Values[5]);
covidrecordArray[lineNum] = new CovidRecord(Values[0], cumulPos, cumulDec, cumuRec, curPos, hosp,
intenCar, new Country(Values[1], Values[2], Values[3], Values[13], latitude, longitude));
If anyone could help it would be greatly appreciated.

As already suggested, use a proper CSV Parser if you can but if for some unknown reason you can't, this could be one way you can do it. Be sure to read the comments in code:
fileStream = new FileInputStream(pFileName);
rdr = new InputStreamReader(fileStream);
bufRdr = new BufferedReader(rdr);
// Remove the following line if there is no Header line in the CSV file.
String line = bufRdr.readLine();
String csvFileDataDelimiter = ",";
List<CovidRecord> recordsList = new ArrayList<>();
// True value calculated later in code (read comments).
int expectedNumberOfElements = 0; // 0 is default
while ((line = bufRdr.readLine()) != null) {
line = line.trim();
// If for some crazy reason a blank line is encountered...skip it.
if (line.isEmpty()) {
continue;
}
/* Get the expected number of elements within each CSV File Data Line.
This is based off of the number of actual delimiters within a file
data line plus 1. This is only calculated from the very first data
line. */
if (expectedNumberOfElements == 0) {
expectedNumberOfElements = line.replaceAll("[^\\" + csvFileDataDelimiter + "]", "").length() + 1;
}
/* Create and fill (with Null String) an array to be the expected
size of a CSV data line. This is done because if a data line
contains nothing for the last data element on that line then
when the line is split, the srray that is created will be short
by one element. This will ensure that there will alsways be a
Null String ("") present within the array when there is nothing
in the CSV data line. This null string is used in data validations
so as to provide a default value (like 0) if an Array Element
contains an actual Null String (""). */
String[] csvLineElements = new String[expectedNumberOfElements];
Arrays.fill(csvLineElements, "");
/* Take the array from the split (values) and place the data into
the csvLineElements[] array. */
String[] values = line.split("\\s*,\\s*"); // Takes care of any comma/whitespace combinations (if any).
for (int i = 0; i < values.length; i++) {
csvLineElements[i] = values[i];
}
/* Is the csvLineElements[] element a String representation of a signed
or unsigned integer data type value ("-?\\d+"). If so, convert the
String array element into an Integer value. If not, provide a default
value of 0. */
int cumulPos = Integer.parseInt(csvLineElements[6].matches("-?\\d+") ? csvLineElements[6] : "0");
int cumulDec = Integer.parseInt(csvLineElements[7].matches("-?\\d+") ? csvLineElements[7] : "0");
int cumuRec = Integer.parseInt(csvLineElements[8].matches("-?\\d+") ? csvLineElements[8] : "0");
int curPos = Integer.parseInt(csvLineElements[9].matches("-?\\d+") ? csvLineElements[9] : "0");
int hosp = Integer.parseInt(csvLineElements[10].matches("-?\\d+") ? csvLineElements[10] : "0");
int intenCar = Integer.parseInt(csvLineElements[11].matches("-?\\d+") ? csvLineElements[11] : "0");
/* Is the csvLineElements[] element a String representation of a signed
or unsigned integer or floating point value ("-?\\d+(\\.\\d+)?").
If so, convert the String array element into an Double data type value.
If not, provide a default value of 0.0 */
double latitude = Double.parseDouble(csvLineElements[4]
.matches("-?\\d+(\\.\\d+)?") ? csvLineElements[4] : "0.0d");
double longitude = Double.parseDouble(csvLineElements[5]
.matches("-?\\d+(\\.\\d+)?") ? csvLineElements[5] : "0.0d");
/* Create an instance of Country to pass into the constructor of
CovidRecord below. */
Country country = new Country(csvLineElements[1], csvLineElements[2],
csvLineElements[3], csvLineElements[13],
latitude, longitude);
// Create an add an instance of CovidRecord to the recordsList List.
recordsList.add(new CovidRecord(csvLineElements[0], cumulPos, cumulDec,
cumuRec, curPos, hosp, intenCar, country));
// Do what you want with the recordList List....
}
For obvious reasons, the code above was not tested. If you have any problems with it then let me know.
You will also notice the instead of the covidrecordArray[] CovidRecord Array I opted to use a List Interface named recordsList. This List can grow dynamically whereas the array is fixed meaning you need to determine the number of data lines within the file when initializing the array. This is not required with the List.

you can create one generic method for null check and check if it's null then return empty string or any thing else based on your needs
int hosp = Integer.parseInt(checkForNull(Values[10]));
public static String checkForNull(String val) {
return (val == null ? " " : val);
}

android- convert arraylist to one string in custom format

i have data in sqlit database so i use it to store categoreis and items
and i get it in arraylist like this:
public ArrayList showDataItems(String id_cate){
ArrayList<Items> arrayListItems = new ArrayList<>();
SQLiteDatabase db = this.getReadableDatabase();
Cursor cr = db.rawQuery("select * from items where id_cate = "+id_cate,null);
cr.moveToFirst();
while (cr.isAfterLast() == false){
String item_id = cr.getString(0);
String ItemName = cr.getString(1);
String Item_quantity = cr.getString(2);
String icon = cr.getString(3);
int isDone = Integer.parseInt(cr.getString(5));
arrayListItems.add(new Items(item_id,ItemName,Item_quantity,R.drawable.shopicon,icon,isDone));
cr.moveToNext();
}
return arrayListItems;
}
so i need to get this data and convert it to string and share it to other application like whatsapp in custom format for example :
1- first one *
2- second one *
3-....
so i use this code for send data
Intent intent = new Intent(Intent.ACTION_SEND);
intent.putExtra(Intent.EXTRA_TEXT,"hello world");
intent.setType("text/plain");
startActivity(Intent.createChooser(intent,"send items i need all
data here"));
so we can use string builder or some thing to get data in one string
please help me!

As you said, there is a StringBuilder class who can help formatting strings.
Here are the java docs
From StringBuilder, see the append(..) method. For your example:
int size = list.size();
for(int i = 0; i< size-1;i++){
stringBuilder.append(i+1).append("- ").append(list.get(i)).append('\n');
}
stringBuilder.append(size).append("- ").append(list.get(size-1));
The last call to append is different from the firstone by not appending the end line

Mapreduce java program to search QuadTree index and also run GeometryEngine.contains to confirm point in polygon using wkt file

This post is a map reduce implementation suggested for my previous question: "How to optimize scan of 1 huge file / table in Hive to confirm/check if lat long point is contained in a wkt geometry shape"
I am not well-versed in writing java programs for map-reduce and I mainly use Hive or Pig or spark to develop in Hadoop eco-system. To give a background of task at hand: I am trying to associate every latitude/longitude ping to corresponding ZIP postal code. I have a WKT multi-polygon shape file (500 MB) with all the zip information. I have loaded this in Hive and can do a join using ST_Contains(polygon, point). However, it takes very long to complete. To over come this bottle neck I am trying to leverage the example in ESRI ("https://github.com/Esri/gis-tools-for-hadoop/tree/master/samples/point-in-polygon-aggregation-mr") by building a quad tree index for searching a point derived from lat-long in polygon.
I have managed to write the code and it clogs up the Java heap memory of the cluster. Any suggestions on improving the code or looking at a different approach will be greatly appreciated:
Error message:
Error: Java heap space
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
My code:
public class MapperClass extends Mapper<LongWritable, Text, Text, IntWritable> {
// column indices for values in the text file
int longitudeIndex;
int latitudeIndex;
int wktZip;
int wktGeom;
int wktLineCount;
int wktStateID;
// in boundaries.wkt, the label for the polygon is "wkt"
//creating ArrayList to hold details of the file
ArrayList<ZipPolyClass> nodes = new ArrayList<ZipPolyClass>();
String labelAttribute;
EsriFeatureClass featureClass;
SpatialReference spatialReference;
QuadTree quadTree;
QuadTreeIterator quadTreeIter;
BufferedReader csvWkt;
// class to store all the values from wkt file and calculate geometryFromWKT
public class ZipPolyClass {
public String zipCode;
public String wktPoly;
public String stateID;
public int indexJkey;
public Geometry wktGeomObj;
public ZipPolyClass(int ijk, String z, String w, String s ){
zipCode = z;
wktPoly = w;
stateID = s;
indexJkey = ijk;
wktGeomObj = GeometryEngine.geometryFromWkt(wktPoly, 0, Geometry.Type.Unknown);
}
}
//building quadTree Index from WKT multiPolygon and creating an iterator
private void buildQuadTree(){
quadTree = new QuadTree(new Envelope2D(-180, -90, 180, 90), 8);
Envelope envelope = new Envelope();
int j=0;
while(j<nodes.size()){
nodes.get(j).wktGeomObj.queryEnvelope(envelope);
quadTree.insert(j, new Envelope2D(envelope.getXMin(), envelope.getYMin(), envelope.getXMax(), envelope.getYMax()));
}
quadTreeIter = quadTree.getIterator();
}
/**
* Query the quadtree for the feature containing the given point
*
* #param pt point as longitude, latitude
* #return index to feature in featureClass or -1 if not found
*/
private int queryQuadTree(Point pt)
{
// reset iterator to the quadrant envelope that contains the point passed
quadTreeIter.resetIterator(pt, 0);
int elmHandle = quadTreeIter.next();
while (elmHandle >= 0){
int featureIndex = quadTree.getElement(elmHandle);
// we know the point and this feature are in the same quadrant, but we need to make sure the feature
// actually contains the point
if (GeometryEngine.contains(nodes.get(featureIndex).wktGeomObj, pt, spatialReference)){
return featureIndex;
}
elmHandle = quadTreeIter.next();
}
// feature not found
return -1;
}
/**
* Sets up mapper with filter geometry provided as argument[0] to the jar
*/
#Override
public void setup(Context context)
{
Configuration config = context.getConfiguration();
spatialReference = SpatialReference.create(4326);
// first pull values from the configuration
String featuresPath = config.get("sample.features.input");
//get column reference from driver class
wktZip = config.getInt("sample.features.col.zip", 0);
wktGeom = config.getInt("sample.features.col.geometry", 18);
wktStateID = config.getInt("sample.features.col.stateID", 3);
latitudeIndex = config.getInt("samples.csvdata.columns.lat", 5);
longitudeIndex = config.getInt("samples.csvdata.columns.long", 6);
FSDataInputStream iStream = null;
try {
// load the text WKT file provided as argument 0
FileSystem hdfs = FileSystem.get(config);
iStream = hdfs.open(new Path(featuresPath));
BufferedReader br = new BufferedReader(new InputStreamReader(iStream));
String wktLine ;
int i=0;
while((wktLine = br.readLine()) != null){
String [] val = wktLine.split("\\|");
String qtZip = val[wktZip];
String poly = val[wktGeom];
String stID = val[wktStateID];
ZipPolyClass zpc = new ZipPolyClass(i, qtZip, poly, stID);
nodes.add(i,zpc);
i++; // increment in the loop before end
}
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
if (iStream != null)
{
try {
iStream.close();
} catch (IOException e) { }
}
}
// build a quadtree of our features for fast queries
if (!nodes.isEmpty()) {
buildQuadTree();
}
}
#Override
public void map(LongWritable key, Text val, Context context)
throws IOException, InterruptedException {
/*
* The TextInputFormat we set in the configuration, by default, splits a text file line by line.
* The key is the byte offset to the first character in the line. The value is the text of the line.
*/
String line = val.toString();
String [] values = line.split(",");
// get lat long from file and convert to float
float latitude = Float.parseFloat(values[latitudeIndex]);
float longitude = Float.parseFloat(values[longitudeIndex]);
// Create our Point directly from longitude and latitude
Point point = new Point(longitude, latitude);
int featureIndex = queryQuadTree(point);
// Each map only processes one record at a time, so we start out with our count
// as 1. Since we have a distinct record file we will not run reducer
IntWritable one = new IntWritable(1);
if (featureIndex >= 0){
String zipTxt =nodes.get(featureIndex).zipCode;
String stateIDTxt = nodes.get(featureIndex).stateID;
String latTxt = values[latitudeIndex];
String longTxt = values[longitudeIndex];
String pointTxt = point.toString();
String name;
name = zipTxt+"\t"+stateIDTxt+"\t"+latTxt+"\t"+longTxt+ "\t" +pointTxt;
context.write(new Text(name), one);
} else {
context.write(new Text("*Outside Feature Set"), one);
}
}
}

I was able to resolve the out of memory issue by modifying the arrayList < classObject > to just hold arrayList < geometry > type.
Creating a class object (around 50k) to hold each row of a text file, consumed all the java heap memory. After this change code ran fine even in a 1-node virtual sandbox. I was able to crunch around 40 million rows in around 6 minutes.

Why aren't my markers showing up on Google Maps? Is the array setup incorrect?

As the title says I have an array of markers that I've populated with the markers I want. However the map doesn't display the markers that have been added to the array. I'm unsure what the error is although I suspect it's to do with my arrays not being populated correctly.
Here is where I try to populate three arrays for each separate parameter (for context, the outputText string is updated and displays all the information correctly so I don't know why there would be a problem with the other three arrays:
try {
String jsonStr = getJS;
JSONObject jsonObj = new JSONObject(jsonStr);
JSONArray Ja = jsonObj.getJSONArray("Users");
int count = 0;
outputText = "\tNames\t" + "\tLongitude\t" + "\tLatitude\n";
while (count < Ja.length()) {
JSONObject ja = Ja.getJSONObject(count);
count++;
outputText += "\t" + ja.getString("name") + "\t\t" + ja.getString("lon") + "\t\t" + ja.getString("lat") + "\n";
String name = ja.getString("name").trim();
names.add(name);
String lat = ja.getString("lat").trim();
Double dLat = Double.parseDouble(lat);
latitudes.add(dLat);
String lon = ja.getString("lon").trim();
Double dLon = Double.parseDouble(lon);
longitudes.add(dLon);
}
I'm then using these three arrays in another activity to try and add an array of markers to Google maps:
#Override
public void onMapReady(GoogleMap googleMap) {
mMap = googleMap;
LatLng uni = new LatLng(51.2985, 1.070884);
m1 = mMap.addMarker(new MarkerOptions().position(uni).title("University senate building").icon(BitmapDescriptorFactory.defaultMarker(BitmapDescriptorFactory.HUE_ROSE)));
List<Marker> markers = new ArrayList<>();
for(int i = 0;i<lf.getNames().size();i++){
String myName = lf.getNames().get(i);
Marker marker = mMap.addMarker(new MarkerOptions().position(new LatLng(lf.getLatitudes().get(i),lf.getLongitudes().get(i))).title(myName).icon(BitmapDescriptorFactory.defaultMarker(BitmapDescriptorFactory.HUE_CYAN)));
markers.add(marker);
}
mMap.moveCamera(CameraUpdateFactory.newLatLng(uni));
mMap.animateCamera(CameraUpdateFactory.zoomTo(18));
}
The application runs without any errors or warnings but no markers are displayed other than the initial 'uni' one that I manually define.
I'm pretty stuck as I've been trying to figure out where I've gone wrong for a while and I can't seem to find it. Thanks in advance.

Passing empty string to db helper

I'm having some real problems passing a single string from my main activity to my db helper class. Here is my main activity method which I am returning a string 'answerSetId':
public String showNextRandomQuestion3() {
SQLDatabaseHelper db = new SQLDatabaseHelper(this);
//get the data from the database
List<List<String>> listList = db.getAllAnswersByQuestion1();
//Get the question/text/answer Strings
List<String> questionStrings = listList.get(0); //question Strings
List<String> answerSetIds = listList.get(4);
//Generate random index
Random r = new Random();
int rand = Math.abs((r.nextInt() % questionStrings.size()));
//get answer description for randomly selected question
String questionString = questionStrings.get(rand);
String answerSetId = answerSetIds.get(rand);
questionView.setText(questionString);
return answerSetId;
}
I specifically want to use this 'answerSetId' in one method in my db helper class (as part of a query). Here is my current code - but it is capturing an empty string '[]':
public List<List<String>> getAnswers(String answerSetId) {
List<String> array1 = new ArrayList<String>();
List<String> array2 = new ArrayList<String>();
System.out.println(answerSetId);
SQLiteDatabase db = this.getReadableDatabase();
String[] columns = new String[]{TDESCR, ADESCR};
String selection = ASID+"=?";
String[] selectionArgs = new String[]{String.valueOf(answerSetId)};
Cursor c = db.query(TABLE_ANSWERS, columns, selection, selectionArgs, null, null, null);
if (c.moveToFirst()) {
do {
String textdescr = c.getString(c.getColumnIndex(TDESCR));
String answersdescr = c.getString(c.getColumnIndex(ADESCR));
array1.add(textdescr);
array2.add(answersdescr);
} while (c.moveToNext());
}
List< List<String> > listArray2 = new ArrayList< List<String> >();
listArray2.add(array1);
listArray2.add(array2);
return listArray2;
}
You can see that I want to use this method to retrun some more array data based on the query using the 'answerSetId' string - you'll also see where I have added a system print out which gives the empty value '[]'. But am only bothered about actually capturing the 'answerSetId' value from my main activity at this point.
How best to achieve this?
the arrays of data I will then pass onto my 'showNextAnswer()' method:
public String showNextAnswers() {
SQLDatabaseHelper db = new SQLDatabaseHelper(this);
//get the data from the database
List<List<String>> listList = db.getAnswers(answerSetId);
//Get the question/text/answer Strings
List<String> textDescrs = listList.get(0); //question Strings
// List<String> answerDescrs = listList.get(1); //text strings
// System.out.println(answerSetId);
answerView3.setText(textDescrs.toString());
String textDescriptions = textDescrs.toString();
// String regex = "\\[|\\]";
// textDescriptions = textDescriptions.replaceAll(regex, "");
//Separate out the question/answer of text description associated with the randomly selected question
StringTokenizer textDescr = new StringTokenizer(textDescriptions, ",");
String first = textDescr.nextToken();
// String second = textDescr.nextToken();
// String third = textDescr.nextToken();
// String fourth = textDescr.nextToken();
subQuestionView1.setText(first);
// subQuestionView2.setText(second);
// subQuestionView3.setText(third);
// answerView3.setText(fourth);
// System.out.println(textDescr);
return null;
}
FYI - I call 'showNextRandomQuestion3()' from an onClick method when user clicks to go to a next question - a new random question and the related answers to that new question will appear each click:
NextQuestionButton.setOnClickListener(new OnClickListener(){;
#Override
public void onClick(View v) {
showNextRandomQuestion3();
showNextAnswers();
countQuestions();
}});

It looks like the solution is very simple. You just capture the return value of showNextRandomQuestion3(), since it returns answerSetId.
You will need to add a String parameter to the showNextAnswers() method to take the String that was returned from showNextRandomQuestion3().
So in your click event, you would do this:
NextQuestionButton.setOnClickListener(new OnClickListener(){;
#Override
public void onClick(View v) {
String answerSetId = showNextRandomQuestion3(); //get return value
showNextAnswers(answerSetId); //give the ID here
countQuestions();
}});
Then, just add the parameter to showNextAnswers(), and you're done!
public String showNextAnswers(String answerSetId) {
SQLDatabaseHelper db = new SQLDatabaseHelper(this);
//get the data from the database
List<List<String>> listList = db.getAnswers(answerSetId); //This will work now!
//.................