We Keep Coding

Filtering in Android - issue that the code works but it doesn't return to the screen as expected

I've created an app in Android to retrieve restaurants in the local area and return them in list view. I've created individual arrays for each of the details in question (name / address / postcode & hygiene rating). Some of the venues are exempt and the array will return the information as -1, so I want to alter this using a filter, which I think I have done.
This is my Async Task which returns the JSON array.
// lets things run in the background, JSON Array to retrieve the list
class RetrieveNearestRestaurantsTask extends AsyncTask<Void, Void, JSONArray> {
private Exception exception;
// performs the search in the background / populates the arraylist
#RequiresApi(api = Build.VERSION_CODES.N)
#Override
protected JSONArray doInBackground(Void... voids) {
// if the searchUrl conversion is empty
if (!searchUrl.toString().isEmpty()) {
// do this instead
try {
URL url = new URL(searchUrl.toString());
URLConnection tc = url.openConnection();
InputStreamReader isr = new InputStreamReader(tc.getInputStream());
BufferedReader in = new BufferedReader(isr);
String line; // variable
// while line in (read in) is not equal to null
// create a new object and output as line in JSON
while ((line = in.readLine()) != null) {
ja = new JSONArray(line);
// run through the length of the array
for (int i = 0; i < ja.length(); i++) {
JSONObject JO = (JSONObject) ja.get(i);
// output to meet Basic Functionality requirement
businessNames.add(JO.getString("BusinessName"));
postCodes.add(JO.getString("PostCode"));
addressList1.add(JO.getString("AddressLine1"));
addressList2.add(JO.getString("AddressLine2"));
addressList3.add(JO.getString("AddressLine3"));
// if the rating of the restaurant is -1, exempt
// should be displayed
ratingValue.add(JO.getString("RatingValue"));
ratingValue.stream()
.filter(x -> x.equals("-1"))
.findFirst()
.ifPresent(v -> System.out.println("Exempt"));
calcDistance.add(JO.getString("DistanceKM"));
// output everything together
ConcatenateSearchResults();
}
}
isr.close();
in.close();
//return ja;
} catch (Exception e) {
this.exception = e;
return ja;
} finally {
//is.close();
}
}
return ja;
}
Loads the search results.
protected void onPostExecute(JSONArray jsonArray) {
LoadSearchResults();
}
}
Concatenates the search results
private ArrayList<String> ConcatenateSearchResults()
{
int length = businessNames.size();
ArrayList<String> concatenatedResults = new ArrayList<>();
if(!businessNames.isEmpty() && !calcDistance.isEmpty() && !ratingValue.isEmpty()
&& !addressList1.isEmpty() && !addressList2.isEmpty() && !addressList3.isEmpty()
&& !postCodes.isEmpty())
{
for(int i=0; i < length; i++)
{
concatenatedResults.add("\n"+businessNames.get(i) +
"\nDistance (in Km) :"+ calcDistance.get(i) +
"\n\nRating: "+ ratingValue.get(i) +
"\nAddress Line 1: "+ addressList1.get(i) +
"\nAddress Line 2: "+ addressList2.get(i)+
"\nAddress Line 3: "+ addressList3.get(i) +
"\nPostcode: "+ postCodes.get(i));
}
}
return concatenatedResults;
}
}
However, I think somewhere I haven't enabled the proper ratingValue variable (?) to return the correct information (exempt instead of -1, where applicable, but I haven't worked out what I might have done wrong. Thank you for any help, I am slooowly getting better at explaining what I don't know.

Speed up reading CSV in Java

I have a relatively inefficent CSVReader code, see below. It takes more than 30 seconds to read 30000+ lines. How to speed up this reading process as fast as possible?
public class DataReader {
private String csvFile;
private List<String> sub = new ArrayList<String>();
private List<List> master = new ArrayList<List>();
public void ReadFromCSV(String csvFile) {
String line = "";
String cvsSplitBy = ",";
try (BufferedReader br = new BufferedReader(new FileReader(csvFile))) {
System.out.println("Header " + br.readLine());
while ((line = br.readLine()) != null) {
// use comma as separator
String[] list = line.split(cvsSplitBy);
// System.out.println("the size is " + country[1]);
for (int i = 0; i < list.length; i++) {
sub.add(list[i]);
}
List<String> temp = (List<String>) ((ArrayList<String>) sub).clone();
// master.add(new ArrayList<String>(sub));
master.add(temp);
sub.removeAll(sub);
}
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(master);
}
public List<List> getMaster() {
return master;
}
}
UPDATE: I have found that my code actually can finish the reading work in less than 1 second if run it separately. As this DataReader is a part used by my simulation model to initialize the relevant properties. And the following part is associated with the use of the data imported, WHICH TAKES 40 SECONDS TO FINISH! Anyone could help by looking at the generic part of the codes?
// add route network
Network<Object> net = (Network<Object>)context.getProjection("IntraCity Network");
IndexedIterable<Object> local_hubs = context.getObjects(LocalHub.class);
for (int i = 0; i <= CSV_reader_route.getMaster().size() - 1; i++) {
String source = (String) CSV_reader_route.getMaster().get(i).get(0);
String target = (String) CSV_reader_route.getMaster().get(i).get(3);
double dist = Double.parseDouble((String) CSV_reader_route.getMaster().get(i).get(6));
double time = Double.parseDouble((String) CSV_reader_route.getMaster().get(i).get(7));
Object source_hub = null;
Object target_hub = null;
Query<Object> source_query = new PropertyEquals<Object>(context, "hub_code", source);
for (Object o : source_query.query()) {
if (o instanceof LocalHub) {
source_hub = (LocalHub) o;
}
if (o instanceof GatewayHub) {
source_hub = (GatewayHub) o;
}
}
Query<Object> target_query = new PropertyEquals<Object>(context, "hub_code", target);
for (Object o : target_query.query()) {
if (o instanceof LocalHub) {
target_hub = (LocalHub) o;
}
if (o instanceof GatewayHub) {
target_hub = (GatewayHub) o;
}
}
// System.out.println(target_hub.getClass() + " " + time);
// Route this_route = (Route) net.addEdge(source_hub, target_hub);
// context.add(this_route);
// System.out.println(net.getEdge(source_hub, target_hub));
if (net.getEdge(source, target) == null) {
Route this_route = (Route) net.addEdge(source, target);
context.add(this_route);
// this_route.setDist(dist);
// this_route.setTime(time); }
}
}

In your code you are doing many write operation to just add the list of values from current row in your master list which is not required. You can replace the existing code with simple one as given below.
Existing code:
String[] list = line.split(cvsSplitBy);
// System.out.println("the size is " + country[1]);
for (int i = 0; i < list.length; i++) {
sub.add(list[i]);
}
List<String> temp = (List<String>) ((ArrayList<String>) sub).clone();
// master.add(new ArrayList<String>(sub));
master.add(temp);
sub.removeAll(sub);
Suggested code:
master.add(Arrays.asList(line.split(cvsSplitBy)));

I don't have a CSV that big, but you could try the following:
public static void main(String[] args) throws IOException {
Path csvPath = Paths.get("path/to/file.csv");
List<List<String>> master = Files.lines(csvPath)
.skip(1)
.map(line -> Arrays.asList(line.split(",")))
.collect(Collectors.toList());
}
EDIT: I tried it with a CSV sample with 50k entries and the code runs in less than one second.

With extends to the answer of #Alex R, you can process it in parallel as well like this:
public static void main(String[] args) throws IOException {
Path csvPath = Paths.get("path/to/file.csv");
List<List<String>> master = Files.lines(csvPath)
.skip(1).parallel()
.map(line -> Arrays.asList(line.split(",")))
.collect(Collectors.toList());
}

read json with timestamps as jsonobject keys

I am trying to log weather data for a university hydrological project using java.
The data is formated as a json file in 5 minute intervals for the last 24 hours in the following way (example):
{
"1482439800":{
"hu":92,
"te":-2.9,
"dp":-4.5,
"pr":1028.4,
"pa":null,
"ws":1.4,
"wd":180
},
"1482440100":{
"hu":92,
"te":-2.9,
"dp":-4.5,
"pr":1028.4,
"pa":null,
"ws":1.4,
"wd":180
}
}
I have already tried to use the following code to access data from the json file:
private static String readUrl(String urlString) throws Exception {
BufferedReader reader = null;
try {
URL url = new URL(urlString);
reader = new BufferedReader(new InputStreamReader(url.openStream()));
StringBuffer buffer = new StringBuffer();
int read;
char[] chars = new char[1024];
while ((read = reader.read(chars)) != -1)
buffer.append(chars, 0, read);
return buffer.toString();
} finally {
if (reader != null)
reader.close();
}
}
public static Object[][] getstation1(){
Object[][] data = null;
try {
JSONObject json = new JSONObject(readUrl("http://netzwerk.wetter.com/api/stationdata/14091/24/"));
Iterator keys = json.keys();
while (keys.hasNext()) {
Object key = keys.next();
JSONObject value = json.getJSONObject((String) key);
double hu = value.getDouble("hu");
System.out.println(hu);
}
} catch (JSONException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return data;
}
This was somewhat successful as it returned data for humidity (hu) but in a seemingly random order.
Now for my question: How do I read the times and return them alongside the corresponding weather data from newest to oldest inside the Object[][]?
Any help is appreciated.
Thank you.

A sorted map would be more appropriate than an Object[][].
Quickly:
TreeMap<String, Object> sorted = new TreeMap<>(json.toMap());
But that will sort alphanumerically (probably fine in your case as the timestamps are all the same length).
You could do a little more work to sort the results into a typed map:
TreeMap<Date, Map<String, Double>> byDate = json.toMap().entrySet().stream()
.collect(Collectors.toMap(
e -> new Date(Long.valueOf(e.getKey()) * 1000),
e -> (Map) e.getValue(),
(a, b) -> {throw new IllegalArgumentException("Duplicate key " + a);},
TreeMap::new
));
If you really need an Object[][] you can remap your data once it's sorted:
Object[][] data = sorted.entrySet().stream().map(e -> new Object[] {e.getKey(), e.getValue()}).toArray(Object[][]::new);
Or consider using an object mapper like jackson or gson.

Thanks for the answer but in the end I decided to go a slightly simpler route.
I retrieved all the key names, sorted them and then read the corresponding data key by key. And as I was getting frequent errors because of data being null I added protection for that as well (I need them as actual numbers).
public static Object[][] getstation1(){
Object[][] data = null;
try {
JSONObject json = new JSONObject(readUrl("http://netzwerk.wetter.com/api/stationdata/14091/2/"));
System.out.println("Fetching "+"http://netzwerk.wetter.com/api/stationdata/14091/2/");
String[] times = json.getNames(json);
Arrays.sort(times);
data = new Object[times.length][8];
for (int i = 0; i < times.length; i++){
Date temp = new Date((long)Integer.parseInt(times[i])*1000);
data[i][0] = temp;
if (json.getJSONObject(times[i]).isNull("hu")){
data[i][1] = 0;
} else {
data[i][1] = json.getJSONObject(times[i]).getDouble("hu");
}
if (json.getJSONObject(times[i]).isNull("te")){
data[i][2] = 0;
} else {
data[i][2] = json.getJSONObject(times[i]).getDouble("te");
}
if (json.getJSONObject(times[i]).isNull("dp")){
data[i][3] = 0;
} else {
data[i][3] = json.getJSONObject(times[i]).getDouble("dp");
}
if (json.getJSONObject(times[i]).isNull("pr")){
data[i][4] = 0;
} else {
data[i][4] = json.getJSONObject(times[i]).getDouble("pr");
}
if (json.getJSONObject(times[i]).isNull("pa")){
data[i][5] = 0;
} else {
data[i][5] = json.getJSONObject(times[i]).getDouble("pa");
}
if (json.getJSONObject(times[i]).isNull("ws")){
data[i][6] = 0;
} else {
data[i][6] = json.getJSONObject(times[i]).getDouble("ws");
}
if (json.getJSONObject(times[i]).isNull("wd")){
data[i][7] = 0;
} else {
data[i][7] = json.getJSONObject(times[i]).getDouble("wd");
}
}
} catch (JSONException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
return data;
}

List renderer issue in codenameone

I used list renderer and all works fine. Then I kept a extract last clicked item as follows: There are 3 buttons in the renderer. button1 works in all, however button2 and button3 works in some and doesn't work in other? why is it?
If I remove if (lastClickedButton != null) from the code below, it gives NullPointerException for the same button which works in other list items before.
MyCode:
#Override
protected boolean initListModelListEmergencyList(List cmp) {
cmp.setModel(new com.codename1.ui.list.DefaultListModel(emergencyPoliceArray));
cmp.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent evt) {
for (int i = 0; i < emergencyPoliceArray.size(); i++) {
if (cmp.getSelectedItem() == emergencyPoliceArray.get(i)) {
lastClickedButton = ((GenericListCellRenderer) cmp.getRenderer()).extractLastClickedComponent();
if (lastClickedButton != null) {
System.out.println("Phn no: " + lastClickedButton.getText());
}
}
}
}
}
);
return true;
}
MyConnection:
private void showEmergencyDetails() {
JSONParser p = new JSONParser();
Hashtable<String, Object> test;
try {
test = p.parse(new InputStreamReader(is));
Vector emergencyVector = (Vector) test.get("root");
emergencyPoliceArray = new ArrayList<Map<String, String>>();
for (int j = 0; j < emergencyVector.size(); j++) {
Hashtable hm = (Hashtable) emergencyVector.get(j);
String title = (String) hm.get("title");
String location = (String) hm.get("location");
Vector phn = (Vector) hm.get("phone");
for (int i = 0; i < phn.size(); i++) {
Hashtable hphn = (Hashtable) phn.get(i);
String phone1 = (String) hphn.get("phn1");
String phone2 = (String) hphn.get("phn2");
String phone3 = (String) hphn.get("phn3");
HashMap<String, String> mp = new HashMap<String, String>();
mp.put("Phn", phone1);
mp.put("Phone2", phone2);
mp.put("Phone3", phone3);
mp.put("NameHeading", "Name");
mp.put("PhnHeading", "Phn no.");
mp.put("LocationHeading", "Location");
mp.put("Title", title);
mp.put("Name", title);
mp.put("Location", location);
emergencyPoliceArray.add(mp);
}
}
} catch (IOException ex) {
}
showForm("EmergencyListDetails", null);
}

If there is variance between the renderer entries (e.g. one entry has the button while another doesn't) this might impact all the list and not just the entry where the button is visible.
When you have interaction we strongly recommend avoiding list. Its hackish at best for such use cases and doesn't provide any performance advantages over container for such common cases.

Java - Determine if a ZipInputStream contains a Entry

I'm currently trying to create a method that merges several ZipFiles into one big. Therefore I created a method that takes a output file and a list of InputStreams.
These InputStreams are later transformed into ZipInputStreams. That works fine!
But I have trouble when a file has already been added to the archive. At this point I need to override the entry already added (InputStreams with a higher index (lower in the list) should override the files from streams with a lower index). I also know how to do that: I just do not add the entry if a archive that would need to override it.
But the problem is how could I check if a entry is contained in a ZipInputStream so I can skip the addition of the entry for the current stream?
My code so far:
public static void makeNewZipFromInputStreamList(File outputFile,
ArrayList<InputStream> inputStreamList,
ArrayList<String> includeList, ArrayList<String> excludeList)
throws IOException, IllegalArgumentException {
final int sizeOfLists[] = new int[] { inputStreamList.size(),
includeList.size(), excludeList.size() };
if ((sizeOfLists[0] != sizeOfLists[1])
|| (sizeOfLists[0] != sizeOfLists[2])
|| (sizeOfLists[1] != sizeOfLists[2]))
throw new IllegalArgumentException(
"The ArrayLists do not have the same size ("
+ sizeOfLists[0] + ", " + sizeOfLists[1] + ", "
+ sizeOfLists[2] + ")");
final ZipOutputStream zipOutputFile = new ZipOutputStream(
new FileOutputStream(outputFile));
final int size = sizeOfLists[0];
InputStream inputStreamTempArray[] = inputStreamList
.toArray(new InputStream[size]);
String includeArray[] = includeList.toArray(new String[size]);
String excludeArray[] = excludeList.toArray(new String[size]);
int i, j;
ZipInputStream stream, streamTmp;
ZipInputStream inputStreamArray[] = new ZipInputStream[size];
String include, exclude, fileName;
ZipEntry entry;
for (i = 0; i < size; i++) {
inputStreamArray[i] = new ZipInputStream(inputStreamTempArray[i]);
if (includeArray[i] == null) {
includeArray[i] = "";
}
if (excludeArray[i] == null) {
excludeArray[i] = "";
}
}
for (i = 0; i < size; i++) {
while ((entry = inputStreamArray[i].getNextEntry()) != null) {
fileName = entry.getName();
for (j = i + 1; j < size; j++) {
// Check if the entry exists in the following archives (Then skip this entry)
}
if (fileName.matches(includeArray[i]) || !fileName.matches(excludeArray[i])) {
zipOutputFile.putNextEntry(entry);
if (!entry.isDirectory()) {
copyStream(inputStreamArray[i], zipOutputFile, false, false);
}
}
}
inputStreamArray[i].close();
}
zipOutputFile.close();
}
copyStream:
private static boolean copyStream(final InputStream is,
final OutputStream os, boolean closeInputStream,
boolean closeOutputStream) {
try {
final byte[] buf = new byte[1024];
int len = 0;
while ((len = is.read(buf)) > 0) {
os.write(buf, 0, len);
}
if (closeInputStream) {
is.close();
}
if (closeOutputStream) {
os.close();
}
return true;
} catch (final IOException e) {
e.printStackTrace();
}
return false;
}
EDIT:
I had the idea to just append the entries the other way round meaning starting from the end of the list and if a entry is already put it is just going to skip.
When I'm doing this I get a really weird error:
java.util.zip.ZipException: invalid entry compressed size (expected 1506 but got 1507 bytes)
at java.util.zip.ZipOutputStream.closeEntry(Unknown Source)
at java.util.zip.ZipOutputStream.putNextEntry(Unknown Source)
at io.brainstone.github.installer.FileUtils.makeNewZipFromInputStreamList(FileUtils.java:304)
at io.brainstone.github.installer.Main.startInstalling(Main.java:224)
at io.brainstone.github.installer.Window$3$1.run(Window.java:183)
This is my current code:
public static void makeNewZipFromInputStreamList(File outputFile,
ArrayList<InputStream> inputStreamList,
ArrayList<String> includeList, ArrayList<String> excludeList)
throws IOException, IllegalArgumentException {
final int sizeOfLists[] = new int[] { inputStreamList.size(),
includeList.size(), excludeList.size() };
if ((sizeOfLists[0] != sizeOfLists[1])
|| (sizeOfLists[0] != sizeOfLists[2])
|| (sizeOfLists[1] != sizeOfLists[2]))
throw new IllegalArgumentException(
"The ArrayLists do not have the same size ("
+ sizeOfLists[0] + ", " + sizeOfLists[1] + ", "
+ sizeOfLists[2] + ")");
final ZipOutputStream zipOutputFile = new ZipOutputStream(
new FileOutputStream(outputFile));
final int size = sizeOfLists[0];
final InputStream inputStreamTempArray[] = inputStreamList
.toArray(new InputStream[size]);
final String includeArray[] = includeList.toArray(new String[size]);
final String excludeArray[] = excludeList.toArray(new String[size]);
final ZipInputStream inputStreamArray[] = new ZipInputStream[size];
HashMap<String, Object[]> tmp;
int i, j;
String fileName;
ZipEntry entry;
for (i = size - 1; i >= 0; i--) {
System.out.println(i);
inputStreamArray[i] = new ZipInputStream(inputStreamTempArray[i]);
if (includeArray[i] == null) {
includeArray[i] = "";
}
if (excludeArray[i] == null) {
excludeArray[i] = "";
}
while ((entry = inputStreamArray[i].getNextEntry()) != null) {
fileName = entry.getName();
if (fileName.matches(includeArray[i])
|| !fileName.matches(excludeArray[i])) {
// Here is where I would check if a entry is already put.
// Probably just by catching the exception thrown in this
// case
zipOutputFile.putNextEntry(entry);
if (!entry.isDirectory()) {
copyStream(inputStreamArray[i], zipOutputFile, false,
false);
}
}
}
inputStreamArray[i].close();
}
zipOutputFile.close();
}

Hold a map from fileName to entry.
Iterate over all entries in all input streams and put the entries in the map, mapped by file name. Last entry will always override previous. When you finish you have only all highest-indexed entries per file name.
Iterate over the map's entries and put them to zipOutputFile.
// (1) here all entries will be stored, overriding low-indexed with high-indexed
final Map<String, ZipEntry> fileNameToZipEntry = new HashMap<String, ZipEntry>();
// (2) Iterate over all entries and store in map, overriding low-indexed
for (i = 0; i < size; i++) {
while ((entry = inputStreamArray[i].getNextEntry()) != null) {
fileName = entry.getName();
fileNameToZipEntry.put(fileName, entry);
}
inputStreamArray[i].close();
}
// (3) Iterating the map that holds only the entries required for zipOutputFile
int j = 0;
for ( Set<Map.Entry<String, ZipEntry>> mapEntry : fileNameToZipEntry.entrySet() ) {
if (fileName.matches(includeArray[j]) || !fileName.matches(excludeArray[j])) {
zipOutputFile.putNextEntry(entry);
if (!entry.isDirectory()) {
copyStream(inputStreamArray[j], zipOutputFile, false, false);
}
}
j++;
}

The simplest way to solve this is iterating backwards through the ArrayLists.
public static void makeNewZipFromInputStreamList(File outputFile,
ArrayList<InputStream> inputStreamList,
ArrayList<String> includeList, ArrayList<String> excludeList)
throws IOException, IllegalArgumentException {
final int sizeOfLists[] = new int[] { inputStreamList.size(),
includeList.size(), excludeList.size() };
if ((sizeOfLists[0] != sizeOfLists[1])
|| (sizeOfLists[0] != sizeOfLists[2])
|| (sizeOfLists[1] != sizeOfLists[2]))
throw new IllegalArgumentException(
"The ArrayLists do not have the same size ("
+ sizeOfLists[0] + ", " + sizeOfLists[1] + ", "
+ sizeOfLists[2] + ")");
final ZipOutputStream zipOutputFile = new ZipOutputStream(
new FileOutputStream(outputFile));
final int size = sizeOfLists[0];
final InputStream inputStreamTempArray[] = inputStreamList
.toArray(new InputStream[size]);
final String includeArray[] = includeList.toArray(new String[size]);
final String excludeArray[] = excludeList.toArray(new String[size]);
final ZipInputStream inputStreamArray[] = new ZipInputStream[size];
HashMap<String, Object[]> tmp;
int i, j;
String fileName;
ZipEntry entry;
for (i = size - 1; i >= 0; i--) {
inputStreamArray[i] = new ZipInputStream(inputStreamTempArray[i]);
if (includeArray[i] == null) {
includeArray[i] = "";
}
if (excludeArray[i] == null) {
excludeArray[i] = "";
}
while ((entry = inputStreamArray[i].getNextEntry()) != null) {
fileName = entry.getName();
if (fileName.matches(includeArray[i])
|| !fileName.matches(excludeArray[i])) {
try {
zipOutputFile.putNextEntry(entry);
if (!entry.isDirectory()) {
copyStream(inputStreamArray[i], zipOutputFile,
false, false);
}
} catch (ZipException ex) {
if (!ex.getMessage()
.matches("duplicate entry: .*\\..*")) {
throw new RuntimeException(
"Unexpected " + ex.getClass() + " (\""
+ ex.getMessage()
+ "\")\n(only duplicate entry execptions are expected!)",
ex);
}
}
}
}
inputStreamArray[i].close();
}
zipOutputFile.close();
}
But thank you anyways!