Aggregating objects by timestamp per day for a timespan of week - java

I have problem with aggregating data based on their timestamp per day for a timespan of one week. There is a SQLite database, which has a table which I save the number of walking steps in (timestamp column is UTC and created_At is local time, but I don't use the created_at column anyway).
What I want to do is get the total data which happened in 7 days ago until the midnight of a day before. So I have this jodatime expression to find start and end for timestamps
long start = new DateTime().withMillisOfDay(0).minusDays(7).getMillis();
long end = new DateTime().withTimeAtStartOfDay().getMillis();
//start milli:1405029600000 DateTime: 2014-07-11 00:00:00
//end milli:1405634400000 DateTime: 2014-07-18 00:00:00
Then I execute this sql command:
SELECT * FROM pa_data WHERE timestamp BETWEEN 1405029600000 AND 1405634400000
And I am pretty sure that it returns the correct rows ( I have compared the android database result with SQLite Database Browser on my pc, both return same number of rows). For this, I tried to use this nested iteration:
the Object I am trying to create is:
public class PhysicalActivityPerDay {
private List<PhysicalActivity> mList;
public PhysicalActivityPerDay(List<PhysicalActivity> list) {
mList = new ArrayList<PhysicalActivity>(list);
}
//methods....
}
Now the problem is, I want to have a data object that can hold the rows for each day.
List<PhysicalActivity> all = getPhysicalActivitiesBetween(start, end);
List<PhysicalActivityPerDay> perDays = new ArrayList<PhysicalActivityPerDay>();
List<PhysicalActivity> tempList;
PhysicalActivityPerDay tempPerDay;
for (int i = 0; i < 7; i++) {
long begin = start;
long stop = (begin + 86400000); //add 24 hours
tempList = new ArrayList<PhysicalActivity>();
for (int j = 0; j < all.size(); j++) {
PhysicalActivity p = all.get(j);
DateTime when = new DateTime(p.getTimestamp());
if (when.isAfter(start) && when.isBefore(stop)) {
tempList.add(p);
all.remove(j); //remove the matching object from the list
}
}
tempPerDay = new PhysicalActivityPerDay(tempList);
perDays.add(tempPerDay);
start += 86400000; //add 24 hours or 1 day for next iteration
}
return perDays;
But the result is totally unexpected. There are many rows which don't match the if statements above. I did a debug and here is what happens:
Log.w(TAG, "There are totally " + all.size() + " physical activities for day for 7 days");
//There are totally 6559 physical activities for day for 7 days
But, when I check the all list (total rows returned by DB) although I am removing matched objects from it, if I query its size after the nested iteration, it surprisingly still contains many objects in it, telling me that the iteration was not successful!
//Remaining: 3278 records after iterations from 6559
What I am doing wrong? please help me findout!

Not sure if that's the only problem :
You are looping over the all List, and removing items.
When you call all.remove(j), the item that used to be at position j+1 moves to poisition j. Which means your for loop would skip that item.
One way to solve this is to increment j only if you don't remove an item from the list.
for (int j = 0; j < all.size();) {
PhysicalActivity p = all.get(j);
DateTime when = new DateTime(p.getTimestamp());
if (when.isAfter(start) && when.isBefore(stop)) {
tempList.add(p);
all.remove(j); //remove the matching object from the list
} else {
j++;
}
}
Actually, I'm not entirely sure if the loop would work after this fix. It depends whether all.size() is evaluated in each iteration. If it isn't, it would expect the list to have the initial number of elements, even though you are removing items. In that case you can expect to get an exception the first time you try to access an index beyond the last index of the array.
If you get an exception, you can replace the loop with a while loop :
Iterator<PhysicalActivity> iter = all.iterator();
while (iter.hasNext ()) {
PhysicalActivity p = iter.next();
...
if (...) {
iter.remove();
}
}
Refer to the definition of List.remove() :
public E remove(int index)
Removes the element at the specified position in this list. Shifts any subsequent elements to the left (subtracts one from their indices).

How about letting SQL perform your aggregation for you
SELECT strftime('%W-%Y',dt) as weekYear, count(1) as occurencePerWeek
FROM SOMETABLE c GROUP BY weekYear;
http://sqlfiddle.com/#!5/e63c3/1

Related

Removing every other element in an array list

for(int i = 0; i < points.size(); i = i+lines) {
points.remove(i);
}
The idea here is that a user can either remove every other space or every third space or every fourth space .. And so forth, of an array list by entering an int "line" that will skip the spaces. However, I realize the list gets smaller each time messing with the skip value. How do I account for this? I'm using the ArrayList library from java so don't have the option of just adding a method in the array list class. Any help would be greatly appreciated.
I've perfomed a benchmark of all the answers proposed to this question so far.
For an ArrayList with ~100K elements (each a string), the results are as follows:
removeUsingRemoveAll took 15018 milliseconds (sleepToken)
removeUsingIter took 216 milliseconds (Arvind Kumar Avinash)
removeFromEnd took 94 milliseconds (WJS)
Removing an element from an ArrayList is an Θ(n) operation, as it has to shift all remaining elements in the array to the left (i.e. it's slow!). WJS's suggestion of removing elements from the end of the list first, appears to be the fastest (inplace) method proposed so far.
However, for this problem, I'd highly suggest considering alternative data structures such as a LinkedList, which is designed to make removing (or adding) elements in the middle of the list fast. Another alternative, if you have sufficient memory, is to build up the results in a separate list rather than trying to modify the list inplace:
removeUsingIterLinked took 12 milliseconds
removeUsingSecondList took 3 milliseconds (sleepToken with WJS's comment)
Use an Iterator with a counter e.g. the following code will remove every other (i.e. every 2nd) element (starting with index, 0):
Iterator<Point> itr = points.iterator();
int i = 0;
while(itr.hasNext()) {
itr.next();
if(i % 2 == 0) {
itr.remove();
}
i++;
}
Here, I've used i as a counter.
Similarly, you can use the condition, i % 3 == 0 to remove every 3rd element (starting with index, 0).
Here is a different approach. Simply start from the end and remove in reverse. That way you won't mess up the index synchronization. To guarantee that removal starts with the second item from the front, ensure you start with the last odd index to begin with. That would be list.size()&~1 - 1. If size is 10, you will start with 9. If size is 11 you will start with 9
List<Integer> list = IntStream.rangeClosed(1,11)
.boxed().collect(Collectors.toList());
for(int i = (list.size()&~1)-1; i>=0; i-=2) {
list.remove(i);
}
System.out.println(list);
Prints
[1, 3, 5, 7, 9, 11]
You could add them to a new ArrayList and then remove all elements after iterating.
You could set count to remove every countth element.
import java.util.ArrayList;
public class Test {
static ArrayList<String> test = new ArrayList<String>();
public static void main(String[] args) {
test.add("a");
test.add("b");
test.add("c");
test.add("d");
test.add("e");
ArrayList<String> toRemove = new ArrayList<String>();
int count = 2;
for (int i = 0; i < test.size(); i++) {
if (i % count == 0) {
toRemove.add(test.get(i));
}
}
test.removeAll(toRemove);
System.out.print(test);
}
}

Loop through an ArrayList to find missing values and create a dummy object

I have a list which stores a object named GoldNetValue containing date and gold rate.There will be a difference of 10 minutes between the two records in the list and, in some cases no data will be available during the particular time interval.
Sample values as below
{GoldNetValue[2018-03-02 13:20 ,87], GoldNetValue[2018-03-02 13:30 ,86.4],GoldNetValue[2018-03-02 13:40 ,85.6]],GoldNetValue[2018-03-02 13:50 ,85.8]],GoldNetValue[2018-03-02 14:10 ,86.1]],GoldNetValue[2018-03-02 14:30 ,86.8]]
i need to loop through the list and create a new GoldNetValue object with missing date field and noDataAvailable flg enabled,then insert it back into the list. The difference is always 10 minutes.
int diffMins = 10;
Date tempDate = new Date();
for(int i= 0; i < goldNetList.size(); i++)
{
GoldNetValue goldValue = (GoldNetValue) goldNetList.get(i);
if(goldValue.getDate() != null && goldValue.getGoldRate() != null)
{
tempDate = goldValue.getDate();
}
if() // logic yet to be implemented
}
lets say from 13:30 to 13:50 pm , there is only one record available , i need to create an object with date as 13:40 and noDataFlag enabled and store it back to the list.
i am a newbie just started to learn coding.
How can i populate through the list and create objects with flag enabled with these type of value combinations?
Thank you for your time

Verifying the data from two arraylist and delete it

I have a master arraylist call toBeDeleted which stored timestamp and email. The following are the sample data inside the toBeDeleted arraylist
[1507075234, bunny#outlook.com]
I have one arraylist call logData1 which stored status,email,timestamps and ID. The following are the sample data inside the logData1 arraylist.
[16, bunny#outlook, 1507075234, 0OX9VQB-01-00P-02]
I hope to delete the data inside the logData1 arraylist by verifying the timestamp first with timestamps stated in toBeDeleted1 arraylist, if the timestamp matched, I will check the email for both arraylist. If both of them are matched, I would like to delete away all the data (status,email,timestamp,ID). But I cant make it work
this is my sample output from my source code
[16, bunny#outlook.com, 1507075234, 0OX9VQB-01-00P-02]
The data inside toBeDeleted1 is :[1507075234, bunny#outlook.com]
The time1 is :1507075234
The email1 is :bunny#outlook.com
The time is :1507075234
The emails is :bunny#outlook.com
The data is :bunny#outlook.com
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -3
at java.util.ArrayList.elementData(Unknown Source)
at java.util.ArrayList.get(Unknown Source)
at EmailReporting.main(EmailReporting.java:83)
This is my sample program
System.out.println(logData1);
System.out.println("The data inside toBeDeleted1 is :"+toBeDeleted1);
for(int v = 0;v<toBeDeleted1.size();v++) //look through the logdata1 for removing the record base on timestamp
{
String time1 = toBeDeleted1.get(v);
String email1 = toBeDeleted1.get(v+1);
System.out.println("The time1 is :"+time1);
System.out.println("The email1 is :"+email1);
for(int f = logData1.size();f>logData1.size()-1;f--)
{
// System.out.println(logData1.size());
// System.out.println("The data in logdata1 is "+logData1.get(f-2));
if(time1.equals(logData1.get(f-2)))
{
System.out.println("The time is :"+logData1.get(f-2));
System.out.println("The emails is :"+logData1.get(f-3));
if(email1.equals(logData1.get(f-3)))
{
System.out.println("The data is :"+logData1.get(f-3));
logData1.remove(f-1);
logData1.remove(f-2);
logData1.remove(f-3);
logData1.remove(f-4);
f-=4;
}
}
}
}
The error occurred after this line of code executed
System.out.println("The data is :"+logData1.get(f-3));
You can find elements in the list in order using Collections.indexOfSubList:
List<String> toFind = Arrays.asList(time1, email1);
int emailIndex = Collections.indexOfSubList(logData1, toFind);
A similar lastIndexOfSubList method also exists. That might be more appropriate for your use case.
You can then use this to remove the elements from toFind:
int emailIndex = Collections.lastIndexOfSubList(logData1, toFind);
if (emailIndex >= 1) {
logData1.subList(emailIndex-1, emailIndex+3).clear();
}
Just do this in a loop to keep going until all occurrences have been removed.
Note that just doing this in a loop naively will keep on searching over the tail of the list repeatedly. Instead, you can use subList to "chop" the end of the list, to avoid re-searching it:
List<String> view = logData1;
int emailIndex;
while ((emailIndex = Collections.lastIndexOfSubList(view, toFind)) >= 1) {
logData1.subList(emailIndex-1, emailIndex+3).clear();
view = logData1.subList(0, emailIndex-1);
}
Additionally, note that deleting from the middle of an ArrayList is inefficient, because the elements after the ones you delete have to be shifted down. This is why using subList(...).clear() is better, because it does all of those shifts at once. But if you are removing lots of 4-element batches, you can do better.
Instead of the subList(...).clear(), you can set the bits of elements to be deleted into a BitSet:
List<String> view = logData1;
BitSet bits = new BitSet(logData1.size());
int emailIndex;
while ((emailIndex = Collections.lastIndexOfSubList(view, toFind)) >= 1) {
bits.set(emailIndex-1, emailIndex+3);
view = logData1.subList(0, emailIndex-1);
}
And then shift all the elements down at once, discarding the elements you want to delete:
int dst = 0;
for (int src = 0; src < logData1.size(); ++src) {
if (!bits.get(src)) {
logData1.set(dst++, logData1.get(src));
}
}
And now truncate the list:
logData1.subList(dst, logData1.size());

Optimization of deletion nodes in 2-d array

I have double dimensional array of dimensions 720x90. Let's denote rows by R and C as columns.
R1 = {C1,...,C90}
....
R720 = {C1,...C90}
Now, I want to see if any of the data in any of the rows appears anywhere else in any other rows. For instance, lets say the data in row 470 and column 67 is a duplicate of row 672 and column 34. In that case, I want to remove both row 470 and row 672 from the data set and continue checking. After I have checked all the rows, I want to print just the index of the rows that have survived. I have written a brute-force method of this. However, when I run this code, it never returns and I am not able to diagnose why. Also, is there a more efficient way to do this?
//check all the subsets of the interleaved data
public static int checkSubsets(String[][] subsets){
List subset = new ArrayList();
for(int i = 0; i< 720; i++){
for(int j = 0; j < 90; j++)
subset.add(subsets[i][j]);
}
Object duplicate;
Iterator itr = subset.iterator();
while(itr.hasNext()){
duplicate = itr.next();
while(itr.hasNext()){
subset.remove(duplicate);
itr=subset.iterator(); //to avoid concurrent modification
itr.next();
}
}
return subset.size();
}
Clarifications: Lets say I am iterating through looking at each value in the matrix. I take the first value in R1 C1 (row 1 - column 1). I find that these values are found somewhere in the 12, 346,123, 356 row. Then I remove all those rows from the matrix. So now the matrix is 5 rows smaller. I stop checking row 1 now and move onto row 2. I continue checking, skipping over row 12, 346, 123, and 356. Hence, I am after a row that is unique (has 90 values that are all unique).
I am not sure what the code you wrote has to do with the requirement, I will give you the approach of the answer yet you have to try it yourself first.
it is clear that you need to iterate on each row to check for possible duplicates yet this will cause a performance failure , you can overcome this with a smiple use of HashMap, first store each entry in the map , the key will be the value of the node of the array, and the value should be the coordinates of this node.
When iterating over the array for each row you should find the y coordinates from the map which is common between all nodes of the row, so duplicate rows detected.
In order to avoid keep checking the already removed rows try to store all the rows to be deleted and remove them once you are done, you can use Set to store them to avoid duplicate.
Good luck with the implemenation.
The algorithm is almost there, but helpfull data-structures are missing.
To add a bit of spice I used Java 8 somewhat.
As you did one can collect the values to check for duplicates.
However one needs to remember the first row of that value, as only there it is still unknown whether a duplicate exists.
public static int checkSubsets(String[][] subsets) {
// The results.
final Set<Integer> duplicateRows = new HashSet<>();
// From the first occurrence of a duplicate value we do not know it yet,
// so need to remember.
final Map<String, Integer> firstRowOfValue = new HashMap<>();
for (int i = 0; i < subsets.length; ++i) {
for (int j = 0; j < subsets[i].length; ++j) {
final String value = subsets[i][j];
Integer oldRow = firstRowOfValue.putIfAbsent(value, i);
if (oldRow != null) { // Duplicates
duplicateRows.add(i);
duplicateRows.add(oldRow);
// oldRow might already be added if third duplicate or same row.
}
}
}
IntStream.rangeOf(0, subsets.length)
.filter(i -> !duplicateRows.contains(i))
.forEach(System.out::println);
return subsets.length - duplicateRows.size();
}
The IntStream part would be in java 7:
for (int i = 0; i < subsets.length; ++i) {
if (!duplicateRows.contains(i)) {
System.out.println(i);
}
}
With java 7 you can safely substitute here putIfAbsent with put.

Delete data from ArrayList with a For-loop

I got a weird problem.
I thought this would cost me few minutes, but I am struggling for few hours now...
Here is what I got:
for (int i = 0; i < size; i++){
if (data.get(i).getCaption().contains("_Hardi")){
data.remove(i);
}
}
The data is the ArrayList.
In the ArrayList I got some strings (total 14 or so), and 9 of them, got the name _Hardi in it.
And with the code above I want to remove them.
If I replace data.remove(i); with a System.out.println then it prints out something 9 times, what is good, because _Hardi is in the ArrayList 9 times.
But when I use data.remove(i); then it doesn't remove all 9, but only a few.
I did some tests and I also saw this:
When I rename the Strings to:
Hardi1
Hardi2
Hardi3
Hardi4
Hardi5
Hardi6
Then it removes only the on-even numbers (1, 3, 5 and so on).
He is skipping 1 all the time, but can't figure out why.
How to fix this? Or maybe another way to remove them?
The Problem here is you are iterating from 0 to size and inside the loop you are deleting items. Deleting the items will reduce the size of the list which will fail when you try to access the indexes which are greater than the effective size(the size after the deleted items).
There are two approaches to do this.
Delete using iterator if you do not want to deal with index.
for (Iterator<Object> it = data.iterator(); it.hasNext();) {
if (it.next().getCaption().contains("_Hardi")) {
it.remove();
}
}
Else, delete from the end.
for (int i = size-1; i >= 0; i--){
if (data.get(i).getCaption().contains("_Hardi")){
data.remove(i);
}
}
You shouldn't remove items from a List while you iterate over it. Instead, use Iterator.remove() like:
for (Iterator<Object> it = list.iterator(); it.hasNext();) {
if ( condition is true ) {
it.remove();
}
}
Every time you remove an item, you are changing the index of the one in front of it (so when you delete list[1], list[2] becomes list[1], hence the skip.
Here's a really easy way around it: (count down instead of up)
for(int i = list.size() - 1; i>=0; i--)
{
if(condition...)
list.remove(i);
}
Its because when you remove an element from a list, the list's elements move up. So if you remove first element ie at index 0 the element at index 1 will be shifted to index 0 but your loop counter will keep increasing in every iteration. so instead you of getting the updated 0th index element you get 1st index element. So just decrease the counter by one everytime you remove an element from your list.
You can use the below code to make it work fine :
for (int i = 0; i < data.size(); i++){
if (data.get(i).getCaption().contains("_Hardi")){
data.remove(i);
i--;
}
}
It makes perfect sense if you think it through. Say you have a list [A, B, C]. The first pass through the loop, i == 0. You see element A and then remove it, so the list is now [B, C], with element 0 being B. Now you increment i at the end of the loop, so you're looking at list[1] which is C.
One solution is to decrement i whenever you remove an item, so that it "canceles out" the subsequent increment. A better solution, as matt b points out above, is to use an Iterator<T> which has a built-in remove() function.
Speaking generally, it's a good idea, when facing a problem like this, to bring out a piece of paper and pretend you're the computer -- go through each step of the loop, writing down all of the variables as you go. That would have made the "skipping" clear.
I don't understand why this solution is the best for most of the people.
for (Iterator<Object> it = data.iterator(); it.hasNext();) {
if (it.next().getCaption().contains("_Hardi")) {
it.remove();
}
}
Third argument is empty, because have been moved to next line. Moreover it.next() not only increment loop's variable but also is using to get data. For me use for loop is misleading. Why you don't using while?
Iterator<Object> it = data.iterator();
while (it.hasNext()) {
Object obj = it.next();
if (obj.getCaption().contains("_Hardi")) {
it.remove();
}
}
Because your index isn't good anymore once you delete a value
Moreover you won't be able to go to size since if you remove one element, the size as changed.
You may use an iterator to achieve that.
for (Iterator<Object> it = data.iterator(); it.hasNext();) {
if ( it.getCaption().contains("_Hardi")) {
it.remove(); // performance is low O(n)
}
}
If your remove operation is required much on list. Its better you use LinkedList which gives better performance Big O(1) (roughly).
Where in ArrayList performance is O(n) (roughly) . So impact is very high on remove operation.
It is late but it might work for someone.
Iterator<YourObject> itr = yourList.iterator();
// remove the objects from list
while (itr.hasNext())
{
YourObject object = itr.next();
if (Your Statement) // id == 0
{
itr.remove();
}
}
In addition to the existing answers, you can use a regular while loop with a conditional increment:
int i = 0;
while (i < data.size()) {
if (data.get(i).getCaption().contains("_Hardi"))
data.remove(i);
else i++;
}
Note that data.size() must be called every time in the loop condition, otherwise you'll end up with an IndexOutOfBoundsException, since every item removed alters your list's original size.
This happens because by deleting the elements you modify the index of an ArrayList.
import java.util.ArrayList;
public class IteratorSample {
public static void main(String[] args) {
// TODO Auto-generated method stub
ArrayList<Integer> al = new ArrayList<Integer>();
al.add(1);
al.add(2);
al.add(3);
al.add(4);
System.out.println("before removal!!");
displayList(al);
for(int i = al.size()-1; i >= 0; i--){
if(al.get(i)==4){
al.remove(i);
}
}
System.out.println("after removal!!");
displayList(al);
}
private static void displayList(ArrayList<Integer> al) {
for(int a:al){
System.out.println(a);
}
}
}
output:
before removal!!
1
2
3
4
after removal!!
1
2
3
There is an easier way to solve this problem without creating a new iterator object. Here is the concept. Suppose your arrayList contains a list of names:
names = [James, Marshall, Susie, Audrey, Matt, Carl];
To remove everything from Susie forward, simply get the index of Susie and assign it to a new variable:
int location = names.indexOf(Susie);//index equals 2
Now that you have the index, tell java to count the number of times you want to remove values from the arrayList:
for (int i = 0; i < 3; i++) { //remove Susie through Carl
names.remove(names.get(location));//remove the value at index 2
}
Every time the loop value runs, the arrayList is reduced in length. Since you have set an index value and are counting the number of times to remove values, you're all set. Here is an example of output after each pass through:
[2]
names = [James, Marshall, Susie, Audrey, Matt, Carl];//first pass to get index and i = 0
[2]
names = [James, Marshall, Audrey, Matt, Carl];//after first pass arrayList decreased and Audrey is now at index 2 and i = 1
[2]
names = [James, Marshall, Matt, Carl];//Matt is now at index 2 and i = 2
[2]
names = [James, Marshall, Carl];//Carl is now at index 3 and i = 3
names = [James, Marshall,]; //for loop ends
Here is a snippet of what your final method may look like:
public void remove_user(String name) {
int location = names.indexOf(name); //assign the int value of name to location
if (names.remove(name)==true) {
for (int i = 0; i < 7; i++) {
names.remove(names.get(location));
}//end if
print(name + " is no longer in the Group.");
}//end method
This is a common problem while using Arraylists and it happens due to the fact that the length (size) of an Arraylist can change. While deleting, the size changes too; so after the first iteration, your code goes haywire. Best advice is either to use Iterator or to loop from the back, I'll recommend the backword loop though because I think it's less complex and it still works fine with numerous elements:
//Let's decrement!
for(int i = size-1; i >= 0; i--){
if (data.get(i).getCaption().contains("_Hardi")){
data.remove(i);
}
}
Still your old code, only looped differently!
I hope this helps...
Merry coding!!!

Categories