We Keep Coding

In Scala, how do I create a column of date arrays of monthly dates between a start and end date?

In Spark Scala, I am trying to create a column that contains an array of monthly dates between a start and an end date (inclusive).
For example, if we have 2018-02-07 and 2018-04-28, the array should contain [2018-02-01, 2018-03-01, 2018-04-01].
Besides the monthly version I would also like to create a quarterly version, i.e. [2018-1, 2018-2].
Example Input Data:
id startDate endDate
1_1 2018-02-07 2018-04-28
1_2 2018-05-06 2018-05-31
2_1 2017-04-13 2017-04-14
Expected (monthly) Output 1:
id startDate endDate dateRange
1_1 2018-02-07 2018-04-28 [2018-02-01, 2018-03-01, 2018-04-01]
1_1 2018-05-06 2018-05-31 [2018-05-01]
2_1 2017-04-13 2017-04-14 [2017-04-01]
Ultimate expected (monthly) output 2:
id Date
1_1 2018-02-01
1_1 2018-03-01
1_1 2018-04-01
1_2 2018-05-01
2_1 2017-04-01
I have spark 2.1.0.167, Scala 2.10.6, and JavaHotSpot 1.8.0_172.
I have tried to implement several answers to similar (day-level) questions on here, but I am struggling with getting a monthly/quarterly version to work.
The below creates an array from start and endDate and explodes it. However I need to explode a column that contains all the monthly (quarterly) dates in-between.
val df1 = df.select($"id", $"startDate", $"endDate").
// This just creates an array of start and end Date
withColumn("start_end_array"), array($"startDate", $"endDate").
withColumn("start_end_array"), explode($"start_end_array"))
Thank you for any leads.

case class MyData(id: String, startDate: String, endDate: String, list: List[String])
val inputData = Seq(("1_1", "2018-02-07", "2018-04-28"), ("1_2", "2018-05-06", "2018-05-31"), ("2_2", "2017-04-13", "2017-04-14"))
inputData.map(x => {
import java.time.temporal._
import java.time._
val startDate = LocalDate.parse(x._2)
val endDate = LocalDate.parse(x._3)
val diff = ChronoUnit.MONTHS.between(startDate, endDate)
var result = List[String]();
for (index <- 0 to diff.toInt) {
result = (startDate.getYear + "-" + (startDate.getMonth.getValue + index) + "-01") :: result
}
new MyData(x._1, x._2, x._3, result)
}).foreach(println)

What does Java's SimpleDateFormat make with milli/micro-seconds?

I have simple test
#SuppressWarnings("deprecation")
#Test
public void test_NO_MILLIS() throws ParseException {
String rabbit = "22-OCT-15 06.37.35";
final String PATTERN = "dd-MMM-yy HH.mm.ss";
Date dateObject = new SimpleDateFormat(PATTERN).parse(rabbit);
Assert.assertNotNull(dateObject);
Assert.assertEquals(22, dateObject.getDate());
Assert.assertEquals(10, dateObject.getMonth() + 1);
Assert.assertEquals(2015, dateObject.getYear() + 1900);
Assert.assertEquals(6, dateObject.getHours());
Assert.assertEquals(37, dateObject.getMinutes());
Assert.assertEquals(35, dateObject.getSeconds());
}
And everything goes right. I get 22 as day in result.
But after I am adding microseconds both to pattern and to string value to be parsed
#Test
public void test_MILLIS() throws ParseException {
String rabbit = "22-OCT-15 06.37.35.586173000";
final String PATTERN = "dd-MMM-yy HH.mm.ss.SSSSSSSSS";
Date dateObject = new SimpleDateFormat(PATTERN).parse(rabbit);
Assert.assertNotNull(dateObject);
Assert.assertEquals(22, dateObject.getDate());
Assert.assertEquals(10, dateObject.getMonth() + 1);
Assert.assertEquals(2015, dateObject.getYear() + 1900);
Assert.assertEquals(6, dateObject.getHours());
Assert.assertEquals(37, dateObject.getMinutes());
Assert.assertEquals(35, dateObject.getSeconds());
}
I get an assert failure
junit.framework.AssertionFailedError: expected:<22> but was:<29>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at main.TestDateFormatTest.test_MILLIS(TestDateFormatTest.java:36)
...
Which means that day has become 29 instead of 22. What has gone wrong?
Tested
Platforms: mac osx 10.9, ubuntu, win7
jdk: 7,6

The format pattern S for milliseconds doesn't take into account mathematical placement values; it just sees 586173000 as the number of milliseconds to add to the rest of the date. That number is equivalent to about 6.784 days, so that explains why the date became 29 instead of 22.
Before parsing, cut off the milliseconds at 3 digits, e.g. "22-OCT-15 06.37.35.586", so it's interpreted as 586 milliseconds.

Java Date Format in "args"

So, I'm using JD-GUI to decompile some Java class files which doesn't have any documentation. This Java Application is running on a Windows Server and running via taskscheduler with a cmd file (I don't know exactly what, intern here).
If some one who worked with an args command can help me out, I would really appreciate it. The code snipped is attached, all what I want is to understand is what the for loop is doing at the end of the code.
public class Main
{
public static void main(String[] args)
{
try
{
List violationList = new ArrayList();
Calendar cal = Calendar.getInstance();
Date inputDate = new Date();
cal.setTime(inputDate);
cal.set(11, 0);
cal.set(12, 0);
cal.set(13, 1);
inputDate = cal.getTime();
SimpleDateFormat dateFormat = new SimpleDateFormat("MM/dd/yy");
for (int ii = 0; ii < args.length; ii++) {
if (args[ii].equals("-date")) {
inputDate = dateFormat.parse(args[(ii + 1)]);
}
}
Here is the run.cmd file along with Java files I found. Notice there is .Main mention at the last line, something to do with those %1 and %2.
RUN.CMD
#ECHO OFF
set CP=.;realapplication.jar; commons- logging.jar;msbase.jar;mail.jar;activation.jar;log4j.jar;
set CP=%CP%sqljdbc4.jar;msutil.jar;spring-1.2.6.jar;commons-httpclient-2.0.1.jar;weblogic_8.1.jar;
set CP=%CP%zipfilemanager.jar;kmoscommon.jar;sessionpool.jar;kmosdao.jar;gendao.jar;rowset-1.0.1.jar;
java -classpath "%CP%" com.adnan.Main %1 %2

This for loop goes over the command line arguments and searches for a pair of arguments in the form of -date 16/11/81 (the given date is just an example, of course). Once it finds it, it parses the second argument (16/11/81, in this case) to a java.util.Date object and stores it in the inputDate variable. If the -date argument is omitted, today's date will be used.

MS Project showing incorrect start date and enddate

I'm using mpxj 4.5 to export ms project. When i open mpx file by ms project, I'm getting incorrect start date and finish date, but it's calculating duration correctly. What am i doing wrong, please tell.
To create mpx task I use following parametrs of mpxj:
public ProjectFile exportToMSProject(Integer projectID){
file.setAutoTaskID(true);
...........
}
private void createMSProjectTask(ProjectFile file, EUser user, ETask eTask)
Task task = file.addTask();
task.setUniqueID(eTask.getObjectID());
task.setName(eTask.getName());
task.setNotes(eTask.getDescription());
task.setType(TaskType.FIXED_DURATION);
long workTime = 0;
if(eTask.getDueDate() != null && eTask.getStartDate() != null){
workTime = (eTask.getDueDate().getTime() - eTask.getStartDate().getTime()) / (1000 * 60 * 60);
}
if(eTask.isAllDay()){
task.setDuration(Duration.getInstance(workTime / 24, TimeUnit.DAYS));
}else {
task.setDuration(Duration.getInstance(workTime, TimeUnit.HOURS));
}
if(eTask.getStartDate() != null)
task.setStart(user.getUserDate(eTask.getStartDate())); //get time from user time zone
if(eTask.getDueDate() != null)
task.setFinish(user.getUserDate(eTask.getDueDate()));
task.setPercentageComplete(eTask.getPercent() != null ? eTask.getPercent() : new Float(0.0));
if(eTask.getActualStartDate() != null)
task.setActualStart(eTask.getActualStartDate());
}
I'm getting repeatedly startdate and incorrect enddate. What is the problem here? Any suggestion will be appreciated.

I had the same issue. From my point of view, if you not have any predecessor or actual start date of your tasks, mpxj sets startdate automatically, I mean, it gets startdate from your project header or first task's start date of your project. That's why you got repeated startdate.
Here is the simple example (I'm using mpxj 4,5 and creating .mpx file):
SimpleDateFormat df = new SimpleDateFormat("dd/MM/yyyy");
ProjectFile file = new ProjectFile();
ProjectHeader header = file.getProjectHeader();
header.setStartDate(df.parse("01/05/2014"));
Task task1 = file.addTask();
task1.setName("Summary Task");
Task task2 = task1.addTask();
task2.setName("First Sub Task");
task2.setDuration(Duration.getInstance(10.5, TimeUnit.HOURS));
task2.setStart(df.parse("01/05/2014"));
task2.setPercentageComplete(NumberUtility.getDouble(50.0));
Task task3 = task1.addTask();
task3.setName("Second Sub Task");
task3.setStart(df.parse("11/05/2014"));
task3.setDuration(Duration.getInstance(10, TimeUnit.HOURS));
Task milestone1 = task1.addTask();
milestone1.setName("Milestone");
milestone1.setStart(df.parse("21/05/2014"));
milestone1.setDuration(Duration.getInstance(0, TimeUnit.HOURS));
Task task4 = file.addTask();
task4.setName("Last Task");
task4.setDuration(Duration.getInstance(8, TimeUnit.HOURS));
task4.setStart(df.parse("02/05/2014"));
task4.setPercentageComplete(NumberUtility.getDouble(70.0));
ProjectWriter writer = getWriter(filename);
writer.write(file, filename);
As you have seen, I'm not giving any predecessor or actual start date. If you run this you will get following result.
In order to get start date correctly, i gave an actual startdate not a startdate, and it worked. In addition if you have predecessor, you will get parent task's due date as a startdate.
task.setActualStart(df.parse("02/05/2014"));
By the way, ms project sets due date based on your duration.

Halting program execution between certain hours of the week

I am writing a Java program that is required to copy files and folders between the following hours:
Mon - 18:00 to 06:30
Tue - 18:00 to 06:30
Wed - 18:00 to 06:30
Thu - 18:00 to 06:30
Fri - 18:00 to 06:30
Sat - all day
Sun - all day
The program will run continuously until it has finished copying all files and folders. However, outside of the above hours the program should just sleep.
I am using a properties file to store the above settings.
UPDATE
I am looking for the simplest possible implementation including the format of the properties in the properties file as well as the code that will make the checks.

I would do it like this
final Map<Integer, String> schedule = new HashMap<>();
// parse your settings and fill schedule
schedule.put(Calendar.MONDAY, "18:00 to 06:30");
// ...
// create timer to fire e.g. every hour
new Timer().scheduleAtFixedRate(new TimerTask() {
public void run() {
Calendar c = Calendar.getInstance();
String s = schedule.get(c.get(Calendar.DAY_OF_WEEK));
if (withinTimeRange(c, s)) { // implement withinTimeRange func
// copy files
}
}}, 0, 1000 * 3600);

Since your program is going to run continuously, the simplest solution is to check the day and time before copying a file. If the time is during off hours, go ahead and copy the next file, otherwise Thread.sleep.
If this is an internal, one-off kind of program, I would go ahead and hard-code the business hours instead of reading the properties file. No need to add complexity.

whenever your program is launched, get the current time, and check day today's day.
check whether it lies in permissible time if yes let it continue. If not, find the time at 00:00am of that 'day'. and find the time at xx:yyZZ (start of permissible time). calculate the difference, and let the program sleep for that much of time.

Thank you for your suggestions.
I came up with a working solution in the end which if it gets enough points I will mark as the answer. The way I attempted to solve this problem was by thinking about non-working hours rather than working hours. This code is just for illustration
# Properties
Mon = 06:30-18:00
Tue = 06:30-18:00
Wed = 06:30-18:00
Thu = 06:30-18:00
Fri = 06:30-18:00
Loop over the properties to get their values
String[] days = { "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat" };
Map<Integer, Integer[]> nonWorkingHours = new HashMap<Integer, Integer[]>();
for( int i = 0; i < days.length; i++ ) // for each property in file
{
// excluded implementation of getConfig
String prop = getConfig( days[ i ] ); // e.g. "06:00-19:00"
// +1 to match CALENDAR.DAY_OF_WEEK
nonWorkingHours.put( i + 1, getHours( prop );
}
My function to parse property excluding error handling
// e.g. if prop = "06:00-19:00" then { 6, 0, 19, 0 } is returned
public Integer[] getHours( String prop )
{
String times = prop.split( "(:|-)" );
Integer[] t = new Integer[4];
for( int i = 0; i < times.length; i++ )
{
t[i] = Integer.parseInt( times[i] );
}
return t;
}
And finally the function that implements the halt
private void sleepIfOutsideWorkingHours()
{
Integer[] data = nonWorkingHours.get( currentDay );
if( data != null )
{
Calendar c = Calendar.getInstance();
Integer currentSeconds = ( Calendar.HOUR_OF_DAY * 3600 ) + ( Calendar.MINUTE * 60 );
Integer stopFrom = ( data[ 0 ] * 3600 ) + ( data[ 1 ] * 60 );
Integer stopTill = ( data[ 2 ] * 3600 ) + ( data[ 3 ] * 60 );
if( currentSeconds > stopFrom && currentSeconds < stopTill )
{
Integer secondsDiff = stopTill - currentSeconds;
if( secondsDiff > 0 )
{
try
{
Thread.sleep( secondsDiff * 1000 ); // turn seconds to milliseconds
}
catch( InterruptedException e )
{
// error handling
}
}
}
}
}
And finally just call the function below just before copying each file and if it is being run outside working hours it will stop the program.
sleepIfOutsideWorkingHours();
I am sure there is a simpler way of doing it :-) but there it is.

You should try using a Continuous Integration system. That scenario can be easily set up using Jenkins CI for example.
The reason i advise doing it so in a ambient like that is that you can keep a better control on the history of your program runs.