We Keep Coding

When using Mockito verify method `times` got null

When I trying to use Mockito verify to test extral call times of testMethod(indeed this method is just execute 2 times), but I got following message.
verify(XXXXXXX, times(2)).testMethod(any(XXXXXX.class))
| | | | | |
| | | null null class XXXXXX
| | Wanted invocations count: 2
| Mock for XXXXXXX, hashCode: 292294397
Mock for XXXXXXX, hashCode: 292294397
If I change the times to 1, and the error message is:
verify(XXXXXXX, times(1)).testMethod(any(XXXXXX.class))
| | | | | |
| | | | null class XXX
| | | org.mockito.exceptions.verification.TooManyActualInvocations:
| | | XXX.XXXX(
| | | <any XXX>
| | | );
| | | Wanted 1 time:
| | | -> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| | | But was 2 times:
| | | -> at XXXX.XXX(XXX.java:129)
| | | -> at XXXXX.XXX(XXX.java:129)
| | |
| | |
| | Wanted invocations count: 1
Does anybody have met this problem before?

JDBC pagination in one to many relation and mapping to pojo?

we have a messaging logs table and we are using this table to provide a search UI which lets to search messages by id or status or auditor or date. Table audit looks like below
+-----------+----------+---------+---------------------+
| messageId | auditor | status | timestamp |
+-----------+----------+---------+---------------------+
| 10 | program1 | Failed | 2020-08-01 10:00:00 |
| 11 | program2 | success | 2020-08-01 10:01:10 |
| 12 | program3 | Failed | 2020-08-01 10:01:15 |
+-----------+----------+---------+---------------------+
Since in a given date range we could have many messages matching the criteria so we added pagination for the query. Now as a new feature we are adding another table with one to many relation which contain tags as the possible reasons for the failure. The table failure_tags will look like below
+-----------+----------+-------+--------+
| messageId | auditor | type | cause |
+-----------+----------+-------+--------+
| 10 | program1 | type1 | cause1 |
| 10 | program1 | type1 | cause2 |
| 10 | program1 | type2 | cause3 |
+-----------+----------+-------+--------+
Now for a general search query for a status = 'Failed' and using left join with the other table will retrieve 4 rows as below
+-----------+----------+-------+--------+---------------------+
| messageId | auditor | type | cause | timestamp |
+-----------+----------+-------+--------+---------------------+
| 10 | program1 | type1 | cause1 | 2020-08-01 10:00:00 |
| 10 | program1 | type1 | cause2 | 2020-08-01 10:00:00 |
| 10 | program1 | type2 | cause3 | 2020-08-01 10:00:00 |
| 12 | program3 | | | 2020-08-01 10:01:15 |
+-----------+----------+-------+--------+---------------------+
The requirement is to since the 3 rows of messageId 10 belongs to same message the requirement is to merge the rows into 1 in json response, so the response will have only 2 elements
[
{
"messageId": "10",
"auditor": "program1",
"failures": [
{
"type": "type1",
"cause": [
"cause1",
"cause2"
]
},
{
"type": "type2",
"cause": [
"cause3"
]
}
],
"date": "2020-08-01 10:00:00"
},
{
"messageId": "12",
"auditor": "program3",
"failures": [],
"date": "2020-08-01 10:01:15"
}
]
Because of this merge for a pagination request of 10 elements after fetching from the database and merging would result in less than 10 results.
The 1 solution, I could think of is after merging, if its less than page size, initiate a search again do the combining process and take the top 10 elements. Is there any better solution to get all the results in 1 query instead of going twice or more to DB ?
We use generic spring - JDBC not the JPA.

JSprit not using closer vehicle, due to time issues

Single-job schedule with two vehicles. One vehicle starts close to the job, the other starts far from the job. Seems it should prefer to use the closer vehicle, as there's a cost-per-distance. But it uses the farther one, if there's a non-zero setCostPerWaitingTime(). Why?
public void testUseCloserVehicleWhenCostsAreSet() throws Exception {
VehicleType type = VehicleTypeImpl.Builder.newInstance("generic")
.setCostPerDistance(0.017753)
//.setCostPerTransportTime(1.0)
.setCostPerWaitingTime(1.0)
.build();
double serviceTime = 420.0;
Location pointA = Location.newInstance(100.0, 100.0);
Location pointB = Location.newInstance(100.0, 200.0);
Location closeToPointA = Location.newInstance(110.0, 110.0);
Location farFromPointA = Location.newInstance(500.0, 110.0);
VehicleRoutingProblem vrp = VehicleRoutingProblem.Builder
.newInstance()
.setFleetSize(VehicleRoutingProblem.FleetSize.FINITE)
.addVehicle(VehicleImpl.Builder.newInstance("CloseBy")
.setType(type)
.setStartLocation(closeToPointA)
.build())
.addVehicle(VehicleImpl.Builder.newInstance("FarAway")
.setType(type)
.setStartLocation(farFromPointA)
.build())
.addJob(Shipment.Builder.newInstance("123")
.setPickupLocation(pointA)
.setPickupServiceTime(serviceTime)
.setDeliveryLocation(pointB)
.setDeliveryServiceTime(serviceTime)
.setPickupTimeWindow(new TimeWindow(36000.0, 36360.0))
.setDeliveryTimeWindow(new TimeWindow(36360.0, 36720.0))
.setMaxTimeInVehicle(720.0)
.build())
.build();
VehicleRoutingAlgorithm algorithm = Jsprit.Builder.newInstance(vrp)
.buildAlgorithm();
VehicleRoutingProblemSolution bestSolution = Solutions.bestOf(algorithm.searchSolutions());
SolutionPrinterWithTimes.print(vrp, bestSolution, SolutionPrinterWithTimes.Print.VERBOSE);
System.out.flush();
assertEquals("CloseBy", bestSolution.getRoutes().iterator().next().getVehicle().getId());
}
Result:
+----------------------------------------------------------+
| solution |
+---------------+------------------------------------------+
| indicator | value |
+---------------+------------------------------------------+
| costs | 35616.03246830352 |
| noVehicles | 1 |
| unassgndJobs | 0 |
+----------------------------------------------------------+
+--------------------------------------------------------------------------------------------------------------------------------+
| detailed solution |
+---------+----------------------+-----------------------+-----------------+-----------------+-----------------+-----------------+
| route | vehicle | activity | job | arrTime | endTime | costs |
+---------+----------------------+-----------------------+-----------------+-----------------+-----------------+-----------------+
| 1 | FarAway | start | - | undef | 0 | 0 |
| 1 | FarAway | pickupShipment | 123 | 400 | 36420 | 35607 |
| 1 | FarAway | deliverShipment | 123 | 36520 | 36940 | 35609 |
| 1 | FarAway | end | - | 37350 | undef | 35616 |
+--------------------------------------------------------------------------------------------------------------------------------+
junit.framework.ComparisonFailure:
Expected :CloseBy
Actual :FarAway
I suspect it has something to do with the vehicle arriving at 400 for a job that can't start until 36000. Is there a way to prevent that, so the vehicle starts only as early as needed to reach the first job? Does setCostPerWaitingTime do something other than what I think?

Here's a comparison of the job with only the CloseBy vehicle.
+--------------------------------------------------------------------------------------------------------------------------------+
| detailed solution |
+---------+----------------------+-----------------------+-----------------+-----------------+-----------------+-----------------+
| route | vehicle | activity | job | arrTime | endTime | costs |
+---------+----------------------+-----------------------+-----------------+-----------------+-----------------+-----------------+
| 1 | FarAway | start | - | undef | 0 | 0 |
| 1 | FarAway | pickupShipment | 123 | 400 | 36420 | 35607 |
| 1 | FarAway | deliverShipment | 123 | 36520 | 36940 | 35609 |
| 1 | FarAway | end | - | 37350 | undef | 35616 |
+--------------------------------------------------------------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------------------------------------------+
| detailed solution |
+---------+----------------------+-----------------------+-----------------+-----------------+-----------------+-----------------+
| route | vehicle | activity | job | arrTime | endTime | costs |
+---------+----------------------+-----------------------+-----------------+-----------------+-----------------+-----------------+
| 1 | CloseBy | start | - | undef | 0 | 0 |
| 1 | CloseBy | pickupShipment | 123 | 14 | 36420 | 35986 |
| 1 | CloseBy | deliverShipment | 123 | 36520 | 36940 | 35988 |
| 1 | CloseBy | end | - | 37031 | undef | 35989 |
+--------------------------------------------------------------------------------------------------------------------------------+
I think the problem is that the CloseBy vehicle arrives sooner, so it pays higher wait costs, while the other vehicle is driving during that time so pays less in wait costs. This would be mitigated if the vehicle didn't start until it needed to, but I'm unsure how to set that up.

Apache Spark find first different preceding row in Dataframe

I have an Apache Spark Dataframe of the following format
| ID | groupId | phaseName |
|----|-----------|-----------|
| 10 | someHash1 | PhaseA |
| 11 | someHash1 | PhaseB |
| 12 | someHash1 | PhaseB |
| 13 | someHash2 | PhaseX |
| 14 | someHash2 | PhaseY |
Each row represents a phase that happens in a procedure that consists of several of these phases. The ID column represents a sequential order of phases and the groupId column shows which phases belong together.
I want to add a new column to the dataframe: previousPhaseName. This column should indicate the previous different phase from the same procedure. The first phase of a process (the one with the minimum ID) will have null as previous phase. When a phase occurs twice or more, the second (third...) occurrence will have the same previousPhaseName For example:
df =
| ID | groupId | phaseName | prevPhaseName |
|----|-----------|-----------|---------------|
| 10 | someHash1 | PhaseA | null |
| 11 | someHash1 | PhaseB | PhaseA |
| 12 | someHash1 | PhaseB | PhaseA |
| 13 | someHash2 | PhaseX | null |
| 14 | someHash2 | PhaseY | PhaseX |
I am not sure how to implement this. My first approach would be:
create a second empty dataframe df2
for each row in df:
find the row with groupId = row.groupId, ID < row.ID, and maximum id
add this row to df2
join df1 and df2
Partial Solution using Window Functions
I used Window Functionsto aggregate the Name of the previous phase, the number of previous occurrences (not necessarily in a row) of the current phase in the group and the information whether the current and previous phase names are equal:
WindowSpec windowSpecPrev = Window
.partitionBy(df.col("groupId"))
.orderBy(df.col("ID"));
WindowSpec windowSpecCount = Window
.partitionBy(df.col("groupId"), df.col("phaseName"))
.orderBy(df.col("ID"))
.rowsBetween(Long.MIN_VALUE, 0);
df
.withColumn("prevPhase", functions.lag("phaseName", 1).over(windowSpecPrev))
.withColumn("phaseCount", functions.count("phaseId").over(windowSpecCount))
.withColumn("prevSame", when(col("prevPhase").equalTo(col("phaseName")),1).otherwise(0))
df =
| ID | groupId | phaseName | prevPhase | phaseCount | prevSame |
|----|-----------|-----------|-------------|------------|----------|
| 10 | someHash1 | PhaseA | null | 1 | 0 |
| 11 | someHash1 | PhaseB | PhaseA | 1 | 0 |
| 12 | someHash1 | PhaseB | PhaseB | 2 | 1 |
| 13 | someHash2 | PhaseX | null | 1 | 0 |
| 14 | someHash2 | PhaseY | PhaseX | 1 | 0 |
This is still not what I wanted to achieve but good enough for now
Further Ideas
To get the the name of the previous distinct phase I see three possibilities that I have not investigated thoroughly:
Implement an own lagfunction that does not take an offset but recursively checks the previous line until it finds a value that is different from the given line. (Though I don't think it's possible to use own analytic window functions in Spark SQL)
Find a way to dynamically set the offset of the lag function according to the value of phaseCount. (That may fail if the previous occurrences of the same phaseName do not appear in a single sequence)
Use a UserDefinedAggregateFunction over the window that stores the ID and phaseName of the first given input and seeks for the highest ID with different phaseName.

I was able to solve this problem in the following way:
Get the (ordinary) previous phase.
Introduce a new id that groups phases that occur in sequential order. (With help of this answer). This takes two steps. First checking whether the current and previous phase names are equal and assigning a groupCount value accordingly. Second computing a cumulative sum over this value.
Assign the previous phase of the first row of a sequential group to all its members.
Implementation
WindowSpec specGroup = Window.partitionBy(col("groupId"))
.orderBy(col("ID"));
WindowSpec specSeqGroupId = Window.partitionBy(col("groupId"))
.orderBy(col("ID"))
.rowsBetween(Long.MIN_VALUE, 0);
WindowSpec specPrevDiff = Window.partitionBy(col("groupId"), col("seqGroupId"))
.orderBy(col("ID"))
.rowsBetween(Long.MIN_VALUE, 0);
df.withColumn("prevPhase", coalesce(lag("phaseName", 1).over(specGroup), lit("NO_PREV")))
.withColumn("seqCount", when(col("prevPhase").equalTo(col("phaseName")).or(col("prevPhase").equalTo("NO_PREV")),0).otherwise(1))
.withColumn("seqGroupId", sum("seqCount").over(specSeqGroupId))
.withColumn("prevDiff", first("prevPhase").over(specPrevDiff));
Result
df =
| ID | groupId | phaseName | prevPhase | seqCount | seqGroupId | prevDiff |
|----|-----------|-----------|-----------|----------|------------|----------|
| 10 | someHash1 | PhaseA | NO_PREV | 0 | 0 | NO_PREV |
| 11 | someHash1 | PhaseB | PhaseA | 1 | 1 | PhaseA |
| 12 | someHash1 | PhaseB | PhaseA | 0 | 1 | PhaseA |
| 13 | someHash2 | PhaseX | NO_PREV | 0 | 0 | NO_PREV |
| 14 | someHash2 | PhaseY | PhaseX | 1 | 1 | PhaseX |
Any suggestions, specially in terms of efficiency of these operations are appreciated.

I guess you can use Spark window (row frame) functions. Check the api documentation and the following post.
https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html

Can't alter table(blocked) while hibernate is connected to mysql server

I have a table name test;
mysql> desc test;
+--------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | 0 | |
| name | varchar(255) | YES | | NULL | |
| gendar | varchar(255) | YES | | NULL | |
+--------+--------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql> select * from test;
+----+------+--------+
| id | name | gendar |
+----+------+--------+
| 0 | John | male |
+----+------+--------+
1 row in set (0.00 sec)
And have a hibernate Entity like this:
#Entity
#Table(name = "test",schema = "", catalog = "mydb")
public class TestEntity {
private int id;
private String name;
private String gendar;
#Id
#Column(name = "id")
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
#Basic
#Column(name = "name")
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
#Basic
#Column(name = "gendar")
public String getGendar() {
return gendar;
}
public void setGendar(String gendar) {
this.gendar = gendar;
}
}
The hibernate.xml is:
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="connection.url">jdbc:mysql://localhost:3306/mydb</property>
<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.username">test</property>
<property name="connection.password">test</property>
<mapping class="me.armnotstrong.sql.TestEntity" />
<!-- DB schema will be updated if needed -->
<!-- <property name="hbm2ddl.auto">update</property> -->
</session-factory>
</hibernate-configuration>
The hibernate session is generated by this factory class:
public class HBSession {
private static final SessionFactory ourSessionFactory;
private static final ServiceRegistry serviceRegistry;
static {
try {
Configuration configuration = new Configuration();
configuration.configure();
serviceRegistry = new ServiceRegistryBuilder().applySettings(configuration.getProperties()).buildServiceRegistry();
ourSessionFactory = configuration.buildSessionFactory(serviceRegistry);
} catch (Throwable ex) {
throw new ExceptionInInitializerError(ex);
}
}
public static Session getSession() throws HibernateException {
Session session = ourSessionFactory.openSession();
return session;
}
}
And I just add a user to the db by this
public class TestHb {
public static void main(String[] argvs){
Session session = HBSession.getSession();
TestEntity testEntity = new TestEntity();
testEntity.setName("John");
testEntity.setGendar("male");
Transaction tx = session.beginTransaction();
session.save(testEntity);
tx.commit();
session.close();
}
}
After run the TestHB code above to add a user to mysql, the session seemed to just hang there, and won't colse, diagnose using netstat -nap just proved my guess, but it's ok, I think. In fact, this just simulate the condition of a long connection to the db in the product environment which hibernate make.
The question comes that, when I alter the table when hibernate connection still there, The mysql client just blocked. and wont do the alter action as presumed unless I restart the mysql service. and in product environment, after restart the mysql service, one thing come after another, The hibernate app will not work anymore.
So what should I do to alter the table and keep my hibernate also working?
as required by #Florent, here is some addition info come out of command SHOW FULL PROCESSLIST and SHOW OPEN TABLES;
mysql> show open tables;
+----------+----------+--------+-------------+
| Database | Table | In_use | Name_locked |
+----------+----------+--------+-------------+
| test | merchant | 0 | 0 |
| test | test | 0 | 0 |
| test | orders | 0 | 0 |
| test | product | 0 | 0 |
| test | codes | 0 | 0 |
+----------+----------+--------+-------------+
5 rows in set (0.02 sec)
mysql> show full processlist;
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| 42 | test | localhost:35790 | test | Sleep | 4 | | NULL |
| 43 | test | localhost:35801 | test | Sleep | 4 | | NULL |
| 44 | test | localhost:35802 | test | Sleep | 4 | | NULL |
| 45 | test | localhost:35803 | test | Sleep | 4 | | NULL |
| 46 | test | localhost:35804 | test | Sleep | 4 | | NULL |
| 157 | test | localhost:51516 | test | Sleep | 174 | | NULL |
| 161 | test | localhost:53988 | test | Sleep | 174 | | NULL |
| 180 | test | localhost:58501 | test | Sleep | 174 | | NULL |
| 192 | test | localhost:47228 | test | Sleep | 7217 | | NULL |
| 193 | test | localhost:49372 | test | Sleep | 4485 | | NULL |
| 196 | test | localhost | test | Sleep | 9256 | | NULL |
| 197 | test | localhost | test | Sleep | 4555 | | NULL |
| 198 | test | localhost:42411 | test | Sleep | 4485 | | NULL |
| 200 | test | localhost | test | Query | 0 | NULL | show full processlist |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
14 rows in set (0.00 sec)