We Keep Coding

Group by inside otherwise clause, spark java

I have this process in SparkJava (IntelliJ app) where I have a problem that I don`t know how to resolve yet. First I declare the dataset:
private static final String CONTRA1 = "contra1";
query = "select contra1, ..., eadfinal, , ..., data_date" + FROM + dbSchema + TBLNAME " + WHERE fech = '" + fechjmCto2 + "' AND s1emp=49";
Dataset<Row> jmCto2 = sql.sql(query);
Then I have the calculations, I analyze some fields to assign some literal values. My problem is in the aggegate function:
Dataset<Row> contrCapOk1 = contrCapOk.join(jmCto2,
contrCapOk.col(CONTRA1).equalTo(jmCto2.col(CONTRA1)),LEFT)
.select(contrCapOk.col("*"),
jmCto2.col("ind"),
functions.when(jmCto2.col(CONTRA1).isNull(),functions.lit(NUEVES))
.when(jmCto2.col("ind").equalTo("N"),functions.lit(UNOS))
.otherwise(jmCto2.groupBy(CONTRA1).agg(functions.sum(jmCto2.col("eadfinal")))).as("EAD"),
What I want is to make the sum in the otherwise part. But when I execute the cluster give me this message in the log.
User class threw exception: java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.Dataset [contra1: int, sum(eadfinal): decimal(33,6)]
in the line 211, the otherwise line.
Do you know what the problem could be?.
Thanks.

You cannot use groupBy and aggregation function in a column clause. To do what you want to do, you have to use a window.
For you case, you can define the following window:
import org.apache.spark.sql.expressions.Window;
import org.apache.spark.sql.expressions.WindowSpec;
...
WindowSpec window = Window
.partitionBy(CONTRA1)
.rangeBetween(Window.unboundedPreceding(), Window.unboundedFollowing());
Where
partitionBy is the equivalent of groupBy for aggregation
rangeBetween determine which rows of the partition will be used by aggregation function, here we take all rows
And then you use this window when calling your aggregation function, as follow:
import org.apache.spark.sql.functions;
...
Dataset<Row> contrCapOk1 = contrCapOk.join(
jmCto2,
contrCapOk.col(CONTRA1).equalTo(jmCto2.col(CONTRA1)),
LEFT
)
.select(
contrCapOk.col("*"),
jmCto2.col("ind"),
functions.when(jmCto2.col(CONTRA1).isNull(), functions.lit(NUEVES))
.when(jmCto2.col("ind").equalTo("N"), functions.lit(UNOS))
.otherwise(functions.sum(jmCto2.col("eadfinal")).over(window))
.as("EAD")
)

How to fetch complex model with a result of select statment using fetchgoups and map

I'm using Jooq with Kotlin and i want to write a statement that fetches data from a query that uses couple of tables using join statement(example attached)
The problem I'm facing is that I want to map the result to my complex model which consist of one to many relationships and also many to many.
According to my knowledge I know i can use fetchgroups operation in Jooq to some how map the records but i still can't figure out how to get the result into my model.
my model:
data class MicroserviceDto(
val microservice_id: Long = 1,
val microservice_name: String? = "",
val endpoint: String? = "",
val mappings: String? = "",
val solutionDefinitionMinimalDtoList: List<SolutionDefinitionDto> = emptyList(),
val projectFileDtoList: List<ProjectFileDto> = emptyList()
)
data class SolutionDefinitionDto(
val solution_definition_id: Long = 0L,
val solution_definition_name: String = "",
val solutionId: String = "",
val solutionVersion: String = ""
)
data class ProjectFileDto(
val project_file_id: Long = 1,
val model: String = "",
val relativePath: String = "",
val fileContentDtoList: List<FileContentDto> = emptyList()
)
data class FileContentDto(
val file_content_id: Long = 1,
val content: ByteArray = ByteArray(0)
)
Link to my schema diagram
Database Diagram visualization
Explanation of the diagram:
Microservice has many to many relationship with SolutionDefinistion
ProjectFile has one to many relationship with Microservice
ProjectFile has one to many relationship with SolutionDefinition
FileContent has one to many with ProjectFile
I've created a view to represent my desired query with all tables and the join statements between them.
Here is the View:
CREATE OR REPLACE VIEW Microservice_Metadata_by_Microservice_Id AS
select
# microservice
M.id as `microservice_id`,
M.name as `microservice_name`,
M.mappings,
M.endpoint,
# solution definition
SD.id as `solution_definition_id`,
SD.name as `solution_definition_name`,
SD.solutionId,
SD.solutionVersion,
# project file of microservice
PF.id as `project_file_id`,
PF.relativePath,
PF.model,
# file content data of project file
FC.id as `file_content_id`,
FC.content
from Microservice M
# get project file
left join Microservice_SolutionDefinition MSD
on MSD.microserviceId = M.id
left join ProjectFile PF
on PF.microserviceId = M.id
# get data content
left JOIN FileContent FC
on PF.id = FC.projectFileId
# get solutions of microservice
left join SolutionDefinition SD
on SD.id = MSD.solutionDefinitionId;
How can I implement such a Jooq dsl query that map the ResultSet to my data model

SPARK SQL not in or NOT EXISTS

I have a scenario where there are two sheets(csv). Created two tables for it.
When there s a good data I could map it with a value( id statistic value) from the second table. If I have bad data I should map it with id statistic value again (but a different value). however, I'm unable to use not exists in spark SQL. And I get the error below:
mismatched input 'from' expecting {, 'WHERE', 'GROUP', 'ORDER', 'HAVING', 'LIMIT', 'LATERAL', 'WINDOW', 'UNION', 'EXCEPT', 'INTERSECT', 'SORT', 'CLUSTER', 'DISTRIBUTE'}(line 1, pos 386)
at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:197)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:99)
at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:45)
CODE:
select
a.ptf_id,a.ptf_code,a.share_id,a.share_code,a.bench_id,a.bench_code
, a.l1_calculation_date,a.l1_begin_date,a.l1_end_date,a.l1_running_date
, a.l1_frequency,a.l1_calculation_step,a.l1_performance_currency
, a.l1_configuration,a.l1_valuation_source,a.l1_nav_valuation_type
, a.l1_setting_reference_type, a.l1_setting_valuation_type
, a.l1_sharpe_ratio_annualized as value,b.id_statistic
from
parquetFile a,
pairRDD b,
stats c
where
a.l1_nav_valuation_type= b.l1_nav_valuation_type
and a.l1_valuation_source = b.l1_valuation_source
and b.l1_Perf = 'l1_sharpe_ratio_annualized'
OR (a.ptf_id not EXISTS (
select e.ptf_id from pairRDD d, parquetFile e
where d.l1_valuation_source = e.l1_valuation_source
AND d.l1_nav_valuation_type = e.l1_nav_valuation_type)
and b.l1_valuation_source ='')
this query works in SQL if I use "NOT in"
please help me on what other options to use in this scenario other than not exists.

The query is a bit hard to follow as written. It's not clear (to me anyway) what your intentions are for the stats table since you don't select anything from it so I removed it. Obviously I haven't tried this myself but at a glance, you might try something like this instead:
select
a.PTF_ID,
a.PTF_CODE,
a.SHARE_ID,
a.SHARE_CODE,
a.BENCH_ID,
a.BENCH_CODE,
a.L1_CALCULATION_DATE,
a.L1_BEGIN_DATE,
a.L1_END_DATE,
a.L1_RUNNING_DATE,
a.L1_FREQUENCY,
a.L1_CALCULATION_STEP,
a.L1_PERFORMANCE_CURRENCY,
a.L1_CONFIGURATION,
a.L1_VALUATION_SOURCE,
a.L1_NAV_VALUATION_TYPE,
a.L1_SETTING_REFERENCE_TYPE,
a.L1_SETTING_VALUATION_TYPE,
a.L1_SHARPE_RATIO_ANNUALIZED as VALUE,
b.ID_STATISTIC
from
PARQUETFILE a
inner join
PAIRRDD b
on
a.L1_NAV_VALUATION_TYPE = b.L1_NAV_VALUATION_TYPE and
a.L1_VALUATION_SOURCE = b.L1_VALUATION_SOURCE
left outer join (
select
e.PTF_ID
from
PAIRRDD d
inner join
PARQUETFILE e
on
d.L1_VALUATION_SOURCE = e.L1_VALUATION_SOURCE AND
d.L1_NAV_VALUATION_TYPE = e.L1_NAV_VALUATION_TYPE
where
b.L1_VALUATION_SOURCE = ''
) x
on
a.PTF_ID = x.PTF_ID
where
b.L1_PERF = 'l1_sharpe_ratio_annualized' or
x.PTF_ID is null

rank() function usage in Spark SQL

Need some pointers in using rank()
I have extracted a column from a dataset..need to do the ranking.
Dataset<Row> inputCol= inputDataset.apply("Colname");
Dataset<Row> DSColAwithIndex=inputDSAAcolonly.withColumn("df1Rank", rank());
DSColAwithIndex.show();
I can sort the column and then append an index column too to get rank...but curious to known syntax and usage of rank()

Window spec need to be specified for rank()
val w = org.apache.spark.sql.expressions.Window.orderBy("date") //some spec
val leadDf = inputDSAAcolonly.withColumn("df1Rank", rank().over(w))
Edit: Java version of answer, as OP using Java
import org.apache.spark.sql.expressions.WindowSpec;
WindowSpec w = org.apache.spark.sql.expressions.Window.orderBy(colName);
Dataset<Row> leadDf = inputDSAAcolonly.withColumn("df1Rank", rank().over(w));

I have been searching for this,applying rank over my dataframe in java.
Using the provided answer in above comment,
import org.apache.spark.sql.expressions.WindowSpec;
WindowSpec w = org.apache.spark.sql.expressions.Window.orderBy(colName);
Dataset<Row> leadDf = inputDSAAcolonly.withColumn("df1Rank", rank().over(w));
worked for me,
Thanks
Gaurav.

DataFrame filtering based on second Dataframe

Using Spark SQL, I have two dataframes, they are created from one, such as:
df = sqlContext.createDataFrame(...);
df1 = df.filter("value = 'abc'"); //[path, value]
df2 = df.filter("value = 'qwe'"); //[path, value]
I want to filter df1, if part of its 'path' is any path in df2.
So if df1 has row with path 'a/b/c/d/e' I would find out if in df2 is a row that path is 'a/b/c'.
In SQL it should be like
SELECT * FROM df1 WHERE udf(path) IN (SELECT path FROM df2)
where udf is user defined function that shorten original path from df1.
Naive solution is to use JOIN and then filter result, but it is slow, since df1 and df2 have each more than 10mil rows.
I also tried following code, but firstly I had to create broadcast variable from df2
static Broadcast<DataFrame> bdf;
bdf = sc.broadcast(df2); //variable 'sc' is JavaSparkContext
sqlContext.createDataFrame(df1.javaRDD().filter(
new Function<Row, Boolean>(){
#Override
public Boolean call(Row row) throws Exception {
String foo = shortenPath(row.getString(0));
return bdf.value().filter("path = '"+foo+"'").count()>0;
}
}
), myClass.class)
the problem I'm having is that Spark got stuck when the return was evaluated/when filtering of df2 was performed.
I would like to know how to work with two dataframes to do this.
I really want to avoid JOIN. Any ideas?
EDIT>>
In my original code df1 has alias 'first' and df2 'second'. This join is not cartesian, and it also does not use broadcast.
df1 = df1.as("first");
df2 = df2.as("second");
df1.join(df2, df1.col("first.path").
lt(df2.col("second.path"))
, "left_outer").
filter("isPrefix(first.path, second.path)").
na().drop("any");
isPrefix is udf
UDF2 isPrefix = new UDF2<String, String, Boolean>() {
#Override
public Boolean call(String p, String s) throws Exception {
//return true if (p.length()+4==s.length()) and s.contains(p)
}};
shortenPath - it cuts last two characters in path
UDF1 shortenPath = new UDF1<String, String>() {
#Override
public String call(String s) throws Exception {
String[] foo = s.split("/");
String result = "";
for (int i = 0; i < foo.length-2; i++) {
result += foo[i];
if(i<foo.length-3) result+="/";
}
return result;
}
};
Example of records. Path is unique.
a/a/a/b/c abc
a/a/a qwe
a/b/c/d/e abc
a/b/c qwe
a/b/b/k foo
a/b/f/a bar
...
So df1 consits of
a/a/a/b/c abc
a/b/c/d/e abc
...
and df2 consits of
a/a/a qwe
a/b/c qwe
...

There at least few problems with your code:
you cannot execute action or transformation inside another action or transformation. It means that filtering broadcasted DataFrame simply cannot work and you should get an exception.
join you use is executed as a Cartesian product followed by filter. Since Spark is using Hashing for joins only equality based joins can be efficiently executed without Cartesian. It is slightly related to Why using a UDF in a SQL query leads to cartesian product?
if both DataFrames are relatively large and have similar size then broadcasting is unlikely to be useful. See Why my BroadcastHashJoin is slower than ShuffledHashJoin in Spark
not important when it comes to performance but isPrefix seems to wrong. In particular it looks like it can match both prefix and suffix
col("first.path").lt(col("second.path")) condition looks wrong. I assume you want a/a/a/b/c from df1 match a/a/a from df2. If so it should be gt not lt.
Probably the best thing you can do is something similar to this:
import org.apache.spark.sql.functions.{col, regexp_extract}
val df = sc.parallelize(Seq(
("a/a/a/b/c", "abc"), ("a/a/a","qwe"),
("a/b/c/d/e", "abc"), ("a/b/c", "qwe"),
("a/b/b/k", "foo"), ("a/b/f/a", "bar")
)).toDF("path", "value")
val df1 = df
.where(col("value") === "abc")
.withColumn("path_short", regexp_extract(col("path"), "^(.*)(/.){2}$", 1))
.as("df1")
val df2 = df.where(col("value") === "qwe").as("df2")
val joined = df1.join(df2, col("df1.path_short") === col("df2.path"))
You can try to broadcast one of the tables like this (Spark >= 1.5.0 only):
import org.apache.spark.sql.functions.broadcast
df1.join(broadcast(df2), col("df1.path_short") === col("df2.path"))
and increase auto broadcast limits, but as I've mentioned above it most likely will be less efficient than plain HashJoin.

As a possible way of implementing IN with subquery, the LEFT SEMI JOIN can be used:
JavaSparkContext javaSparkContext = new JavaSparkContext("local", "testApp");
SQLContext sqlContext = new SQLContext(javaSparkContext);
StructType schema = DataTypes.createStructType(new StructField[]{
DataTypes.createStructField("path", DataTypes.StringType, false),
DataTypes.createStructField("value", DataTypes.StringType, false)
});
// Prepare First DataFrame
List<Row> dataForFirstDF = new ArrayList<>();
dataForFirstDF.add(RowFactory.create("a/a/a/b/c", "abc"));
dataForFirstDF.add(RowFactory.create("a/b/c/d/e", "abc"));
dataForFirstDF.add(RowFactory.create("x/y/z", "xyz"));
DataFrame df1 = sqlContext.createDataFrame(javaSparkContext.parallelize(dataForFirstDF), schema);
//
df1.show();
//
// +---------+-----+
// | path|value|
// +---------+-----+
// |a/a/a/b/c| abc|
// |a/b/c/d/e| abc|
// | x/y/z| xyz|
// +---------+-----+
// Prepare Second DataFrame
List<Row> dataForSecondDF = new ArrayList<>();
dataForSecondDF.add(RowFactory.create("a/a/a", "qwe"));
dataForSecondDF.add(RowFactory.create("a/b/c", "qwe"));
DataFrame df2 = sqlContext.createDataFrame(javaSparkContext.parallelize(dataForSecondDF), schema);
// Use left semi join to filter out df1 based on path in df2
Column pathContains = functions.column("firstDF.path").contains(functions.column("secondDF.path"));
DataFrame result = df1.as("firstDF").join(df2.as("secondDF"), pathContains, "leftsemi");
//
result.show();
//
// +---------+-----+
// | path|value|
// +---------+-----+
// |a/a/a/b/c| abc|
// |a/b/c/d/e| abc|
// +---------+-----+
The Physical Plan of such query will look like this:
== Physical Plan ==
Limit 21
ConvertToSafe
LeftSemiJoinBNL Some(Contains(path#0, path#2))
ConvertToUnsafe
Scan PhysicalRDD[path#0,value#1]
TungstenProject [path#2]
Scan PhysicalRDD[path#2,value#3]
It will use the LeftSemiJoinBNL for the actual join operation, which should broadcast values internally. From more details refer to the actual implementation in Spark - LeftSemiJoinBNL.scala
P.S. I didn't quite understand the need for removing the last two characters, but if that's needed - it can be done, like #zero323 proposed (using regexp_extract).