We Keep Coding

Scala - how to format a collection into a String?

I'm trying to parse Metrics data into a formatted String so that there is a header and each record below starts from a new line. Initially I wanted to get something close to a table formatting like this:
Id | Name | Rate | Value
1L | Name1 | 1 | value_1
2L | Name2 | 2 | value_2
3L | Name3 | 3 | value_3
But my current implementation results in the following Error:
java.util.MissingFormatArgumentException: Format specifier '%-70s'
What should I change in my code to get it formatted correctly?
import spark.implicits._
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
case class BaseMetric(val id: Long,
val name: String,
val rate: String,
val value: String,
val count: Long,
val isValid: Boolean
) {
def makeCustomMetric: String = Seq(id, name, rate, value).mkString("\t")
}
val metric1 = new BaseMetric(1L, "Name1", "1", "value_1", 10L, true)
val metric2 = new BaseMetric(2L, "Name2", "2", "value_2", 20L, false)
val metric3 = new BaseMetric(3L, "Name3", "3", "value_3", 30L, true)
val metrics = Seq(metric1, metric1, metric1)
def formatMetrics(metrics: Seq[BaseMetric]): String = {
val pattern = "%-50s | %-70s | %-55s | %-65s | %f"
val formattedMetrics: String = pattern.format(metrics.map(_.makeCustomMetric))
.mkString("Id | Name | Rate | Value\n", "\n", "\nId | Name | Rate | Value")
formattedMetrics
}
val metricsString = formatMetrics(metrics)

The specific error is due to the fact that you pass a Seq[String] to format which expects Any*. You only pass one parameter instead of five. The error says it doesn't find an argument for your second format string.
You want to apply the pattern on every metric, not all the metrics on the pattern.
The paddings in the format string are too big for what you want to achieve.
val pattern = "%-2s | %-5s | %-4s | %-6s"
metrics.map(m => pattern.format(m.makeCustomMetric: _*))
.mkString("Id | Name | Rate | Value\n", "\n", "\nId | Name | Rate | Value")
The _* tells the compiler that you want to pass a list as variable length argument.
makeCustomMetric should return only the List then, instead of a string.
def makeCustomMetric: String = Seq(id, name, rate, value)

Scala string interpolation is the optimized way to concat/foramt strings.
Reference: https://docs.scala-lang.org/overviews/core/string-interpolation.html
s"id: $id ,name: $name ,rate: $rate ,value: $value ,count: $count, isValid: $isValid"

Spark2 dataframe get difference of two tables, with some exclusion

Lets say I have two tables, tableA and tableB, they have the same schema. Now Id like to get the mismatch of the two tables with the same primary key but with some exclusion of some column values.
So, if either table ref column contains xx, we consider this would match with the other table column value. Can anyone help me with this in java? I had a hard time reading scala code
tableA
+--+------+------+------+
|id| name | type| ref |
+--+------+------+------+
| 1|aaa | a| a1|
| 2|bbb | b| xx|
| 3|ccc | c| c3|
| 4|ddd | d| d4|
| 6|fff | f| f6|
| 7|ggg | 0| g7|
+--+------+------+------+
tableB
+--+------+------+------+
|id| name | type| ref |
+--+------+------+------+
| 1|aaa | a| a1|
| 2|bbb | b| b2|
| 3|ccc | c| xx|
| 5|eee | e| e5|
| 6|fff | f| f66|
| 7|ggg | g| g7|
+--+------+------+------+
Expected results:
+--+------+------+-------------+
|id| name | type| ref |
+--+------+------+-------------+
| 6|fff | f| [f6 ->f66]|
| 7|ggg | 0| [0 -> g7 ]|
+--+------+------+-------------+
This seems working fine, but I dont have strong confidence of it.
Dataset<Row> join = data1.join(data2, data1.col("id").equalTo(data2.col("id"))
.and(data1.col("name").equalTo(data2.col("name")))
.and(data1.col("type").equalTo(data2.col("type"))
.and(data1.col("ref").equalTo(data2.col("ref"))
.or(data1.col("ref").equalTo(lit("xx")))
.or(data2.col("ref").equalTo(lit("xx"))))), "left_semi");

you can simply use UDF (User Defined Functions) to achieve this once you have joined your two dataframes like below:
import sparkSession.sqlContext.implicits._
val df1 = Seq((1, "aaa", "a", "a1"), (2, "bbb", "b", "xx"), (3, "ccc", "c", "c3"), (4, "ddd", "d", "d4"), (6, "fff", "f", "f6")).toDF("id", "name", "type", "ref")
val df2 = Seq((1, "aaa", "a", "a1"), (2, "bbb", "b", "b2"), (3, "ccc", "c", "xx"), (4, "ddd", "d", "d4"), (6, "fff", "f", "f66")).toDF("id", "name", "type", "ref")
val diffCondition: UserDefinedFunction = udf {
(ref1: String, ref2: String) => {
var result: String = null
if (!ref1.equals(ref2) && !"xx".equals(ref1) && !"xx".equals(ref2)) {
result = s"$ref1 -> $ref2"
}
result
}
}
df1.join(df2, Seq("id", "name", "type"))
.withColumn("difference", diffCondition(df1("ref"), df2("ref")))
.filter("difference is not null")
.show()
and the output is
+---+----+----+---+---+----------+
| id|name|type|ref|ref|difference|
+---+----+----+---+---+----------+
| 6| fff| f| f6|f66| f6 -> f66|
+---+----+----+---+---+----------+

MySQL query to fetch list of data using logical operations

The following are the list of different kinds of books that customers read in a library. The values are stored with the power of 2 in a column called bookType.
I need to fetch list of books with the combinations of persons who read
only Novel Or only Fairytale Or only BedTime Or both Novel + Fairytale
from the database with logical operational query.
Fetch list for the following combinations :
person who reads only novel(Stored in DB as 1)
person who reads both novel and fairy tale(Stored in DB as 1+2 = 3)
person who reads all the three i.e {novel + fairy tale + bed time} (stored in DB as 1+2+4 = 7)
The count of these are stored in the database in a column called BookType(marked with red in fig.)
How can I fetch the above list using MySQL query
From the example, I need to fetch users like novel readers (1,3,5,7).

The heart of this question is conversion of decimal to binary and mysql has a function to do just - CONV(num , from_base , to_base );
In this case from_base would be 10 and to_base would be 2.
I would wrap this in a UDF
So given
MariaDB [sandbox]> select id,username
-> from users
-> where id < 8;
+----+----------+
| id | username |
+----+----------+
| 1 | John |
| 2 | Jane |
| 3 | Ali |
| 6 | Bruce |
| 7 | Martha |
+----+----------+
5 rows in set (0.00 sec)
MariaDB [sandbox]> select * from t;
+------+------------+
| id | type |
+------+------------+
| 1 | novel |
| 2 | fairy Tale |
| 3 | bedtime |
+------+------------+
3 rows in set (0.00 sec)
This UDF
drop function if exists book_type;
delimiter //
CREATE DEFINER=`root`#`localhost` FUNCTION `book_type`(
`indec` int
)
RETURNS varchar(255) CHARSET latin1
LANGUAGE SQL
NOT DETERMINISTIC
CONTAINS SQL
SQL SECURITY DEFINER
COMMENT ''
begin
declare tempstring varchar(100);
declare outstring varchar(100);
declare book_types varchar(100);
declare bin_position int;
declare str_length int;
declare checkit int;
set tempstring = reverse(lpad(conv(indec,10,2),4,0));
set str_length = length(tempstring);
set checkit = 0;
set bin_position = 0;
set book_types = '';
looper: while bin_position < str_length do
set bin_position = bin_position + 1;
set outstring = substr(tempstring,bin_position,1);
if outstring = 1 then
set book_types = concat(book_types,(select trim(type) from t where id = bin_position),',');
end if;
end while;
set outstring = book_types;
return outstring;
end //
delimiter ;
Results in
+----+----------+---------------------------+
| id | username | book_type(id) |
+----+----------+---------------------------+
| 1 | John | novel, |
| 2 | Jane | fairy Tale, |
| 3 | Ali | novel,fairy Tale, |
| 6 | Bruce | fairy Tale,bedtime, |
| 7 | Martha | novel,fairy Tale,bedtime, |
+----+----------+---------------------------+
5 rows in set (0.00 sec)
Note the loop in the UDF to walk through the binary string and that the position of the 1's relate to the ids in the look up table;
I leave it to you to code for errors and tidy up.

Match text between empty lines

I have a text in following blocks:
AAAAAAA
BBBBBBB
CCCCCCC
DDDDDD. YYYYYYYYYYYYYYYYYYYYYY
EEEEE 1234567890
Some random text
Some text random
Random text
Text
Some random text
ZZZZZZZZZZZZZZZZ
UUUUUUUUUUUUUUUU
How to select with regexp a following block?
Some random text
Some text random
Random text
Text
Some random text
From the original text I know that this block goes after line DDDDDD. YYYYYYYYYYYYYYYYYYYYYY which is optionally followed by line EEEEE 1234567890 and also that block is between lines that contain only \s symbols.
I have tried pattern DDDDDD.*\\s+(.*)\\s+ it doesn't work.

You can use the following Pattern to match your expected text:
String text = "AAAAAAA\nBBBBBBB\nCCCCCCC\n\nDDDDDD. YYYYYYYYYYYYYYYYYYYYYY "
+ "\nEEEEE 1234567890 "
+ "\n\nSome random text\nSome text random\nRandom text\nText \nSome random text\n\n"
+ "ZZZZZZZZZZZZZZZZ\nUUUUUUUUUUUUUUUU";
Pattern p = Pattern.compile(
// | 6 "D"s
// | | actual dot
// | | | some whitespace
// | | | | 22 "Y"s
// | | | | | more whitespace
// | | | | | | optional:
// | | | | | || 5 "E"s
// | | | | | || | whitespace
// | | | | | || | | 10 digits
// | | | | | || | | | more whitespace including line breaks
// | | | | | || | | | | your text
// | | | | | || | | | | | followed by any "Z" sequence
"D{6}\\.\\s+Y{22}\\s+(E{5}\\s\\d{10}\\s+)?(.+?)(?=Z+)",
Pattern.DOTALL
);
Matcher m = p.matcher(text);
if (m.find()) {
System.out.println(m.group(2));
}
Output
Some random text
Some text random
Random text
Text
Some random text
Note
Not sure how to delimit the final part, so I just used a capitalized Z sequence (1+).
Up to you to refine.

Regular Expression in Java for UMASK

I need a Java regular expression to get the following two values:
the value of the UMASK parameter in file /etc/default/security should be set to 077. [Current value: 022] [AA.1.9.3]
the value of UMASK should be set to 077 in /etc/skel/.profile [AA.1.9.3]
I need to get the file name from the input string, as well as the current value if existing.
I wrote a regex as .* (.*?/.*?) (?:\\[Current value\\: (\\d+)\\])?.* for this one, it can match both strings, also to get the file name, but can NOT get the current value.
Then another regex: .* (.*?/.*?) (?:\\[Current value\\: (\\d+)\\])? .* comparing with the first one, there is a space before the last .* for this one, it can match the string 1, and get file name and current value, but it can NOT match the string 2...
What how can I correct these regular expressions to obtain the values described above?

If I understand your requisites correctly (file name and current octal permissions value), you can use the following Pattern:
String input =
"Value for parameter UMASK in file /etc/default/security should be set to 077. " +
"[Current value: 022] [AA.1.9.3] - " +
"Value of UMASK should be set to 077 in /etc/skel/.profile [AA.1.9.3]";
// | "file " marker
// | | group 1: the file path
// | | | space after
// | | || any characters
// | | || | escaped square bracket
// | | || | | "Current value: " marker
// | | || | | | group 2:
// | | || | | | digits for value
// | | || | | | | closing bracket
Pattern p = Pattern.compile("file (.+?) .+?\\[Current value: (\\d+)\\]");
Matcher m = p.matcher(input);
// iterates, but will find only once in this instance (which is desirable)
while (m.find()) {
System.out.printf("File: %s%nCurrent value: %s%n", m.group(1), m.group(2));
}
Output
File: /etc/default/security
Current value: 022