I have found a solution to my problem here Create new column with function in Spark Dataframe
But i am having difficulty in converting the below code to Java since it's in Scala
import org.apache.spark.sql.functions._
val myDF = sqlContext.parquetFile("hdfs:/to/my/file.parquet")
val coder: (Int => String) = (arg: Int) => {if (arg < 100) "little" else "big"}
val sqlfunc = udf(coder)
myDF.withColumn("Code", sqlfunc(col("Amt")))
Can someone provide me the Java equivalent code for this?. I am stuck in converting below 2 lines
val coder: (Int => String) = (arg: Int) => {if (arg < 100) "little" else "big"}
val sqlfunc = udf(coder)
Thanks,
Create your User Defined Function:
public class CodeUdf implements UDF1<Integer, String>{
#Override
public String call(Integer integer) throws Exception {
if(integer < 100)
return "little";
else
return"big";
}
}
Tell Spark about it
sqlContext.udf().register("Code", new CodeUdf(), DataTypes.IntegerType);
Use it in a select.
df.selectExpr("value", "Code(value)").show();
import org.apache.spark.sql.functions._
val myDF = sqlContext.parquetFile("hdfs:/to/my/file.parquet")
//val coder: (Int => String) = (arg: Int) => {if (arg < 100) "little" else "big"}
//val sqlfunc = udf(coder)
myDF.selectExpr("Code", "case when Amt < 100 'little' else 'big' end ")
Related
Please i need help to optimize the input filter on my customEditText
Requirement:
min float = 0.001
max float = 1000.000
decimal number = 3
I want to avoid 0 from the user set but he can enter something like 0.01 , 0.001 , 1.555 and 1000.000
This is what i've tested so far:
I used this function to set the min and max
class EditTextInputFilter(min: Float, max: Float) : InputFilter {
private val min: Float = min.coerceAtMost(max)
private val max: Float = min.coerceAtLeast(max)
override fun filter(source: CharSequence, i: Int, i2: Int, spanned: Spanned, i3: Int, i4: Int): CharSequence? {
try {
val input = (spanned.toString() + source.toString()).toFloatOrZero()
if (isInRange(min, max, input)) {
return null
}
} catch (nfe: NumberFormatException) {
Logger.error(nfe.localizedMessage!!)
}
return ""
}
private fun isInRange(min: Float, max: Float, value: Float): Boolean {
return value in min..max
}
}
I used this Decimal filter to remove comment and un-useful dots
class DecimalFilter(private val decimalDigits: Int) : InputFilter {
override fun filter(source: CharSequence, i: Int, i2: Int, spanned: Spanned, i3: Int, i4: Int): CharSequence? {
var dotPos = -1
val len = spanned.length
for (decimalsI in 0 until len) {
val c = spanned[decimalsI]
if (c == '.' || c == ',') {
dotPos = decimalsI
break
}
}
if (dotPos >= 0) {
// protects against many dots
if (source == "." || source == ",") return ""
// if the text is entered before the dot
if (i4 <= dotPos) return null
if (len - dotPos > decimalDigits) return ""
}
return null
}
}
class CustomEditText : TextInputEditText {
private var decimals = 3
private var min = 0.001f
private var max = 1000f
private lateinit var oldFilters: MutableList<InputFilter>
constructor(context: Context?) : super(context!!) {
init(null)
}
constructor(context: Context?, attrs: AttributeSet?) : super(context!!, attrs) {
init(attrs)
}
constructor(context: Context?, attrs: AttributeSet?, defStyle: Int) : super(context!!, attrs, defStyle) {
init(attrs)
}
private fun init(attrs: AttributeSet?) {
isInEditMode
if (attrs != null) {
val a = context.theme.obtainStyledAttributes(attrs, R.styleable.CustomEditText, 0, 0)
decimals = a.getInt(R.styleable.CustomEditText_decimals, decimals)
min = a.getFloat(R.styleable.CustomEditText_min, min)
max = a.getFloat(R.styleable.CustomEditText_max, max)
}
oldFilters = filters.toMutableList()
addFilters()
}
private fun addFilters() {
val inputFilters = ArrayList<InputFilter>(oldFilters.size + 1)
inputFilters.addAll(oldFilters)
inputFilters.add(EditTextInputFilter(min, max))
inputFilters.add(DecimalFilter(decimals))
filters = inputFilters.toTypedArray()
}
fun setMin(min: Float) {
this.min = min
addFilters()
}
fun setMax(max: Float) {
this.max = max
addFilters()
}
fun setDecimals(decimals: Int) {
this.decimals = decimals
addFilters()
}
}
My problem now is, i can not enter 0
The problem is that you're validating the field as the user types and since the range of allowed values is 0.001..1000.000 you cannot type the number 0 (even if the intention is to enter 0.1).
This is not a coding but a specification problem.
The way it's specified is:
don't accept values outside 0.001..1000.000
validate as the user types
It's impossible to meet these requirements. I see a couple of solutions:
validate once the user submits (not as the user types)
validate without blocking the user input (by using setError in a TextWatcher)
allow 0 values in your filter (range 0..1000.000) but add another validation when the user submits to eliminate the 0
allow 0 values in your filter (range 0..1000.000) and disable the submit button (or whatever you have to process the input) if the value isn't in range
From a ux perspective I'd vote for the last option and maybe also indicate to the user what values are allowed so they know why they can't continue
#EmanuelMoecklin After reading your proposal answer i reviewed my code with this code below and it works perfectly and also i can re-use it for other input field.
I use my EditTextInputFilter class to set the min and max. The DecimalFilter class to set the number of decimal
class EditTextInputFilter(min: Float, max: Float) : InputFilter {
private val min: Float = min.coerceAtMost(max)
private val max: Float = min.coerceAtLeast(max)
override fun filter(source: CharSequence, i: Int, i2: Int, spanned: Spanned, i3: Int, i4: Int): CharSequence? {
try {
val input = (spanned.toString() + source.toString()).toFloatOrZero()
if (isInRange(min, max, input)) {
return null
}
} catch (nfe: NumberFormatException) {
Logger.error(nfe.localizedMessage!!)
}
return ""
}
private fun isInRange(min: Float, max: Float, value: Float): Boolean {
return value in min..max
}
}
class DecimalFilter(private val decimalDigits: Int) : InputFilter {
override fun filter(source: CharSequence, i: Int, i2: Int, spanned: Spanned, i3: Int, i4: Int): CharSequence? {
var dotPos = -1
val len = spanned.length
for (decimalsI in 0 until len) {
val c = spanned[decimalsI]
if (c == '.' || c == ',') {
dotPos = decimalsI
break
}
}
if (dotPos >= 0) {
// protects against many dots
if (source == "." || source == ",") return ""
// if the text is entered before the dot
if (i4 <= dotPos) return null
if (len - dotPos > decimalDigits) return ""
}
return null
}
}
I've created another validate function :
private fun validateInput(input: String): Boolean {
val double = input.toDoubleOrNull()
return double != null && double in 0.001f...1000f
}
Than i set the input filter like this on my input field:
binding.valueTiet.filters = arrayOf(EditTextInputFilter(0f, 1000f), DecimalFilter(3))
binding.valueTiet.doAfterTextChanged {
if (validateInput(it)) {
binding.valueTietTil.error = ""
Logger.info("Value accepted")
} else {
binding.valueTietTil.error = "Allowed range [0.001-1000]"
}
}
How can I port https://github.com/davidmoten/rtree2/blob/master/src/test/java/com/github/davidmoten/rtree2/LatLongExampleTest.java#L55
Iterables.filter(tree
// do the first search using the bounds
.search(bounds),
// refine using the exact distance
entry -> {
Point p = entry.geometry();
Position position = Position.create(p.y(), p.x());
return from.getDistanceToKm(position) < distanceKm;
});
from Java to scala? My approach below fails:
import com.github.davidmoten.grumpy.core.Position
import com.github.davidmoten.rtree2.{Iterables, RTree}
import com.github.davidmoten.rtree2.geometry.{Geometries, Point}
val sydney = Geometries.point(151.2094, -33.86)
val canberra = Geometries.point(149.1244, -35.3075)
val brisbane = Geometries.point(153.0278, -27.4679)
val bungendore = Geometries.point(149.4500, -35.2500)
var tree = RTree.star.create[String, Point]
tree = tree.add("Sydney", sydney)
tree = tree.add("Brisbane", brisbane)
val distanceKm = 300
val list = Iterables.toList(search(tree, canberra, distanceKm))
def createBounds(from: Position, distanceKm: Double) = { // this calculates a pretty accurate bounding box. Depending on the
// performance you require you wouldn't have to be this accurate because
// accuracy is enforced later
val north = from.predict(distanceKm, 0)
val south = from.predict(distanceKm, 180)
val east = from.predict(distanceKm, 90)
val west = from.predict(distanceKm, 270)
Geometries.rectangle(west.getLon, south.getLat, east.getLon, north.getLat)
}
import com.github.davidmoten.grumpy.core.Position
import com.github.davidmoten.rtree2.RTree
def search[T](tree: RTree[String, Point], lonLat: Point, distanceKm: Double) = { // First we need to calculate an enclosing lat long rectangle for this
// distance then we refine on the exact distance
val from = Position.create(lonLat.y, lonLat.x)
val bounds = createBounds(from, distanceKm)
Iterables.filter(tree.search // do the first search using the bounds
(bounds), // refine using the exact distance
(entry) => {
def foo(entry) = {
val p = entry.geometry
val position = Position.create(p.y, p.x)
from.getDistanceToKm(position) < distanceKm
}
foo(entry)
})
}
as the type of entry does not seem to be well defined.
tree has type RTree[String, Point] so T=String, S=Point. So tree.search(bounds) has type Iterable[Entry[String, Point]]. So entry has type Entry[String, Point].
Try
(entry: Entry[String,Point]) => {
def foo(entry: Entry[String,Point]) = {
val p = entry.geometry
val position = Position.create(p.y, p.x)
from.getDistanceToKm(position) < distanceKm
}
foo(entry)
})
Tested in Scala 2.13.0, rtree2 0.9-RC1, grumpy-core 0.2.4.
In 2.11 this should be just
import scala.compat.java8.FunctionConverters._
((entry: Entry[String,Point]) => {
def foo(entry: Entry[String,Point]) = {
val p = entry.geometry
val position = Position.create(p.y, p.x)
from.getDistanceToKm(position) < distanceKm
}
foo(entry)
}).asJava
libraryDependencies += "org.scala-lang.modules" %% "scala-java8-compat" % "0.9.0"
I want to sort the objects of class A based on the values of members: a, b ,c: A is given the most preference -> b is the next -> c has the least preference.
Comparator is used by:
x = x.sortWith(comparator)
here, x is a ArrayBuffer[A]
class A {var a,b,c = 0}
def comparator(f1:A, f2:A) = {
if(f1.a == f2.a) {
if(f1.b == f2.b) f1.c > f2.c
else f1.b > f2.b
}
else f1.a > f2.a
}
Using this I am getting:
**java.lang.IllegalArgumentException: Comparison method violates its general contract!**
at java.util.TimSort.mergeLo(TimSort.java:747)
at java.util.TimSort.mergeAt(TimSort.java:483)
at java.util.TimSort.mergeCollapse(TimSort.java:410)
at java.util.TimSort.sort(TimSort.java:214)
at java.util.TimSort.sort(TimSort.java:173)
at java.util.Arrays.sort(Arrays.java:659)
at scala.collection.SeqLike$class.sorted(SeqLike.scala:618)
at scala.collection.AbstractSeq.sorted(Seq.scala:41)
at scala.collection.SeqLike$class.sortWith(SeqLike.scala:575)
at scala.collection.AbstractSeq.sortWith(Seq.scala:41)
This is much easier to do with sortBy and a tuple:
x sortBy {f => (f.a, f.b, f.c)}
Can you provide a more comprehensive test case? This example ran fine for me:
class A {var a,b,c = 0}
def comparator(f1:A, f2:A) = {
if(f1.a == f2.a) {
if(f1.b == f2.b) f1.c > f2.c
else f1.b > f2.b
}
else f1.a > f2.a
}
val testList = new scala.collection.mutable.ArrayBuffer[A]
val val1 = new A()
val val2 = new A()
val val3 = new A()
val1.a=1
val1.b=1
val1.c=1
val2.a = 1
val2.b = 1
val2.c = 1
val3.a = 2
val3.b = 2
val3.c = 3
testList += val1
testList += val2
testList += val3
testList.sortWith(comparator)
I want to implement a Repeat Until in Scala.
The commented code will throw a StackOverflowException.
But the code now run in main method works.
But I don't know why and what's the difference between these two snippets.
object REPEAT{
class REPEAT(command: => Unit){
def UNTIL(cond: => Boolean):Unit = {
command
if(!cond) UNTIL(cond)
}
}
def REPEAT(command: => Unit): REPEAT = {
new REPEAT(command)
}
def main(args: Array[String]) {
/* this code work well */
var i = 0
REPEAT { i+= 1; println(i)} UNTIL (i == 100)
/*
// something wrong in this code.
var i = 0
new REPEAT {
i += 1; println(i)
} UNTIL (i == 100)
*/
}
}
ps: I found the wrong code just change the var i to 1 but don't change it anymore.
The wrong code's output:
1
Exception in thread "main" java.lang.StackOverflowError at
REPEAT$$anon$1$$anonfun$$lessinit$greater$1.apply$mcV$sp(REPEAT.scala:15)
at REPEAT$REPEAT.UNTIL(REPEAT.scala:4)
That's the problem:
new REPEAT {
i += 1; println(i)
} UNTIL (i == 100)
is equivalent to:
new REPEAT() {
i += 1; println(i)
} UNTIL (i == 100)
So you're actually creating new instance of REPEAT class with command = (), and i += 1; println(i) in constructor - so i++ repeated only once and () repeated infinitely.
The right way to call it:
new REPEAT({
i += 1; println(i)
}) UNTIL (i == 100)
The reason why is you don't have to specify single parameter with Unit type explicitly in scala:
scala> class Aaaa(a: Unit)
defined class Aaaa
scala> new Aaaa //parenthesis is not required for 'new'
res10: Aaaa = Aaaa#17de44d4
scala> def aaaa(a: Unit) = 5
aaaa: (a: Unit)Int
scala> aaaa()
res13: Int = 5
I have a code to validate the date in the form of MMYY format.This shows dynamic popup dialog box before submitting the button. For this the code is given below as:
<ext:text name="caseDetailForm" property="phaseData.phaseDateExpiration" styleId="_dateExp"
style="width:100px" maxlength="4" titleKey="prompt.exp.date" onkeyup="validateCardExpiry(this);" />
and the function is:
function validateCardExpiry(field) {
var cardExpRegEx1or2char = /^(0[1-9]?|1[0-2]?)$/;
var cardExpRegEx3or4char = /^(0[1-9]?|1[0-2])[0-9]?[0-9]?$/;
var cardExpDate = field.value;
var msg = '<bean:message key="errors.bad.input.characters.detected"/> - '+cardExpDate;
+ '\n' + "<bean:message key="prompt.exp.date"/>";
var failed = false;
for (var x = cardExpDate.length; x >= 0; x--) {
cardExpDate = cardExpDate.substring(0, x);
if (cardExpDate.length > 0 &&
((cardExpDate.length <= 2 && !cardExpRegEx1or2char.test(cardExpDate))
|| (cardExpDate.length > 2 && !cardExpRegEx3or4char.test(cardExpDate)))) {
failed = true;
}
else {
field.value = cardExpDate;
break;
}
}
if (failed) {
alert(msg);
}
}
for MMYY validation is working fine with above code. But i need MM/DD/YYYY validation. For this how can i validate with popup window in the same scenario.
You can do something like:
function validateDate(s) {
var bits = s.split('/');
var d = new Date(bits[2], ++bits[0], bits[1]);
return d.getFullYear == bits[2] && d.getDate == bits[1];
}
but the format mm/dd/yyyy is not used by that many people, most use either dd/mm/yyyy or yyyy-mm-dd.