We Keep Coding

How to create a Java UDF that performs an aggregate sum in Snowflake?

I'm dealing with a dataset that contains number that are in the 10*6 to 10**80 scale. The value column that holds this data is of string type.
One of the common queries performed is a sum across the value column, for 100,000+ rows. However, there are strict requirements on precision so the double conversion that Snowflake performs causes it to lose precision.
Is it possible to create a Java UDF that performs an aggregate sum in the select part of a query? I tried looking up the documentation and couldn't really find a good example. The closest thing I found was UDTFs but it doesn't look like I can call them in the select part of a query.
If not, if anyone knows of strategies to deal with datasets with numbers much larger than the database can handle while maintaining data accuracy, that would be really helpful too.

A UDTF is the right answer, until Snowflake supports the type of UDFs you are asking for.
This Java UDTF does the job:
create or replace function add_java_udtf(id string, x string)
returns table(id string, big_number string)
language java
handler='MyClass'
as
$$
import java.util.stream.Stream;
import java.math.BigDecimal;
class OutputRow {
public final String id;
public final String big_number;
public OutputRow(String id, String outputValue) {
this.id = id;
this.big_number = outputValue;
}
}
class MyClass {
private BigDecimal _x;
private String _id;
public MyClass() {
_x = new BigDecimal(0);
}
public static Class getOutputClass() {
return OutputRow.class;
}
public Stream<OutputRow> process(String id, String inputValue) {
_id = id;
_x = _x.add(new BigDecimal(inputValue));
// Return nothing until whole partition computed.
return Stream.empty();
}
public Stream<OutputRow> endPartition() {
return Stream.of(new OutputRow(_id, _x.toString()));
}
}
$$;
Sample use:
with data(id, n) as (select $1, $2::string from values ('a', 1),('b', 2),('a', 5))
select r.id, r.big_number
from data, table(add_java_udtf(id, n) over(partition by id)) r
;

change a value from the List using java8

I want a value get nullified before sending to UI.
public class MedicalInfoVO {
private Integer medicalDeoId;
private List<MedicalCoverageVO> medicalCoverages;
private List<MedicalSignsSymptomVO> medicalSignsSymptoms;
}
public class MedicalSignsSymptomVO {
private Integer medicalSignsSymptomId;
private Integer symptomId;
private String symptomType;
private Integer medicalInfoId;
private Integer diagnosisId;
private String diagnosisValue;
}
I need the medicalSignsSymptomId and medicalInfoId get be nullified before send back to the Service call.
medicalInfoVO = retrieveMedicalInfoDetails(authorizationId, claimId);
if(null ==medicalInfoVO || null == medicalInfoVO.getMedicalDeoId()){
medicalInfoVO = retrieveMedicalInfoDetails(authorizationId, null);
medicalInfoVO.setClaimId(claimId);
medicalInfoVO.setMedicalDeoId(null);
if(CollectionUtils.isNotEmpty(medicalInfoVO.getMedicalSignsSymptoms())) {
List<Integer> medicalSignsSymptomIdList = medicalInfoVO.getMedicalSignsSymptoms().stream()
.map(map -> map.getMedicalSignsSymptomId()).collect(Collectors.toList());
medicalSignsSymptomIdList.clear();
List<Integer> medicalInfoIdList = medicalInfoVO.getMedicalSignsSymptoms().stream()
.map(map -> map.getMedicalInfoId()).collect(Collectors.toList());
medicalInfoIdList.clear();
}
}
When I do it with stream functionality the object value is still not getting cleared.
Suppose the obejct MedicalSignsSymptomVO have medicalSignsSymptomId as 3,
I need assign null to it. same for medicalInfoId as well.
Can I know how this can be done using stream functionality, instead of using the foreach functionality.

Not sure what you want to do, but basically the following is "streamy:"
medicalInfoVO.getMedicalSignsSymptoms().forEach(mSSymptoms -> {
mSSymptoms.setMedicalSignsSymptomId(null);
mSSymptoms.setMedicalInfoId(null);
});
One could combine it as follows:
Set<Integer> medicalSignsSymptomIdList = medicalInfoVO.getMedicalSignsSymptoms().stream()
.map(mSSymptoms -> {
Integer id = mSSymptoms.getMedicalSignsSymptomId();
mSSymptoms.setMedicalSignsSymptomId(null);
mSSymptoms.setMedicalInfoId(null);
return id;
})
.collect(Collectors.toSet());
I used a set here, which seems more logical.
Not necessarily good style, to combine too many things in a stream, for better maintenance. Here: changing elements and return a new list.

If you are sending objects to UI, you might not need to set values to null manually.
If you are using Jackson, you can simply put JSONIgnore to ignore fields when object is being serialized/deserialized.
e.g.
#JsonIgnoreProperties(ignoreUnknown = true)
public class MedicalSignsSymptomVO {
#JSONIgnore
private Integer medicalSignsSymptomId;
private Integer symptomId;
private String symptomType;
private Integer medicalInfoId;
private Integer diagnosisId;
private String diagnosisValue;
}
In case you need those fields serialized else where then you can use forEach method as described by #Joop Eggen's answer.

local variable is not known within for loop in lambda java 8 [duplicate]

Modifying a local variable in forEach gives a compile error:
Normal
int ordinal = 0;
for (Example s : list) {
s.setOrdinal(ordinal);
ordinal++;
}
With Lambda
int ordinal = 0;
list.forEach(s -> {
s.setOrdinal(ordinal);
ordinal++;
});
Any idea how to resolve this?

Use a wrapper
Any kind of wrapper is good.
With Java 10+, use this construct as it's very easy to setup:
var wrapper = new Object(){ int ordinal = 0; };
list.forEach(s -> {
s.setOrdinal(wrapper.ordinal++);
});
With Java 8+, use either an AtomicInteger:
AtomicInteger ordinal = new AtomicInteger(0);
list.forEach(s -> {
s.setOrdinal(ordinal.getAndIncrement());
});
... or an array:
int[] ordinal = { 0 };
list.forEach(s -> {
s.setOrdinal(ordinal[0]++);
});
Note: be very careful if you use a parallel stream. You might not end up with the expected result. Other solutions like Stuart's might be more adapted for those cases.
For types other than int
Of course, this is still valid for types other than int.
For instance, with Java 10+:
var wrapper = new Object(){ String value = ""; };
list.forEach(s->{
wrapper.value += "blah";
});
Or if you're stuck with Java 8 or 9, use the same kind of construct as we did above, but with an AtomicReference...
AtomicReference<String> value = new AtomicReference<>("");
list.forEach(s -> {
value.set(value.get() + s);
});
... or an array:
String[] value = { "" };
list.forEach(s-> {
value[0] += s;
});

This is fairly close to an XY problem. That is, the question being asked is essentially how to mutate a captured local variable from a lambda. But the actual task at hand is how to number the elements of a list.
In my experience, upward of 80% of the time there is a question of how to mutate a captured local from within a lambda, there's a better way to proceed. Usually this involves reduction, but in this case the technique of running a stream over the list indexes applies well:
IntStream.range(0, list.size())
.forEach(i -> list.get(i).setOrdinal(i));

If you only need to pass the value from the outside into the lambda, and not get it out, you can do it with a regular anonymous class instead of a lambda:
list.forEach(new Consumer<Example>() {
int ordinal = 0;
public void accept(Example s) {
s.setOrdinal(ordinal);
ordinal++;
}
});

As the used variables from outside the lamda have to be (implicitly) final, you have to use something like AtomicInteger or write your own data structure.
See
https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html#accessing-local-variables.

An alternative to AtomicInteger is to use an array (or any other object able to store a value):
final int ordinal[] = new int[] { 0 };
list.forEach ( s -> s.setOrdinal ( ordinal[ 0 ]++ ) );
But see the Stuart's answer: there might be a better way to deal with your case.

Yes, you can modify local variables from inside lambdas (in the way shown by the other answers), but you should not do it. Lambdas have been made for functional style of programming and this means: No side effects. What you want to do is considered bad style. It is also dangerous in case of parallel streams.
You should either find a solution without side effects or use a traditional for loop.

If you are on Java 10, you can use var for that:
var ordinal = new Object() { int value; };
list.forEach(s -> {
s.setOrdinal(ordinal.value);
ordinal.value++;
});

You can wrap it up to workaround the compiler but please remember that side effects in lambdas are discouraged.
To quote the javadoc
Side-effects in behavioral parameters to stream operations are, in general, discouraged, as they can often lead to unwitting violations of the statelessness requirement
A small number of stream operations, such as forEach() and peek(), can operate only via side-effects; these should be used with care

I had a slightly different problem. Instead of incrementing a local variable in the forEach, I needed to assign an object to the local variable.
I solved this by defining a private inner domain class that wraps both the list I want to iterate over (countryList) and the output I hope to get from that list (foundCountry). Then using Java 8 "forEach", I iterate over the list field, and when the object I want is found, I assign that object to the output field. So this assigns a value to a field of the local variable, not changing the local variable itself. I believe that since the local variable itself is not changed, the compiler doesn't complain. I can then use the value that I captured in the output field, outside of the list.
Domain Object:
public class Country {
private int id;
private String countryName;
public Country(int id, String countryName){
this.id = id;
this.countryName = countryName;
}
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getCountryName() {
return countryName;
}
public void setCountryName(String countryName) {
this.countryName = countryName;
}
}
Wrapper object:
private class CountryFound{
private final List<Country> countryList;
private Country foundCountry;
public CountryFound(List<Country> countryList, Country foundCountry){
this.countryList = countryList;
this.foundCountry = foundCountry;
}
public List<Country> getCountryList() {
return countryList;
}
public void setCountryList(List<Country> countryList) {
this.countryList = countryList;
}
public Country getFoundCountry() {
return foundCountry;
}
public void setFoundCountry(Country foundCountry) {
this.foundCountry = foundCountry;
}
}
Iterate operation:
int id = 5;
CountryFound countryFound = new CountryFound(countryList, null);
countryFound.getCountryList().forEach(c -> {
if(c.getId() == id){
countryFound.setFoundCountry(c);
}
});
System.out.println("Country found: " + countryFound.getFoundCountry().getCountryName());
You could remove the wrapper class method "setCountryList()" and make the field "countryList" final, but I did not get compilation errors leaving these details as-is.

To have a more general solution, you can write a generic Wrapper class:
public static class Wrapper<T> {
public T obj;
public Wrapper(T obj) { this.obj = obj; }
}
...
Wrapper<Integer> w = new Wrapper<>(0);
this.forEach(s -> {
s.setOrdinal(w.obj);
w.obj++;
});
(this is a variant of the solution given by Almir Campos).
In the specific case this is not a good solution, as Integer is worse than int for your purpose, anyway this solution is more general I think.

How does 'unique' work in the TotallyLazy library

How can I use the 'unique' method in the TotallyLazy library for Java and Objective-C?
I have the following code, but I cannot complete it because I'm not sure how the Callable1 instance passed into 'unique' should be composed. Here's what I have so far:
seq
.sort(new Comparator<T>() {
#Override
public int compare(
final T pt1,
final T pt2
) {
return pt1.compareTo(pt2);
}
})
.unique(new Callable1<T,T>() {
#Override
public T call(final T pt) throws Exception {
final int result = pt.compareTo(..?);
return result != 0;
}});
As you can see, I can sort successfully, but when it comes to "result = ..." in the 'unique' call, what should I put?

Firstly for sorting you should probably use one of the provided Comparators:
seq.sort(Comparators.<T>ascending());
Then you could just unique with no arguments if uniqueness is based on equality of T as a whole. The overload for unique allows one to get a unique sequence based on some property of T.
For example if you had a Sequence<User> you could say I want them to be unique based on their firstName field:
class User {
String firstName, lastName;
User(...){...} // Constructor
}
sequence(new User("Chris", "Nash")).unique(user -> user.firstName)
TotallyLazy has extensive Tests that document the features but the first place to start would always be the SequenceTest or have a look in the whole test package for more examples.

I am making a safe, compile-time String.format(...) equivalent. An issue still persist

Most people understand the innate benefits that enum brings into a program verses the use of int or String. See here and here if you don't know. Anyway, I came across a problem that I wanted to solve that kind of is on the same playing field as using int or String to represent a constant instead of using an enum. This deals specifically with String.format(...).
With String.format, there seems to be a large opening for programmatic error that isn't found at compile-time. This can make fixing errors more complex and / or take longer.
This was the issue for me that I set out to fix (or hack a solution). I came close, but I am not close enough. For this problem, this is more certainly over-engineered. I understand that, but I just want to find a good compile-time solution to this, that provides the least amount of boiler-plate code.
I was writing some non-production code just to write code with the following rules.
Abstraction was key.
Readability was very important
Yet the simplest way to the above was preferred.
I am running on...
Java 7 / JDK 1.7
Android Studio 0.8.2
These are unsatisfactory
Is there a typesafe alternative to String.format(...)
How to get string.format to complain at compile time
My Solution
My solution uses the same idea that enums do. You should use enum types any time you need to represent a fixed set of constants...data sets where you know all possible values at compile time(docs.oracle.com). The first argument in String.format seems to fit that bill. You know the whole string beforehand, and you can split it up into several parts (or just one), so it can be represented as a fixed set of "constants".
By the way, my project is a simple calculator that you probably seen online already - 2 input numbers, 1 result, and 4 buttons (+, -, ×, and ÷). I also have a second duplicate calculator that has only 1 input number, but everything else is the same
Enum - Expression.java & DogeExpression.java
public enum Expression implements IExpression {
Number1 ("%s"),
Operator (" %s "),
Number2 ("%s"),
Result (" = %s");
protected String defaultFormat;
protected String updatedString = "";
private Expression(String format) { this.defaultFormat = format; }
// I think implementing this in ever enum is a necessary evil. Could use a switch statement instead. But it would be nice to have a default update method that you could overload if needed. Just wish the variables could be hidden.
public <T> boolean update(T value) {
String replaceValue
= this.equals(Expression.Operator)
? value.toString()
: Number.parse(value.toString()).toString();
this.updatedString = this.defaultFormat.replace("%s", replaceValue);
return true;
}
}
...and...
public enum DogeExpression implements IExpression {
Total ("Wow. Such Calculation. %s");
// Same general code as public enum Expression
}
Current Issue
IExpression.java - This is a HUGE issue. Without this fixed, my solution cannot work!!
public interface IExpression {
public <T> boolean update(T Value);
class Update { // I cannot have static methods in interfaces in Java 7. Workaround
public static String print() {
String replacedString = "";
// for (Expression expression : Expression.values()) { // ISSUE!! Switch to this for Expression
for (DogeExpression expression : DogeExpression.values()) {
replacedString += expression.updatedString;
}
return replacedString;
}
}
}
So Why Is This An Issues
With IExpression.java, this had to hacked to work with Java 7. I feel that Java 8 would have played a lot nicer with me. However, the issue I am having is paramount to getting my current implementation working The issue is that IExpression does not know which enum to iterate through. So I have to comment / uncomment code to get it to work now.
How can I fix the above issue??

How about something like this:
public enum Operator {
addition("+"),
subtraction("-"),
multiplication("x"),
division("÷");
private final String expressed;
private Operator(String expressed) { this.expressed = expressed; }
public String expressedAs() { return this.expressed; }
}
public class ExpressionBuilder {
private Number n1;
private Number n2;
private Operator o1;
private Number r;
public void setN1(Number n1) { this.n1 = n1; }
public void setN2(Number n2) { this.n2 = n2; }
public void setO1(Operator o1) { this.o1 = o1; }
public void setR(Number r) { this.r = r; }
public String build() {
final StringBuilder sb = new StringBuilder();
sb.append(format(n1));
sb.append(o1.expressedAs());
sb.append(format(n2));
sb.append(" = ");
sb.append(format(r));
return sb.toString();
}
private String format(Number n) {
return n.toString(); // Could use java.text.NumberFormat
}
}