From reading a Google groups post from 2016 : “.map() is converted to a .via()”
src : https://groups.google.com/g/akka-user/c/EzHygZpcCHg
Are the following lines of code equivalent :
Source.repeat(json).take(3).via(mapToDtoFlow).to(printSink).run(actorSystem);
Source.repeat(json).take(3).map(x -> mapper.readValue(x, RequestDto.class)).to(printSink).run(actorSystem);
Are there scenarios when a map should be used instead of flow?
src :
RequestDTO :
import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.Builder;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
import lombok.extern.jackson.Jacksonized;
import java.util.Date;
#Getter
#Setter
#Builder
#ToString
#Jacksonized
public class RequestDto {
#JsonFormat(pattern = "yyyy-MM-dd HH:mm:sss")
private final Date datePurchased;
}
StreamManager (contains main method) :
import akka.Done;
import akka.NotUsed;
import akka.actor.typed.ActorSystem;
import akka.actor.typed.javadsl.Behaviors;
import akka.stream.javadsl.Flow;
import akka.stream.javadsl.Sink;
import akka.stream.javadsl.Source;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.concurrent.CompletionStage;
public class StreamManager {
final static ObjectMapper mapper = new ObjectMapper();
private static final Flow<String, RequestDto, NotUsed> mapToDtoFlow = Flow.of(String.class)
.map(input -> mapper.readValue(input, RequestDto.class))
.log("error");
public static void main(String args[]) {
final ActorSystem actorSystem = ActorSystem.create(Behaviors.empty(), "actorSystem");
final Sink<RequestDto, CompletionStage<Done>> printSink = Sink.foreach(System.out::println);
final String json = "{\"datePurchased\":\"2022-03-03 21:32:017\"}";
Source.repeat(json).take(3).via(mapToDtoFlow).to(printSink).run(actorSystem);
Source.repeat(json).take(3).map(x -> mapper.readValue(x, RequestDto.class)).to(printSink).run(actorSystem);
}
}
map is converted to a via, but it's not an exactly syntactically equivalent via as you'd get from Flow.of().map().
The first would translate to a .via(Map(f)), where Map is a GraphStage which implements the map operation.
In the second case, the mapToDtoFlow (ignoring the log) would itself be (in Scala notation) Flow[String].via(Map(f)) so you'd be adding another layer of via: .via(Flow[String].via(Map(f))).
For all intents and purposes, they're the same (I suspect that the materializer, when it comes time to interpret the RunnableGraph you've built, will treat them identically).
Taking the .log into account, mapToDtoFlow is equivalent (again in Scala):
Flow[String]
.via(Map(f))
.via(Log(...))
There are basically three levels of defining streams in Akka Streams, from highest level to lowest level:
the Java/Scala DSLs
the Java/Scala Graph DSLs
GraphStages
The DSLs merely specify succinct ways of building GraphStages and the fundamental way to link GraphStages with Flow shape is through the via operation.
Related
I have a project with 3 files in Quarkus
application.properties
conf.obj[0].name=name0
conf.obj[0].code=code0
conf.obj[0].versions[0].number=1
conf.obj[1].name=name1
conf.obj[1].code=code1
conf.obj[1].versions[0].number=1
conf.obj[2].name=name2
conf.obj[2].code=code2
conf.obj[2].versions[0].number=1
conf.obj[2].versions[1].number=2
AvailableConfig.java
package com.example;
import io.quarkus.runtime.annotations.StaticInitSafe;
import io.smallrye.config.ConfigMapping;
import java.util.List;
#StaticInitSafe
#ConfigMapping(prefix = "conf")
public interface AvailableConfig {
List<listObject> obj();
interface listObject {
String name();
String code();
List<Version> versions();
interface Version {
Integer number();
}
}
}
MainService.java
package com.example;
import io.quarkus.runtime.StartupEvent;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.event.Observes;
import javax.inject.Inject;
import javax.json.bind.Jsonb;
import java.util.List;
#ApplicationScoped
public class MainService {
#Inject
AvailableConfig availableConfig;
#Inject
Jsonb jsonb;
void onStart(#Observes StartupEvent ev) {
List<AvailableConfig.listObject> config = availableConfig.obj();
String result = jsonb.toJson(config);
}
}
As a result of execution, the correct object "config" is created.
But when it is serialized, an empty json "[{},{},{}]" is obtained.
How do I properly serialize things like this?
I don't know why jsonb behaves like this, but I found several solutions:
1. use Gson
Gson gson = new Gson();
String resultGson = gson.toJson(config);
2. use jackson
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY);
String resultJackson = objectMapper.writeValueAsString(config);
But I'm not sure if these options are ready for AOT-compilation in Quarkus environment.
3. Therefore, the best way is not to store such things in the config format, it is better to put them in your json-file.
I'm building a series of linked classes whose instances I want to be able to marshall to XML so I can save them to a file and read them in again later.
At present I'm using the following code as a test case:
import javax.xml.bind.annotation.*;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import java.time.LocalDate;
public class LocalDateExample
{
#XmlRootElement
private static class WrapperTest {
public LocalDate startDate;
}
public static void main(String[] args) throws JAXBException
{
WrapperTest wt = new WrapperTest();
LocalDate ld = LocalDate.of(2016, 3, 1);
wt.startDate = ld;
marshall(wt);
}
public static void marshall(Object jaxbObject) throws JAXBException
{
JAXBContext context = JAXBContext.newInstance(jaxbObject.getClass());
Marshaller marshaller = context.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
marshaller.marshal(jaxbObject, System.out);
}
}
The XML output is:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wrapperTest>
<startDate/>
</wrapperTest>
Is there a reason why the startDate element is empty? I would like it to contain the string representation of the date (i.e. toString()). Do I need to write some code of my own in order to do this?
The output of java -version is:
openjdk version "1.8.0_66-internal"
OpenJDK Runtime Environment (build 1.8.0_66-internal-b17)
OpenJDK 64-Bit Server VM (build 25.66-b17, mixed mode)
You will have to create an XmlAdapter like this:
public class LocalDateAdapter extends XmlAdapter<String, LocalDate> {
public LocalDate unmarshal(String v) throws Exception {
return LocalDate.parse(v);
}
public String marshal(LocalDate v) throws Exception {
return v.toString();
}
}
And annotate your field using
#XmlJavaTypeAdapter(value = LocalDateAdapter.class)
See also javax.xml.bind.annotation.adapters.XmlJavaTypeAdapters if you want to define your adapters on a package level.
http://blog.bdoughan.com/2011/05/jaxb-and-joda-time-dates-and-times.html describes the hole setup.
Joda-Time provides an alternative to the Date and Calendar classes currently provided in Java SE. Since they are provided in a separate library JAXB does not provide a default mapping for these classes.
To register the adapter for all files in a package. you can add package-info.java in the package you want to register it.
#XmlJavaTypeAdapters({
#XmlJavaTypeAdapter(type=LocalDate.class,
value=LocalDateAdapter.class),
})
package PACKAGE_NAME;
import java.time.LocalDate;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapters;
The adapter looks like:
import javax.xml.bind.annotation.adapters.XmlAdapter;
import java.time.LocalDate;
public class LocalDateAdapter extends XmlAdapter<String, LocalDate>{
public LocalDate unmarshal(String v) throws Exception {
return LocalDate.parse(v);
}
public String marshal(LocalDate v) throws Exception {
return v.toString();
}
}
As Nikolay Antipov mentioned in this commment there is already a well-tested threeten library that provides 14 type adapters (as of 2021-04-26) where one of them is the:
LocalDateXmlAdapter
and can be used e.g. like this on some Java field (generally not recommended by me though):
#XmlJavaTypeAdapter(value = LocalDateXmlAdapter.class) myLocalDt;
Instead I would recommend the package-info.java approach to do it implicitly in each package on the package level so it is applied automagically to e.g. all fields of the LocalDate type in all package classes:
create a file named src/java/my/xmlconv/classes/package-info.java
its content:
#XmlJavaTypeAdapters({
#XmlJavaTypeAdapter(value=LocalDateTimeXmlAdapter.class, type=LocalDateTime.class)
})
package my.xmlconv.classes;
import java.time.LocalDateTime;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapters;
import io.github.threetenjaxb.core.LocalDateTimeXmlAdapter;
I have some CSV files with several columns, but I just need to import some of these columns. The problem is that there are some columns at the beginning, some in the middle and some at the end of the line, like this:
id;name;address;bla;bla;bla;status;bla;bla;bla;value;another_value
And I want to write an object with only the necessary fields, like:
long id;
String address;
boolean status;
double value;
double another_value;
The problem is that these files have 79 columns, and I don't want to write a class with 79 properties and use only some of these properties.
You can deserialise the whole row to a Map and after that convert it to a required POJO. Below you can find an example:
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.MappingIterator;
import com.fasterxml.jackson.dataformat.csv.CsvMapper;
import com.fasterxml.jackson.dataformat.csv.CsvSchema;
import lombok.Data;
import lombok.ToString;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
public class CsvApp {
public static void main(String[] args) throws Exception {
File csvFile = new File("./resource/test.csv").getAbsoluteFile();
CsvMapper csvMapper = CsvMapper.builder()
//ignore unknown columns
.disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
.build();
CsvSchema schema = CsvSchema.emptySchema().withHeader().withColumnSeparator(';');
MappingIterator<Map> rowsIterator = csvMapper
.readerWithSchemaFor(Map.class)
.with(schema)
.readValues(csvFile);
List<Row> rows = new ArrayList<>();
while (rowsIterator.hasNext()) {
Map rowAsMap = rowsIterator.next();
Row row = csvMapper.convertValue(rowAsMap, Row.class);
rows.add(row);
}
rows.forEach(System.out::println);
}
}
#Data
#ToString
class Row {
long id;
String address;
boolean status;
double value;
double another_value;
}
I used Lombok library to make an example short but you do not have to use it.
For below CSV payload:
id;name;address;bla1;bla2;bla3;status;bla4;bla5;bla6;value;another_value
33;Jack;Cave Street;1;2;3;true;4;5;6;123;234
Above code prints:
Row(id=33, address=Cave Street, status=true, value=123.0, another_value=234.0)
It may be easier to just make a conversion to String[] and then write some code to convert into into an object.
In a Spring Boot application, we're already having a fully functional GraphQL endpoint, serving .graphqls files via GraphQL Java Tools (we included the graphql-spring-boot-starter dependency) and handling the data resolution through our base Query class implementing GraphQLQueryResolver and subsequent GraphQLResolver's.
For a business need, we have to re-create standard REST API endpoints, so I wonder why not just making calls to GraphQL (instead of having to re-implement "by hand" the data resolution once again)?
And as it's in the same backend application, no need to make HTTP or servlet (ForwardRequest) calls, just call some API's in Java.
The thing is I don't know how to proceed.
I read this example, but it's with basic GraphQL Java (not Tools):
https://www.graphql-java.com/documentation/v9/execution/
I know this should possible because we are allowed to do this in tests:
https://github.com/graphql-java-kickstart/graphql-spring-boot/blob/master/example-graphql-tools/src/test/java/com/graphql/sample/boot/GraphQLToolsSampleApplicationTest.java
But how to do it in regular code? There is not such thing as a GraphQLTemplate.
I also tried to search through examples at:
https://github.com/graphql-java-kickstart/graphql-java-tools/tree/master/example
https://github.com/graphql-java-kickstart/graphql-spring-boot
but found nothing relevant to our need.
Found nothing more in Documentation:
https://www.graphql-java-kickstart.com/tools/
https://www.graphql-java-kickstart.com/spring-boot/
What did I miss? Ideally I'm looking to inject some GraphQLSomething like this:
#RestController
#RequestMapping(path = "api")
public class CompanyController {
#Autowired
private GraphQLSomething graphQLSomething;
#GetMapping("company/{id}")
public ResponseEntity<?> societe(#PathVariable #NotNull Integer id) {
GraphQLSomethingResult result = GraphQLSomething.query("company(id: $id) { id name andsoone }", "{ \"id\": 123456 }").execute(); // not really sure of the GraphQL syntax here, but it'll need some tests...
return result.getDataOrElse();
}
}
Finally found how to do the thing as I wanted:
import java.util.Map;
import java.util.Optional;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Positive;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import com.google.common.collect.ImmutableMap;
import graphql.ExecutionResult;
import graphql.servlet.core.GraphQLQueryInvoker;
import graphql.servlet.core.internal.GraphQLRequest;
import graphql.servlet.input.GraphQLInvocationInputFactory;
import graphql.servlet.input.GraphQLSingleInvocationInput;
import lombok.extern.slf4j.Slf4j;
#Slf4j
#Validated
#RestController
#RequestMapping(path = "api")
public class CompanyController {
#Autowired
private GraphQLInvocationInputFactory invocationInputFactory;
#Autowired
private GraphQLQueryInvoker queryInvoker;
#GetMapping("company/{id}")
public ResponseEntity<?> societe(#PathVariable #NotNull Integer id) {
String query = "query ($id: Int!) { company(id: $id) { id name andsoon } }";
/*
* ImmutableMap is a Guava class; you can build the map (e.g. a HashMap) on your
* own, or simply Map.to(..) in Java 9, or even #PathVariable Map<String,
* Object> variables as the method's parameter instead (but you'll miss the
* validation).
*/
Map<String, Object> variables = ImmutableMap.of("id", id);
GraphQLRequest request = new GraphQLRequest(query, variables, null);
GraphQLSingleInvocationInput invocationInput = invocationInputFactory.create(request);
ExecutionResult result = queryInvoker.query(invocationInput);
/*
* Of course result.getData() can be null here - see also result.isDataPresent()
* - but data/error handling's left to you
*/
Optional<Object> company = Optional.ofNullable(result.getData().get("company"));
return ResponseEntity.of(company);
}
}
FYI, to get the dependencies for the above code, you'll need this:
<dependency>
<groupId>com.graphql-java</groupId>
<artifactId>graphql-spring-boot-autoconfigure</artifactId>
<version>5.0.2</version>
</dependency>
<dependency>
<groupId>com.graphql-java</groupId>
<artifactId>graphql-java-servlet</artifactId>
<version>6.1.3</version>
</dependency>
The following is a simplification of a more complicated setup. I describe this simple case to show the effect.
I start Spring Boot and call a method. In this method all the contents of a MySQL database table is read via
Iterable<myPojo> myPojos = myPojoRepository.findAll();
Afterwards I leave this method.
After finishing this method I get the message
Started Application in 70.893 seconds (JVM running for 72.899)
So Spring Boot is idling afterwards.
But still the memory is not released.
How can I avoid that the JVM Heap is not released after the application is idling?
This is the result of VisualJM after the application is idling:
char[] 1.080.623.712 (21.0%) 24.040.578 (23.4%)
byte[] 1.034.070.352 (20.1%) 17.280.824 (16.8%)
java.lang.String 768.935.872 (14.9%) 24.029.246 (23.4%)
java.lang.Object[] 556.181.104 (10.8%) 5.320.276 (5.1%)
org.hibernate.engine.internal.MutableEntityEntry
231.287.232 (4.5%) 2.628.264 (2.5%)
org.hibernate.engine.spi.EntityKey
224.752.040 (4.3%) 5.618.801 (5.4%)
byte[][] 212.407.904 (4.1%) 3.318.832 (3.2%)
hello.web.model.MyPojo 185.852.968 (3.6%) 3.318.803 (3.2%)
java.util.HashMap$Node 145.238.976 (2.8%) 3.025.812 (2.9%)
com.mysql.jdbc.ByteArrayRow 132.752.120 (2.5%) 3.318.803 (3.2%)
org.hibernate.engine.internal.EntityEntryContext$ManagedEntityImpl
126.156.720 (2.4%) 2.628.265 (2.5%)
hello.web.model.MyPojoCompoundKey
120.376.680 (2.3%) 3.009.417 (2.9%)
java.util.HashMap$Node[] 108.307.328 (2.1%) 16.558 (0.0%)
java.lang.Float 79.651.320 (1.5%) 3.318.805 (3.2%)
int[] 41.885.056 (0.8%) 54.511 (0.0%)
java.util.LinkedHashMap$Entry 15.519.616 (0.3%) 242.494 (0.2%)
java.io.File 11.323.392 (0.2%) 235.904 (0.2%)
org.springframework.boot.devtools.filewatch.FileSnapshot
10.550.400 (0.2%) 219.800 (0.2%)
java.lang.String[] 8.018.808 (0.1%) 52.031 (0.0%)
java.lang.reflect.Method 6.015.040 (0.1%) 37.594 (0.0%)
java.io.File[] 2.283.528 (0.0%) 16.746 (0.0%)
My effective-pom.xml shows that hibernate 5.0.9.Final is used.
The table my_pojo contains 3.3 million entries.
MyPojoRepository:
package hello.web.model;
import com.querydsl.core.types.Predicate;
import org.springframework.data.querydsl.QueryDslPredicateExecutor;
import org.springframework.data.repository.PagingAndSortingRepository;
import java.util.List;
public interface MyPojoRepository
extends PagingAndSortingRepository<MyPojo, Long>,
QueryDslPredicateExecutor<MyPojo> {
List<MyPojo> findAll(Predicate predicate);
}
MyPojo:
package hello.web.model;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import org.apache.commons.beanutils.BeanComparator;
import org.apache.commons.collections.comparators.ComparatorChain;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.IdClass;
import java.io.Serializable;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
#Data
#Entity
#Builder
#AllArgsConstructor
#IdClass(MyPojoCompoundKey.class)
public class MyPojo implements Serializable, Comparable<MyPojo> {
public MyPojo() { }
#Id
private String myId1;
#Id
private String myId2;
#Id
private String myId3;
private Float myId4;
private String myId5;
#Override
public int compareTo(MyPojo o) {
return this.getMyId3().compareTo(o.getMyId3());
}
protected boolean canEqual(Object other) {
return other instanceof MyPojo;
}
public static void sortByMyId1MyId3(List<MyPojo> myPojos) {
ComparatorChain chain = new ComparatorChain(Arrays.asList(
new BeanComparator("myId1"),
new BeanComparator("myId3")
));
Collections.sort(myPojos, chain);
}
}
myId1-3 and myId5 have a length of 10 characters in avarage.
So again:
How can I avoid that the JVM Heap is not released after the application is idling?