We Keep Coding

Is there any way to generate an Avro schema from a hashmap of values?

We are putting all our data points for one row into a hashmap. We don't want to use a pojo because the values are a different set each time. For example, we might get "place" on some records and we might get "hometown" on others. Actually we have thousands of different column names to choose from. Our code looks like this:
Map<String, Object> aMap = new HashMap<>();
aMap.put("id", Integer.valueOf(1));
aMap.put("age", Integer.valueOf(45));
aMap.put("name", "mark");
aMap.put("place", "home");
final GenericRecord record = new GenericData.Record(avroSchema);
aMap.forEach((k, v) -> {
record.put(k, v);
});
writer.write(record);
We would like to put all the values in a map and then generate a schema. Since using the Reflect api, it can be done for a pojo, I was wondering if it could be done from a hashmap as well?
As a side question, Is there any way to eliminate the forEach above and just write the map?

Here is what we came up with. We also had nested columns.
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.apache.avro.Schema;
import org.apache.avro.Schema.Parser;
import org.apache.avro.Schema.Type;
/**
* This does NOT do all types. So far just the types we think we need. See
* https://docs.oracle.com/database/nosql-12.1.3.0/GettingStartedGuide/avroschemas.html
* <p>
* We need some error handlling here and when we don't have the correct type, call it out!
* <p>
* This runs in 1-2ms even with a large payload.
*/
public class AvroSchemaBuilder {
/**
* Construct!
*/
private AvroSchemaBuilder() {
//private constructor. All methods are static.
}
/**
* Build the Avro schema and return it.
*
* #param name Name of object.
* #param nameTypeConsumer The nameTypeConsumer of objects being saved.
* #return the Avro schema.
*/
public static Schema getAvroSchema(String name, NameTypeConsumer nameTypeConsumer) {
String json = Lson.toJson(getAvroSchemaAsMap(name, nameTypeConsumer, true));
Parser parser = new Parser().setValidate(true);
return parser.parse(json);
}
/**
* Returns the map with all the attributes to build a schema. This would be recursive if we need
* to build a complex schema. For example for Trends this would build a complex schema where some
* of the types are maps that are themselves described as another nested schema.
*/
private static Map<String, Object> getAvroSchemaAsMap(String name,
NameTypeConsumer nameTypeConsumer,
boolean addNameSpace) {
Map<String, Object> schemaMap = new LinkedHashMap<>();
schemaMap.put("type", "record");
schemaMap.put("name", name);
if (addNameSpace) {
schemaMap.put("namespace", "com.blah.blah");
}
List<Field> fields = new ArrayList();
nameTypeConsumer.consumeNestedNameType((columnName, nestedNameType) -> {
Object avroType;
if (nestedNameType.getNameTypeConsumer() != null) {
avroType = getAvroSchemaAsMap(columnName, nestedNameType.getNameTypeConsumer(), false);
} else {
avroType = getAvroType(nestedNameType.getType()).getName();
}
Object[] types = {"null", avroType}; //adding null first always.
fields.add(new Field(columnName, types));
});
schemaMap.put("fields", fields);
return schemaMap;
}
/**
* Finds the avro type by class.
*
* #param type the Type (this is an avro type).
* #return avro constant.
*/
private static Type getAvroType(Class<?> type) {
if (type.equals(Integer.class)) {
return Type.INT;
}
if (type.equals(Long.class)) {
return Type.LONG;
}
if (type.equals(Float.class)) {
return Type.FLOAT;
}
if (type.equals(Double.class)) {
return Type.DOUBLE;
}
if (type.equals(String.class)) {
return Type.STRING;
}
if (type.equals(Boolean.class)) {
return Type.BOOLEAN;
}
throw new GenericRuntimeException("Cannot get Avro type for type " + type.getName());
}
/**
* Nested class to make our field.
*/
private static class Field {
public final String name;
public final Object[] type;
public Field(String name, Object[] type) {
this.name = name;
this.type = type;
}
}
}

How to get all the set parameters when executing a Maven plugin goal?

/**
* #goal query
*/
public class MyQueryMojo
extends AbstractMojo
{
/**
* #parameter
*/
private String param1;
/**
* #parameter
*/
private String param2;
public void execute()
throws MojoExecutionException
{
showAllParams();
}
}
How to implement showAllParams() to print the set/passed values of all the parameters when executing mvn myquery:query -Dquery.param1=val1 -Dquery.param2=val2?
Expected StdOut:
query.param1=val1
query.param2=val2
The intention is different from using help:describe because help:describe will return the metadata for all params while I need to get the actual passed values.
Edit: showAllParams() should not be implemented in a hardcoded fashion and it should work even with numbers of params changed.
Instead of:
if (param1 != null) this.getLog.info(param1);
if (param2 != null) this.getLog.info(param2);
...
if (param9 != null) this.getLog.info(param9);
Is it possible to do it in this fashion:
Map<String, String> params = getAllParams();
getLog().info(params.toString());

Well, considering that param1 and param2 are not initialized and have no default value, you could just check all of the parameters, and if a parameter isn't null, then you print it. In your case:
if (param1 != null) this.getLog.info(param1);
if (param2 != null) this.getLog.info(param2);
However, if you have a parameter with a default value, then according to the following link (http://maven.40175.n5.nabble.com/Check-if-parameter-is-explicitly-set-td5905937.html), there is no way of telling whether a value is explicitly set by the command line or is set as a default value.

This would be my take on it.
Notice that you can pass named parameters (i.e. "-Dquery.param1=1") or an array of parameter values.
Array type parameters are configured by specifying the parameter multiple times.
(i.e. "-Dquery.param1=1 -Dquery.param1=11" , etc.)
More information on the docs:
https://maven.apache.org/guides/plugin/guide-java-plugin-development.html
/**
* #goal query
*/
public class MyQueryMojo extends AbstractMojo{
private final static String PARAM1_DEFAULT_VALUE="param1";
private final static String PARAM2_DEFAULT_VALUE="param2";
private final static String PARAM3_DEFAULT_VALUE="param3";
/**
* param1
*/
#Parameter( property = "query.param1", defaultValue = PARAM1_DEFAULT_VALUE )
private String param1;
/**
* param1
*/
#Parameter( property = "query.param2", defaultValue = PARAM2_DEFAULT_VALUE )
private String param2;
/**
* param1
*/
#Parameter( property = "query.param3", defaultValue = PARAM3_DEFAULT_VALUE )
private String param3;
/**
* paramArray
*/
#Parameter( property = "query.param")
private String[] paramArray;
public void execute() throws MojoExecutionException{
// print individual parameters
printAllParams();
// print all in array
printArrayParam();
}
private void printArrayParam(){
for(String p : paramArray){
getLog.info( "paramArray: " + p );
}
}
private void printAllParams(){
getLog().info("param1:" + param1);
getLog().info("param2:" + param2);
getLog().info("param3:" + param3);
}
}

Getting column names from a JPA Native Query

I have an administrative console in my web application that allows an admin to perform a custom SQL SELECT query on our database.
Underneath, the application is using Hibernate, but these queries are not HQL, they're pure SQL, so I'm using a Native Query like this:
protected EntityManager em;
public List<Object[]> execute(String query) {
Query q = em.createNativeQuery(query);
List<Object[]> result = q.getResultList();
return result;
}
This works correctly, but it only returns the rows of data, with no extra information. What I would like is to also get the column names, so when I print the results back to the user I can also print a header to show what the various columns are.
Is there any way to do this?

Query query = entityManager.createNamedQuery(namedQuery);
NativeQueryImpl nativeQuery = (NativeQueryImpl) query;
nativeQuery.setResultTransformer(AliasToEntityMapResultTransformer.INSTANCE);
List<Map<String,Object>> result = nativeQuery.getResultList();
And now you have Map<String,Object> . You can see your Column Names

2020
With hibernate 5.2.11.Final is actually pretty easy.
In my example you can see how I get the column names for every row. And how I get values by column name.
Query q = em.createNativeQuery("SELECT columnA, columnB FROM table");
List<Tuple> result = q.getResultList();
for (Tuple row: result){
// Get Column Names
List<TupleElement<Object>> elements = row.getElements();
for (TupleElement<Object> element : elements ) {
System.out.println(element.getAlias());
}
// Get Objects by Column Name
Object columnA;
Object columnB;
try {
columnA = row.get("columnA");
columnB= row.get("columnB");
} catch (IllegalArgumentException e) {
System.out.println("A column was not found");
}
}

This code worked for me
DTO Class :
public class ItemResponse<T> {
private T item;
public ItemResponse() {
}
public ItemResponse(T item) {
super();
this.item = item;
}
public T getItem() {
return item;
}
public void setItem(T item) {
this.item = item;
}
}
Service Class is in the below
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import javax.persistence.Query;
import org.springframework.stereotype.Service;
import org.hibernate.transform.AliasToEntityMapResultTransformer;
#Service
public class ServiceClass{
#PersistenceContext
public EntityManager entityManager;
public ItemResponse exceuteQueryResponse(String queryString) {
ItemResponse itemResponse=new ItemResponse();
Query jpaQuery = entityManager.createNativeQuery(queryString);
org.hibernate.Query hibernateQuery =((org.hibernate.jpa.HibernateQuery)jpaQuery).getHibernateQuery();
hibernateQuery.setResultTransformer(AliasToEntityMapResultTransformer.INSTANCE);
List<Map<String,Object>> res = hibernateQuery.list();
itemResponse.setItem(res);
return itemResponse;
}
}

Ryiad's answer DTO adds some confusion, you should have kept it away.
You should have explained that it works only with hibernate.
If like me you needs to keep the order of columns, you can specify your own transformer. i copied the code from hibernate and changed the HashMap to LinkedHashMap:
import java.util.LinkedHashMap;
import java.util.Map;
import org.hibernate.transform.AliasedTupleSubsetResultTransformer;
import org.hibernate.transform.ResultTransformer;
/**
* {#link ResultTransformer} implementation which builds a map for each "row", made up of each aliased value where the
* alias is the map key. Inspired by {#link org.hibernate.transform.AliasToEntityMapResultTransformer}, but kepping the
* ordering of elements.
* <p/>
* Since this transformer is stateless, all instances would be considered equal. So for optimization purposes we limit
* it to a single, singleton {#link #INSTANCE instance}.
*/
public class AliasToEntityMapResultTransformer extends AliasedTupleSubsetResultTransformer {
public static final AliasToEntityMapResultTransformer INSTANCE = new AliasToEntityMapResultTransformer();
/**
* Disallow instantiation of AliasToEntityMapResultTransformer.
*/
private AliasToEntityMapResultTransformer() {
}
#Override
public Object transformTuple(Object[] tuple, String[] aliases) {
Map result = new LinkedHashMap<>(tuple.length);
for (int i = 0; i < tuple.length; i++) {
String alias = aliases[i];
if (alias != null) {
result.put(alias, tuple[i]);
}
}
return result;
}
#Override
public boolean isTransformedValueATupleElement(String[] aliases, int tupleLength) {
return false;
}
/**
* Serialization hook for ensuring singleton uniqueing.
*
* #return The singleton instance : {#link #INSTANCE}
*/
private Object readResolve() {
return INSTANCE;
}
}
With this transformer you can used Ryiad's solution with Hibernate:
Query jpaQuery = entityManager.createNativeQuery(queryString);
org.hibernate.Query hibernateQuery =((org.hibernate.jpa.HibernateQuery)jpaQuery).getHibernateQuery();
hibernateQuery.setResultTransformer(AliasToEntityMapResultTransformer.INSTANCE);
List<Map<String,Object>> res = hibernateQuery.list();

After a long time without any answer, And based on my own further research, It seems that it can not be possible, Unfortunately.

If the JPA provider does not support the retrieval of query metadata, another solution could be the use of a SQL parser like JSQLParser, ZQL or General SQL Parser (comercial), which extracts the fields from the SELECT statement.

cast query to hibernate query, then use hibernate method
//normal use, javax.persistence.Query interface
Query dbQuery = entityManager.createNativeQuery(sql);
//cast to hibernate query
org.hibernate.Query hibernateQuery =((org.hibernate.jpa.HibernateQuery)dbQuery)
.getHibernateQuery();
hibernateQuery.setResultTransformer(AliasToEntityMapResultTransformer.INSTANCE);
List<Map<String,Object>> res = hibernateQuery.list();
List<TxTestModel> txTestModels = new ArrayList<>();
res.forEach(e->{
TxTestModel txTestModel = new ObjectMapper().convertValue(e, TxTestModel.class);
// txTestModels.add(new TxTestModel().setIdd((Integer) e.get("idd")).setMmm((String) e.get("mmm")).setDdd((Date) e.get("ddd")));
txTestModels.add(txTestModel);
});
System.out.println(txTestModels.size());

To enforce em.createNativeQuery(..).getResultList() to return List<Tuple> specify it with Tuple.class when creating native queries :
Query q = em.createNativeQuery("SELECT columnA, columnB FROM table", Tuple.class );
List<Tuple> result = q.getResultList();
for (Tuple row: result){
// Get Column Names
List<TupleElement<Object>> elements = row.getElements();
for (TupleElement<Object> element : elements ) {
System.out.println(element.getAlias());
}
// Get Objects by Column Name
Object columnA;
Object columnB;
try {
columnA = row.get("columnA");
columnB= row.get("columnB");
} catch (IllegalArgumentException e) {
System.out.println("A column was not found");
}
}

This worked for me:
final Query emQuery = em.createNativeQuery(query, Tuple.class);
final List<Tuple> queryRows = emQuery.getResultList();
final List<Map<String, Object>> formattedRows = new ArrayList<>();
queryRows.forEach(row -> {
final Map<String, Object> formattedRow = new HashMap<>();
row.getElements().forEach(column -> {
final String columnName = column.getAlias();
final Object columnValue = row.get(column);
formattedRow.put(columnName, columnValue);
});
formattedRows.add(formattedRow);
});
return formattedRows;

This is the working solution
Below example return the objectlist from the query.
Looping the same and from the first object cast it to hasmap, and hashmap.keyset will give you all the coulmn names in a set.
List dataList = session.createSQLQuery("SLECT * FROM EMPLOYEETABLE").setResultTransformer(Transformers.ALIAS_TO_ENTITY_MAP).list();
for (Object obj : dataList) {
HashMap<String, Object> hashMap = (HashMap<String, Object>) obj;
Set<String> keySet = hashMap.keySet();
break;
}

I also faced a similar problem working with JPA. There is no direct way in JPA to access the resultset metadata. The solution can be extracting column names from the query itself or use JDBC to get the metadata.

JDBI resultsetmapper create list of objects from query resultset?

Opting for an alternative to JPA and Spring-Data I wanted to try out JDBI for my Repository implementation with SQLite
Repository Code
/**
* SQLite implementation of Foo Repository
*/
public class FooRepository implements FooRepository {
private final DBI connection;
/**
* The constructor initialises the connection to the local SQLite file
*
* #param dataSource jdbc connection string e.g. "jdbc:sqlite::resource:db/foo.db"
* #throws IllegalArgumentException when an invalid DB file is given
*/
public FooRepository(final SQLiteDataSource dataSource) {
checkNotNull(dataSource, "dataSource required");
connection = new DBI(dataSource);
}
/**
* Returns a list of Foo objects for a website locale in the DB
* #return List
* #throws SQLException error querying
*/
#Override
public List<Foo> getFoosByWebsiteLocale(f) throws SQLException {
checkNotNull(websiteLocale, "websiteLocale required");
final String fooQuery = query...
Handle queryHandler = connection.open();
final List<Foo> fooList = queryHandler.createQuery(fooQuery)
.map(FooMapper.class);
queryHandler.close();
return fooList;
}
}
Mapper
public class FooMapper implements ResultSetMapper {
/**
* Construct a Foo object from a record in the result set
* #param index row number
* #param resultRow row
* #param ctx statementcontext
* #return Foo object
* #throws SQLException when accessing sql result set
*/
#Override
public Foo map(final int index, final ResultSet resultRow, final StatementContext ctx) throws SQLException {
return Foo.builder()
.name(resultRow.getString("foo_name"))
.type(resultRow.getString("foo_type"))
.build();
}
}
I am struggling to understand how I will create a list of Foo objects using ResultSetMapper.
The JDBI documentation also appears to be broken on this area :
http://jdbi.org/maven_site/apidocs/org/skife/jdbi/v2/tweak/ResultSetMapper.html
Help would be appreciated on how to make this work.

Your mapper only needs to map one row to one Foo object. JDBI will create the list and put the objects in the list for you.
I.e.:
final List<Foo> fooList = queryHandler.createQuery(fooQuery).map(FooMapper.class).list();

How to persist an entity declaring a field with a fixed-size array of a few integers using JPA/Hibernate

I have a class with a field wich references a fixed-size array of integers:
class Tracking{
private int[] tracks;
}
What is the best way of mapping the tracks field to a database using JPA (backed by Hibernate)?
The tracks array will contain always 10 elements. I know about #ElementCollection annotation but that implies that I'll end up with two tables (Tracking and Tracking_tracks) and it seems to be unnecessary.
The only approach that comes to mind is to use a String field mapping the 10 numbers into a single String separated by colon. I would make that field persistent (a simple basic #Column) and then in the getters and setters I would perform the parsing into an int[].
Any suggestions? I just want an efficient way of persisting this information. The Tracking objects will be generated permanently, there will be thousands of them and, as the number of integers is always 10, it seems overkilling to persist them into a separate table and perform joins and selects to retrieve them.

This might depend on your persistence provider. but you could try using a special column definition and let your persistence provider handle the conversion. If this won't work, a lot of persistence provider support custom types. In hibernate they are called user types, open jpa uses custom field mappings. Your database may support raw fixed size byte storage.

Following the suggestion made by SpaceTrucker about Hibernate UserType, here is the full code required to provide an Hibernate custom UserType (many thanks to blog post of Kunaal A Trehan for the code in which this class is based on):
package org.mypackage;
import java.io.Serializable;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Types;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.StringTokenizer;
import org.hibernate.HibernateException;
import org.hibernate.engine.spi.SessionImplementor;
import org.hibernate.tool.hbm2ddl.SchemaExport.Type;
import org.hibernate.usertype.UserType;
import org.springframework.util.ObjectUtils;
/**
* An Hibernate {#link UserType} that allows persisting a list of integers in one single column in the table.
* Based in code from http://javadata.blogspot.com.ar/2011/07/hibernate-and-user-types.html.
*
* In order to use this as the mapper for a column, use the following in the field of your entity:
* #Type(type=IntegerListUserType.NAME)
* List<Integer> tracks;
*
* #author dds
* */
public class IntegerListUserType implements UserType {
public static final String NAME = "org.mypackage.IntegerListUserType"; //WARNING this must match class name!
#Override
public int[] sqlTypes() {
return new int[] { Types.VARCHAR };
}
#SuppressWarnings("rawtypes")
#Override
public Class returnedClass() {
return List.class;
}
#Override
public boolean equals(Object x, Object y) throws HibernateException {
return ObjectUtils.nullSafeEquals(x, y);
}
#Override
public int hashCode(Object x) throws HibernateException {
if (x != null)
return x.hashCode();
else
return 0;
}
#Override
public Object nullSafeGet(ResultSet rs, String[] names,
SessionImplementor session, Object owner)
throws HibernateException, SQLException {
List<Integer> list = null;
String nameVal = rs.getString(names[0]);
if (nameVal != null) {
list = new ArrayList<Integer>();
StringTokenizer tokenizer = new StringTokenizer(nameVal, ",");
while (tokenizer.hasMoreElements()) {
String number = (String) tokenizer.nextElement();
list.add(Integer.valueOf(number));
}
}
return list;
}
#SuppressWarnings("unchecked")
#Override
public void nullSafeSet(PreparedStatement st, Object value, int index,
SessionImplementor session) throws HibernateException, SQLException {
if (value == null) {
st.setNull(index, Types.VARCHAR);
} else {
st.setString(index, serialize((List<Integer>) value));
}
}
private String serialize(List<Integer> list) {
StringBuilder strbul = new StringBuilder();
Iterator<Integer> iter = list.iterator();
while (iter.hasNext()) {
strbul.append(iter.next());
if (iter.hasNext()) {
strbul.append(",");
}
}
return strbul.toString();
}
#SuppressWarnings("unchecked")
#Override
public Object deepCopy(Object value) throws HibernateException {
if (value == null)
return null;
else {
List<Integer> newObj = new ArrayList<Integer>();
List<Integer> existObj = (List<Integer>) value;
newObj.addAll(existObj);
return newObj;
}
}
#Override
public boolean isMutable() {
return false;
}
#Override
public Serializable disassemble(Object value) throws HibernateException {
Object deepCopy = deepCopy(value);
if (!(deepCopy instanceof Serializable))
return (Serializable) deepCopy;
return null;
}
#Override
public Object assemble(Serializable cached, Object owner)
throws HibernateException {
return deepCopy(cached);
}
#Override
public Object replace(Object original, Object target, Object owner)
throws HibernateException {
return deepCopy(original);
}
}
Note that I'm converting String to List < Integer > instead of int[] just for simplicity.