I am working on ETL Java project and it does 3 things
extract - read the data from a table
transform the data to JSON
Load the data
It works fine. The issue is I am doing it for each table. The way I have right now is
class ETLHelper
{
private Person read(ResultSet results){
Person p = new Person();
p.setPersonId(results.getString("PERSON_ID"));
p.setPersonName(results.getString("PERSON_NAME"));
return p;
}
private String transform(Person p){
TransformPerson t = new TransformPerson();
t.setTransformPersonId(p.getPersonId);
t.setTransformPersonName(p.getPersonName);
PersonEData eData = new PersonEData();
eData.setDate1(p.date1);
eData.setDate2(p.date2);
t.seteData(eData);
PersonDetails pd = new PersonDetails();
pd.settransformdata(t);
return writeValueAsString(pd);
}
public void etl(){
Connection c = null;
PreparedStatement p = null;
ResultSet r = null;
c = getConnection();
p = c.prepareStatement(getSql());
r = p.executeQuery();
while(r.next()){
messages.add(transform(read(r)));
/*code for loading data*/
}
}
}
Person.Java:
#JsonTypeName(value = "PERSON")
#JsonTypeInfo(include = JsonTypeInfo.As.WRAPPER_OBJECT, use = JsonTypeInfo.Id.NAME)
public class Person{
#JsonProperty(value = "PERSON_ID")
private String personId;
//getter and setter for personId
#JsonProperty(value = "PERSON_NAME")
private String personName;
//getter and setter for personName
}
TransformPerson.java:
#JsonRootName(value = "Person")
class TransformPerson{
private String transformPersonName;
private String transformPersonId;
/*getter and setter for transformPersonName and tranformPersonId*/
#override
String toString(){
return "Person [name =" + transformPersonName + ", id = " + transformPeronId "]";
}
}
PersonEdata:
private String date1;
private String date2;
/*getter and setter*/
#override
public String toString(){
return "PersonEdata [date1=" + date1 +", date2=" + date2 + "]";
}
So a Person class, a class needed for transformation and etl class is written for each table. There are also some additional classes like PersonEdata that returns JSON when toString() is called. Is there anyway can I change this design to avoid writing the similar code for each table? There are some constraints. Each table is different and they transformation class is needed because there are other programs that uses the JSON generated so we need to generate JSON that needs to understood by those programs.
In your current solution, you have created :
Person class - to hold the data retrieved from DB
PersonTransform class - to copy the data from Person to other representation and have extended the capability to create JSON by overrinding toString()
To keep it simple what you can do is:
Have single Class like Person for each entity (Table) - which is JSON Serializable.
Don't override the toString method to represent the JSON representation - use JSON serializer instead.
Related
inside my test class
String json="{\n" +
"\t\"masterName\": \"test1\",\n" +
"\t\"masterSubjectName\": \"testsubject\",\n" +
"\t\"masterRoll\": \"534\",\n" +
"\t\"firstName\": \"studentFirstName\",\n" +
"\t\"rollNumber\": \"23\"\n" +
"}";
Student studentDTO=new Gson().fromJson(json, Student.class);
System.out.println(studentDTO);
Student.java
#Data
public class Student {
#Delegate(types = Master.class)
private Master master=new Master();
private String firstName;
private String lastName;
private int rollNumber;
}
Master.java
#Data
public class Master {
private String masterName;
private String masterSubjectName;
private int masterRoll;
}
This gives Response:
Student(master=Master(masterName=null, masterSubjectName=null, masterRoll=0), firstName=studentFirstName, lastName=null, rollNumber=23)
When I parse the json string "json" to the Student class,
Why values not getting passed to "Master.java" inner dto?
I need something like Student(masterName=test1, masterSubjectName=testsubject, masterRoll=534, firstName=studentFirstName,rollNumber=23)
Your 'types' value in #Delegate should be Interface with getter methods for example getMasterName() , getMasterSubjectName... and so on
documentation:
https://projectlombok.org/features/experimental/Delegate
I am using Jackson to deserialize XML. It works fine when serializing into a list when there are no other elements in between, but if i insert some other field in the middle somewhere, it seems to only put the things below the field into my list. See below only the ProductA objects after the Product B tag are being included.
I set a breakpoint to get the JsonNode object using the xmlMapper readTree method, and the properties above the ProductB tag are not there in the node tree.
Is there a way using jackson (or another library) to get all ProductA elements inside the Warehouse element, independent of the ordering of the Warehouse child elements?
public class Warehouse {
#JacksonXmlElementWrapper(useWrapping = false)
#JacksonXmlProperty(localName = "ProductA")
private List<ProductA> productAList;
#JacksonXmlProperty(localName = "ProductB")
private String productB;
//getters and setters
}
public class ProductA {
#JacksonXmlProperty(localName = "PropA")
private String propA;
//getters and setters
}
<Warehouse>
<ProductA>
<propA>abc</propA>
</ProductA>
<ProductA>
<propA>abc</propA>
</ProductA>
<ProductB>def</ProductB>
<ProductA>
<propA>abc</propA>
</ProductA>
</Warehouse>
Change your Warehouse class to use a method just for setting the value of productAList from XML, and remove the annotations from the productAList property.
#JsonSetter(value = "ProductA")
public void setProductAListFromXml(ProductA productA) {
if (this.productAList == null) {
this.productAList = new ArrayList<ProductA>();
}
this.productAList.add(productA);
}
Here is the full Warehouse class:
#JacksonXmlRootElement(localName = "Warehouse")
public class Warehouse {
private List<ProductA> productAList;
#JacksonXmlProperty(localName = "ProductB")
private String productB;
public List<ProductA> getProductAList() {
return productAList;
}
public void setProductAList(List<ProductA> productAList) {
this.productAList = productAList;
}
#JsonSetter(value = "ProductA")
public void setProductAListFromXml(ProductA productA) {
if (this.productAList == null) {
this.productAList = new ArrayList<ProductA>();
}
this.productAList.add(productA);
}
public String getProductB() {
return productB;
}
public List<ProductA> getProductA() {
return productAList;
}
}
Using the class like this:
String str = "<Warehouse>\r\n" +
" <ProductA>\r\n" +
" <propA>abc</propA>\r\n" +
" </ProductA>\r\n" +
" <ProductA>\r\n" +
" <propA>abc</propA>\r\n" +
" </ProductA>\r\n" +
" <ProductB>def</ProductB>\r\n" +
" <ProductA>\r\n" +
" <propA>abc</propA>\r\n" +
" </ProductA>\r\n" +
"</Warehouse>";
XmlMapper mapper = new XmlMapper();
Warehouse warehouse = mapper.readValue(str, Warehouse.class);
System.out.println(warehouse.getProductB());
System.out.println(warehouse.getProductA());
Produces this output:
def
[ProductA [propA=abc], ProductA [propA=abc], ProductA [propA=abc]]
My JSON in my #RequestBody ObjectNode objectNode
{
"script":
{"id":2,"nom":"tes","libelleprerequiss":
[{"id":1,"libelle_prerequis":"Soc devis VAM","produit":{"id":1,"nom":"VAM"},"typologie":{"id":1,"nom":"devis"}}]
},
"libellePrerequis":
{"id":3,"libelle_prerequis":"Soc contrat VAM","produit":{"id":1,"nom":"VAM"},"typologie":{"id":2,"nom":"contrat"}
}
}
When i do :
String id_script = objectNode.get("script").get("id").asText();
String id_libelleprerequis = objectNode.get("libellePrerequis").get("id").asText();
System.out.println("Script ID = " + id_script + "...");
System.out.println("Libelle Prerequis ID = " + id_libelleprerequis + "...");
i Have a result :
Update Script - Script ID = 2...
Update Script - Libelle Prerequis ID = 3...
But when i do :
String libelleprerequis = objectNode.get("libellePrerequis").asText();
System.out.println("Update Script - Libelle Prerequis = " + libelleprerequis + "...");
I have :
Update Script - Libelle Prerequis = ...
I would like to recover my "LibellePrerequis" to integrate it to my script in my entity (many to many)
"libellePrerequis" is an object.
asText() works only if your node is a value.
You can check with objectNode.get("libellePrerequis").isValueNode();
To get a node object as String, you need to use toString():
objectNode.get("libellePrerequis").toString();
EDIT1:
To transform your node into object you need to create some objects which represent the tree of your node.
public class Libell{
private int id;
private String libelle_prerequis;
private Produit produit;
private Typologie typologie;
//getters, setters
}
public class Typologie{
private int id;
private String nom;
//getters, setters
}
public class Produit{
private int id;
private String nom;
//getters, setters
}
The most important thing is to have the same name for attributes(id, libelle_prerequis, ...) as in json and pay attention to their type. So the name of objects (Libell, Typologie and Produit) doesn't matter if their structure is like in json.
Now, to map your JSonNode to an object you can do this:
ObjectMapper objMap = new ObjectMapper();
Libell libell = objMap.convertValue(objectNode.get("libellePrerequis"), Libell.class);
System.out.println(libell.getId());
System.out.println(libell.getProduit().getId());
System.out.println(libell.getTypologie().getId());
I want to separate the result data with query by "~" within the same entity, to separate another entity data by ","
My code
mainRepository.java
public interface mainRepository extends CrudRepository<Error, Long> {
#Query(value= "SELECT * FROM Error t where t.applicationID = :applicationid", nativeQuery= true)
List<Error> findListByApp(#Param("applicationid") String applicationid);
}
and in another class I call that function
String cb = errorRepository.findListByApp("application1").toString();
System.out.println(cb);
the result if i execute cb is
[com.info.main.Error#6ec8b40e, com.info.main.Error#6ec8b40e, com.info.main.Error#6ec8b40e]
I want to keep it sorted by app first, then name, then email.
the result that i want to achieve is just like this :
[app1~name1~email1, app2~name2~email2, app3~name3~email3]
Why you are using toString with the result List?
String cb = errorRepository.findListByApp("application1").toString();//why toString?
Instead you have to use a List like so :
List<Error> result = errorRepository.findListByApp("application1");
//this will print the list or Error it will call to String for each error
System.out.println(result);
to show the result you need just override toString method in your Error entity with the format you want for example.
#Entity
public class Error {
private String app;
private String name;
private String email;
//getter setter
#Override
public String toString() {
return app + "~" + name + "~" + email;
}
}
I have a lots of classes that extends from one class
Also I have one method that its argument is that parent class and create query base on attribute of those classes.
sometimes I need to ignore some attribute from result query.
so is it possible to remove some attribute of object?
class A1 extends Model {
public String field1 = "";
public String field2 = "";
public String table = "A1";
#Override
public String getTable() {
return this.table;
}
}
class A2 extends Model {
public String field1 = "";
public String field2 = "";
public String field3 = "";
public String table = "A2";
#Override
public String getTable() {
return this.table;
}
}
class Utility {
public static String query(Model params) {
Field[] fields = params.getClass().getFields();
String head = "INSERT INTO " + params.getTable() + "(";
String tail = "VALUES (";
for(Field field : fields) {
String key = field.getName();
String val;
try {
val = field.get(params);
} catch (Exception e) {
val = null;
}
head += key + ",";
tail += "'" + val + "',";
}
head = head.substring(head,0,head.length() -1) + ")";
tail = tail.substring(tail,0,tail.length() -1) + ")";
return head + tail;
}
}
I call query method by sending one model
A1 data = new A1();
data.field1 = "Name";
data.field2 = "Family";
String query = Utility.query(data);
I just want to remove field2 from query how can I do that?
thanks for any help
You could implement an annotiation. Let's call it #DontPersist. Use it to mark fields which should not get persisted. In Utility.query() you can check for the annotation with reflection.
As your Model class does not implement anything (it could be an interface, but that's another topic), you can extend it creating a class with less attributes when necessary (an anonymous class will do the job).
Anyway, I think you should refactor your code: why not using a List to store fields? It's easier and it does not need reflection.
I'll use something like:
public class Model extends ArrayList{
public Model(String name) { tableName=name;}
private String tableName;
public String getTable() {return tableName}
}
And then you can iterate over the Array to obtain the field names.