So I had an AVRO file and have not had any experience with that type of file, so I read the contents of that file and saved it to a text file, and so now I am trying to parse each line and add that to a MySQL table. I know how to connect to a MySQL database using Java and will basically execute a query that adds the data from each line.
But the part I am having trouble with is parsing my data, basically this is what each line looks like (and each value is a 'String'):
{"content": "HTML", "GLOBALEVENTID": "331284989", "SQLDATE": "20140111", "MonthYear": "201401", "Year": "2014"}
So there are more columns than this but I shortened it, also the "content" field is actually the HTML of a webpage so it can contain a lot of random characters which I think could be an issue when parsing. But so my question is that I am trying to do parse out the values of each column and add it into an array (content, GLOBALEVENTID, etc.), so then I can add it to a MySQL table that already has these columns defined? Anything that can help me point me in the right direction is appreciated!
There are two approaches to solve this problem, depending on the what you are trying to achieve:
Case 1) If this is just a one time load
Answer: For a one time load, reading the AVRO file, parsing it to text file and then seeding data to MySQL using RDBMS APIs is too much work.
Instead, I would suggest to use the MySQL Import Utility.
If you go to the Schema Browser, and right click on the table name, you will find an option "Import..."
The options are explanatory. Usually, one time loads are done using a CSV or XLS file. You can modify your already existing program to convert AVRO file in to CSV file and use this file to Import data into MySQL table.
Case 2) If AVRO file is to be read through a program, and this action will be done multiple times in future.
In this case, you may use one of the many libraries (eg: Jackson/GSON), to parse the modified AVRO file into valid Java Object POJO. Make sure that the Object representation is a ORM (e.g: JPA/Hibernate) entity.
For example:
JSON: {"content": "HTML", "GLOBALEVENTID": "331284989", "SQLDATE": "20140111", "MonthYear": "201401", "Year": "2014"}
Class File:
#Entity
#Table(name = "CONTENT")
class Content {
#Id
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator = "SOME_SEQUENCE")
private Long id;
#Column(name = "DATA")
private String data;
#Column(name = "GLOBALEVENTID")
private String globalEventId;
#Column(name = "DATE")
#Temporal(TemporalType.TIMESTAMP)
private String date;
....
....
}
Once the data has been parsed into the ORM entity, saving it to the Database should be very easy. As per your convinece, you may use entityManager.save/update or entitryManager.saveAll/updateAll
Related
i will try to be more clear as possible. My project uses 2 containers to execute a service and get data from my DB2 . The first container passes input data through an IRestRequestTransformer that convert my data to be readable by the second container. The second container execute a query on my DB2 and extract the data I need. This data returns to the first container through an IRestResponseTransformer. In my case the data i'm trying to get is a Timestamp with microseconds precision that for example will be "2022-09-23 11.25.52.660135" . I can see that this Timestamp is correct when the query is executed on the second container :
but when it passes through the IRestResponseTransformer it gets cut and on my first container i only get "2022-09-23 11.25.52.660" without microseconds:
The model of the table contains the mapping of all the columns with getter and setter like this :
#Column(name = "TIMESTAMP_INS_B03")
private Timestamp timestampB03;
public Timestamp getTimestampInsB03() {
return timestampInsB03;
}
public void setTimestampInsB03(Timestamp timestampInsB03) {
this.timestampInsB03 = timestampInsB03;
}
I work with json and I wonder if exist an annotation like JsonFormat to specify that I need to keep microseconds or something similar. If you need more informations tell me and I will answer.
I do have a Java Web Applicaiton (struts2, hibernate, beans) + PostreSQL as DB. The task is to save the base64 encoded text in the db for some specific table. That base64 is generated from pdf file, which is then ciphered with a specific algorithm. The pdf files <1mb, mostly <300kb.
I did a search and it's suggested to save the base64 as a Text field in the DB. It's not problem to create it within the PostgreSQL itself, but I have to create it via a Model class + hibernate.
What I did:
Imported import org.apache.struts2.components.Text;
Generated getters/setters. Added one row to my *.hbm.xml file.
<property name="base64signed" column="base64signed" />
And I got this error:
Could not determine type for: org.apache.struts2.components.Text
I think you should go with this annotation :
#Lob(type = LobType.CLOB)
I don't think Hibernate supports conversion of org.apache.struts2.components.Text to DB's varchar.
So you store it as LOB or CLOB as mentioned in above
#Lob
private Text base64signed;
Or you can make it easy by declaring your 'base64signed' field as String, it will take less memory in DB
#Column
private String base64signed;
I'm using Spring Boot with Hibernate, JPA and PostgreSQL. I'm wanting to convert database large objects into text content. Previously I was defining my long text in my JPA entity as #Lob:
#Lob
String description;
I then discovered that often problems are created using #Lob's and decided to rather change them to:
#Type(type="org.hibernate.type.StringClobType")
String description;
Which is represented in the database as a text type. Unfortunately, now the reference numbers (oid's) of the previous large objects are stored in my rows instead of the actual content. For example:
id | description
---------------------
1 | 463784 <- This is a reference to the object rather than the content
instead of:
id | description
---------------------
1 | Once upon a time, in a galaxy...
My question is now that we have thousands of rows of data in the database, how do I write a function or perform a query to replace the large object id with the actual text content stored in the large object?
Special thanks to #BohuslavBurghardt for pointing me to this answer. For your convenience:
UPDATE table_name SET column_name = lo_get(cast(column_name as bigint))
I needed some additional conversion:
UPDATE table_name SET text_no_lob = convert_from(lo_get(text::oid), 'UTF8');
I had the same problem with Spring, Postgres and JPA (Hibernate). I had a payload field that was like below
#NotBlank
#Column(name = "PAYLOAD")
private String payload;
I wanted to change the data type to text to support large data. So I used #Lob and I got the same error. To resolve that I first changed my field in my Entity like below:
#NotBlank
#Column(name = "PAYLOAD")
#Lob
#Type(type = "org.hibernate.type.TextType")
private String payload;
And because my data in this column was some scalar(Number) I have changed it to normal text with below command in Postgres:
UPDATE MYTABLE SET PAYLOAD = lo_get(cast(PAYLOAD as bigint))
Thanks a lot #Sipder.
Hi am parsing an xml file through JAXB and saving the data in the database table and am able to do this appropriately. My question is if the xml file returns an empty data for a particular field it should display as === in the database table. How can I do this while processing the xml file.
The xml file has two nodes, abc and xyz and the xml file should contain any one of these.There are two coloumns available in the database say name and version. these two coloumns will be derived by comparing the abc and xyz nodes of teh xml file in the database by using a common id and fetch the values for name and title. Can some one please help me in understanding how to handle this by processing the xml file.
I Wish I could post the code, but the code is too huge to post it.
What you are asking sounds weird, but assuming your data is a String, you could do this:
if (data == null || data.isEmpty())
{
data = "===";
}
From the tags on your question you appear to have two steps in your processing:
XML to object using JAXB
object to database using ?
I would put the logic for storing === in the database as part of the object-to-database processing. If you are using JPA 2.1 (part of Java EE 7) you could look at JPA converters to encapsulate this logic.
I am working with a 3rd product called JPOS and it has an XMLPackager whereby I get a string from this packager that contains a record in an XML format such as:
<MACHINE><B000>STRING_VALUE</B000><B002>STRING_VALUE</B002><B003>STRING_VALUE</B003><B004>STRING_VALUE</B004><B007>STRING_VALUE</B007><B011>STRING_VALUE</B011><B012>STRING_VALUE</B012><B013>STRING_VALUE</B013><B015>STRING_VALUE</B015><B018>STRING_VALUE</B018><B028>STRING_VALUE</B028><B032>STRING_VALUE</B032><B035>STRING_VALUE</B035><B037>STRING_VALUE</B037><B039>STRING_VALUE</B039><B041>STRING_VALUE</B041><B043>STRING_VALUE</B043><B048>STRING_VALUE</B048><B049>STRING_VALUE</B049><B058>STRING_VALUE</B058><B061>STRING_VALUE</B061><B063>STRING_VALUE</B063><B127>STRING_VALUE</B127></MACHINE>
I have a SQL server table that contains a column for each of the listed. Not that it matters but I could potentially have thru defined with specific STRING_VALUEs. I'm not sure what is the best way to go about this in Java. My understanding is that SQL Server can take an XML string (not document) and do an insert. Is it best to parse each value and then put into a list that populate each value into? This is the first time I've used an XML file and therefore trying to get some help/direction.
Thanks.
Sorry, one of my colleagues was able to help and provide a quick answer. I'll try it from my Java code and it looks like it should work great. Thanks anyway.
Here is the SP that she created whereby I can pass in my XML string and bit value:
CREATE PROCEDURE [dbo].[sbssp_InsertArchivedMessages]
(
#doc varchar(max),
#fromTo bit
)
AS
BEGIN
DECLARE #idoc int, #lastId int
EXEC sp_xml_preparedocument #idoc OUTPUT, #doc
INSERT INTO [dbo].[tblArchivedMessages]
SELECT * FROM OPENXML(#idoc, '/MACHINE', 2) WITH [dbo].[tblArchivedMessages]
SET #lastId = (SELECT IDENT_CURRENT('tblArchivedMessages'))
UPDATE [dbo].[tblArchivedMessages]
SET FromToMach = #fromTo
WHERE ID = #lastId
END
GO
Regards.