We Keep Coding

Parsing a text file using java with multiple values per line to be extracted

I'm not going to lie I'm really bad at making regular expressions. I'm currently trying to parse a text file that is giving me a lot of issues. The goal is to extract the data between their respective "tags/titles". The file in question is a .qbo file laid out as follows personal information replaced with "DATA": The parts that I care about retrieving are between the "STMTTRM" and "/STMTTRM" tags as the rest I don't plan on putting in my database, but I figured it would help others see the file content I'm working with. I apologize for any confusion prior to this update.
FXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
<OFX>
<SIGNONMSGSRSV1><SONRS>
<STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY></STATUS>
<DTSERVER>20190917133617.000[-4:EDT]</DTSERVER>
<LANGUAGE>ENG</LANGUAGE>
<FI>
<ORG>DATA</ORG>
<FID>DATA</FID>
</FI>
<INTU.BID>DATA</INTU.BID>
<INTU.USERID>DATA</INTU.USERID>
</SONRS></SIGNONMSGSRSV1>
<BANKMSGSRSV1>
<STMTTRNRS>
<TRNUID>0</TRNUID>
<STATUS><CODE>0</CODE><SEVERITY>INFO</SEVERITY></STATUS>
<STMTRS>
<CURDEF>USD</CURDEF>
<BANKACCTFROM>
<BANKID>DATA</BANKID>
<ACCTID>DATA</ACCTID>
<ACCTTYPE>CHECKING</ACCTTYPE>
<NICKNAME>FREEDOM CHECKING</NICKNAME>
</BANKACCTFROM>
<BANKTRANLIST>
<DTSTART>20190717</DTSTART><DTEND>20190917</DTEND>
<STMTTRN><TRNTYPE>POS</TRNTYPE><DTPOSTED>20190717071500</DTPOSTED><TRNAMT>-5.81</TRNAMT><FITID>3893120190717WO</FITID><NAME>DATA</NAME><MEMO>POS Withdrawal</MEMO></STMTTRN>
<STMTTRN><TRNTYPE>DIRECTDEBIT</TRNTYPE><DTPOSTED>20190717085000</DTPOSTED><TRNAMT>-728.11</TRNAMT><FITID>4649920190717WE</FITID><NAME>CHASE CREDIT CRD</NAME><MEMO>DATA</MEMO></STMTTRN>
<STMTTRN><TRNTYPE>ATM</TRNTYPE><DTPOSTED>20190717160900</DTPOSTED><TRNAMT>-201.99</TRNAMT><FITID>6674020190717WA</FITID><NAME>DATA</NAME><MEMO>ATM Withdrawal</MEMO></STMTTRN>
</BANKTRANLIST>
<LEDGERBAL><BALAMT>2024.16</BALAMT><DTASOF>20190917133617.000[-4:EDT]</DTASOF></LEDGERBAL>
<AVAILBAL><BALAMT>2020.66</BALAMT><DTASOF>20190917133617.000[-4:EDT]</DTASOF></AVAILBAL>
</STMTRS>
</STMTTRNRS>
</BANKMSGSRSV1>
</OFX>
I want to be able to end with data that looks or acts like the following so that each row of data can easily be added to a database:
Example Parse

As David has already answered, It is good to parse the POS output XML using Java. If you are more interested about about regex to get all the information, you can use this regular expression.
<[^>]+>|\\n+
You can test in the following sites.
https://rubular.com/
https://www.regextester.com/

Given this is XML, I would do one of two things:
either use the Java DOM objects to marshall/unmarshall to/from Java objects (nodes and elements), or
use JAXB to achieve something similar but with better POJO representation.
Mkyong has tutorials for both. Try the dom parsing or jaxb. His tutorials are simple and easy to follow.
JAXB requires more work and dependencies. So try DOM first.

I would propose the following approach.
Read file line by line with Files:
final List<String> lines = Files.readAllLines(Paths.get("/path/to/file"));
At this point you would have all file line separated and ready to convert the string lines into something more useful. But you should create class beforehand.
Create a class for your data in line, something like:
public class STMTTRN {
private String TRNTYPE;
private String DTPOSTED;
...
...
//constructors
//getters and setters
}
Now when you have a data in each separate string and a class to hold the data, you can convert lines to objects with Jackson:
final XmlMapper xmlMapper = new XmlMapper();
final STMTTRN stmttrn = xmlMapper.readValue(lines[0], STMTTRN.class);
You may want to create a loop or make use of stream with a mapper and a collector to get the list of STMTTRN objects:
final List<STMTTRN> stmttrnData = lines.stream().map(this::mapLine).collect(Collectors.toList());
Where the mapper might be:
private STMTTRN mapLine(final String line) {
final XmlMapper xmlMapper = new XmlMapper();
try {
return xmlMapper.readValue(line, STMTTRN.class);
} catch (IOException e) {
throw new RuntimeException(e);
}
}

Can I transform a JSON-LD to a Java object?

EDIT: I changed my mind. I would find a way to generate the Java class and load the JSON as an object of that class.
I just discovered that exists a variant of JSON called JSON-LD.
It seems to me a more structured way of defining JSON, that reminds me XML with an associated schema, like XSD.
Can I create a Java class from JSON-LD, load it at runtime and use it to convert JSON-LD to an instantiation of that class?
I read the documentation of both the implementations but I found nothing about it. Maybe I read them bad?

Doing a Google search brought me to a library that will decode the JSON-LD into an "undefined" Object.
// Open a valid json(-ld) input file
InputStream inputStream = new FileInputStream("input.json");
// Read the file into an Object (The type of this object will be a List, Map, String, Boolean,
// Number or null depending on the root object in the file).
Object jsonObject = JsonUtils.fromInputStream(inputStream);
// Create a context JSON map containing prefixes and definitions
Map context = new HashMap();
// Customise context...
// Create an instance of JsonLdOptions with the standard JSON-LD options
JsonLdOptions options = new JsonLdOptions();
// Customise options...
// Call whichever JSONLD function you want! (e.g. compact)
Object compact = JsonLdProcessor.compact(jsonObject, context, options);
// Print out the result (or don't, it's your call!)
System.out.println(JsonUtils.toPrettyString(compact));
https://github.com/jsonld-java/jsonld-java
Apparently, it can take it from just a string as well, as if reading it from a file or some other source. How you access the contents of the object, I can't tell. The documentation seems to be moderately decent, though.
It seems to be an active project, as the last commit was only 4 days ago and has 30 contributors. The license is BSD 3-Clause, if that makes any difference to you.
I'm not in any way associate with this project. I'm not an author nor have I made any pull requests. It's just something I found.
Good luck and I hope this helped!

see this page: JSON-LD Module for Jackson

Get EC2 Instance XML Description using AWS Java SDK?

We have a scenario in which we need to retrieve the description info for EC2 instances running on AWS. To accomplish this, we are using the AWS Java SDK. In 90% of our use case, the com.amazonaws.services.ec2.model.Instance class is exactly what we need. However, there is also a small use-case where it would be beneficial to get the raw XML describing the instance. That is, the XML data before it is converted into the Instance object. Is there any way to obtain both the Instance object and the XML string using the AWS Java SDK? Is there a way to manually convert from one to the other? Or, would we be forced to make a separate call using HttpClient or something similar to get the XML data?

Make an EC2Client by adding request handler and override the beforeUnmarshalling() method like below
AmazonEC2ClientBuilder.standard().withRegion("us-east-1")
.withRequestHandlers(
new RequestHandler2() {
#Override
public HttpResponse beforeUnmarshalling(Request<?> request, HttpResponse httpResponse) {
// httpResponse.getContent() is the raw xml response from AWS
// you either save it to a file or to a XML document
return new HTTPResponse(...);
// if you consumed httpResponse.getContent(), you need to provide new HTTPResponse
}
}
).build():

If you have xml (e.g. from using AWS rest API directly), then you can use com.amazonaws.services.ec2.model.transform.* classes to convert xml to java objects. Unfortunately, it only provides classes required for SDK itself. So you, for example, can convert raw XML to an Instance using InstanceStaxUnmarshaller, but can't convert Instance to XML unless you write such converter.
Here is an example how to parse an Instance XML:
XMLEventReader eventReader = XMLInputFactory.newInstance().createXMLEventReader(new StringReader(instanceXml));
StaxUnmarshallerContext suc = new StaxUnmarshallerContext(eventReader, new TreeMap<>());
InstanceStaxUnmarshaller isu = new InstanceStaxUnmarshaller();
Instance i = isu.unmarshall(suc);
System.out.println(i.toString());
You probably can try to intercept raw AWS response, so that you can keep raw XML while still using SDK most of the time. But I wouldn't call that easy as it will require quite a bit of coding.

You could use JAXB.marshal like following. JAXB (Java Architecture for XML Binding) could convert Java object to / from XML file.
StringWriter sw = new StringWriter();
JAXB.marshal(instance, sw);
String xmlString = sw.toString();

You can use AWS rest API to replace Java SDK. A bonus will be slight performance gain because you'll not send statistic data to Amazon as the SDK does.

Best approach to serialize XML to stream with Java?

We serialize/deserialize XML using XStream... and just got an OutOfMemory exception.
Firstly I don't understand why we're getting the error as we have 500MB allocated to the server.
Question is - what changes should we make to stay out of trouble? We want to ensure this implementation scales.
Currently we have ~60K objects, each ~50 bytes. We load the 60K POJO's in memory, and serialize them to a String which we send to a web service using HttpClient. When receiving, we get the entire String, then convert to POJO's. The XML/object hierarchy is like:
<root>
<meta>
<date>10/10/2009</date>
<type>abc</type>
</meta>
<data>
<field>x</field>
</data>
[thousands of <data>]
</root>
I gather the best approach is to not store the POJO's in memory and not write the contents to a single String. Instead we should write the individual <data> POJO's to a stream. XStream supports this but seems like the <meta> element wouldn't be supported. Data would need to be in form:
<root>
<data>
<field>x</field>
</data>
[thousands of <data>]
</root>
So what approach is easiest to stream the entire tree?

You definitely want to avoid serializing your POJOs into a humongous String and then writing that String out. Use the XStream APIs to serialize the POJOs directly to your OutputStream. I ran into the same situation earlier this year when I found that I was generating 200-300Mb XML documents and getting OutOfMemoryErrors. It was very easy to make the switch.
And ditto of course for the reading side. Don't read the XML into a String and ask XStream to deserialize from that String: deserialize directly from the InputStream.
You mention a second issue regarding not being able to serialize the <meta> element and the <data> elements. I don't think this is an XStream problem or limitation as I routinely serialize much more complex structures on the order of:
<myobject>
<item>foo</item>
<anotheritem>foo</anotheritem>
<alist>
<alistitem>
<value1>v1</value1>
<value2>v2</value2>
<value3>v3</value3>
...
</alistitem>
...
<alistitem>
<value1>v1</value1>
<value2>v2</value2>
<value3>v3</value3>
...
</alistitem>
</alist>
<anotherlist>
<anotherlistitem>
<valA>A</valA>
<valB>B</valB>
<valC>C</valC>
...
</anotherlistitem>
...
</anotherlist>
</myobject>
I've successfully serialized and deserialized nested lists too.

Not sure what the problem is here...you've found your answer on that webpage.
The example code on the link you provided suggests:
Writer someWriter = new FileWriter("filename.xml");
ObjectOutputStream out = xstream.createObjectOutputStream(someWriter, "root");
out.writeObject(dataObject);
// iterate over your objects...
out.close();
and for reading nearly identical but with Reader for Writer and Input for Output:
Reader someReader = new FileReader("filename.xml");
ObjectInputStream in = xstream.createObjectInputStream(someReader);
DataObject foo = (DataObject)in.readObject();
// do some stuff here while there's more objects...
in.close();

I'd suggest using tools like Visual VM or Eclipse Memory Analyzer to make sure you don't have a memory leak/problem.
Also, how do you know each object is 50 bytes? That doesn't sound likely.

Use XMLStreamWriter (or XStream) to serialize it, you can write whatever you want on it. If you have the option of getting the input stream instead of the entire string, use a SAXParser, it is event based and, although the implementation maybe a little bit clumsy, you will be able to read any XML that is thrown at you, even if it the XML is huge (I have parse 2GB+ more XML files with SAXParser).
Just as a side note, you should send the binary data and not the string to a XML parser. XML parsers will read the encoding of the byte array that is going to come next through the xml tag in the beginning of the XML sequence:
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
A string is encoded in something already. It's better practice to let the XML parse the original stream before you create a String with it.

Reading an XML File using FileInputStream (for Java)?

For my project I have to serialize and deserialize a random tree using Java and XStream. My teacher made the Tree/RandomTree algorithms, so I don't have to worry about that. What I don't know how to do is this: I am using FileInputStream to read/write the xml file that I serialized and deserialized, but when I deserialize, I do not know the method used to read the file. After I read the file I should be able to convert it from XML and then print it out as a string. Here's what I have so far. (I imported everything correctly, just didn't add it to my code segment).
FileInputStream fin;
try
{
// Open an input stream
fin = new FileInputStream ("/Users/Pat/programs/randomtree.xml");
//I don't know what to put below this, to read FileInpuStream object fin
String dexml = (String)xstream.fromXML(fin);
System.out.println(dexml);
// Close our input stream
fin.close();
System.out.println(dexml);
// Close our input stream
fin.close();
}
// Catches any error conditions
catch (IOException e)
{
System.err.println ("Unable to read from file");
System.exit(-1);
}
Edit: I figured it out; I don't think I have to print it as a string, I just needed to make a benchmarking framework to time it and such, but thanks again!

The xstream.fromXML() method will do the reading from the input stream for you. I think the problem is that you are casting the return value from xstream.fromXML(fin) into a String when it should be cast to the type of object you originally serialized (RandomTree I assume). So the code would look like this:
RandomTree tree = (RandomTree)xstream.fromXML(fin);
EDIT: after clarification in comments, the author's goal is to first read into a String so the XML contents can be printed before deserialization. With that goal in mind, I recommend taking a look at the IOUtils library mentioned in this thread

From what I understand from http://x-stream.github.io/tutorial.html (I've never worked with XStream before), you need to define your types first. Casting to String is definitely wrong, you probably want a customized type (depending on what's inside your random XML), then you need to map the XML tags to your members:
e.g.
xstream.alias("person", Person.class);
xstream.alias("phonenumber", PhoneNumber.class);
meaning that it maps the "person" tag inside your XML to your Person class.
To derserialize, you can do:
RandomTree myRandomTree = (RandomTree)xstream.fromXML( xml );
Also, you are closing your stream twice, and you probably want to do it in a finally block :)
edit: Having read your comment above...
Your task involves two steps:
Deserialization
Serialization
In order to serialize your object, you must deserialize it first from your input file.
To output your Object as String, simply do
String xml = xstream.toXML( myRandomTree );