We Keep Coding

How to parse mulitple XML files in Java with XMLStreamReader

We have to parse xml info from a generator that creates fake weather data for a bunch of weatherstations. Currently we're just printing it, but we'll have to do stuff with it later.
However, the data we receive consists of multiple XML "files". Is there a way to separate the data and split it at a new <?xml...?>? (The data is a continuous stream that randomly splits)
our code:
public class Main {
static private final int portNumber = Null;
public static void main(String[] args) {
try {
ServerSocket serverSocket = new ServerSocket(portNumber);
Socket clientSocket = serverSocket.accept();
BufferedReader clientReader = new BufferedReader(
new InputStreamReader(clientSocket.getInputStream()));
XMLInputFactory factory = XMLInputFactory.newInstance();
XMLStreamReader reader = factory.createXMLStreamReader(clientReader);
while (reader.hasNext()) {
int event = reader.next();
if (event == XMLStreamConstants.START_ELEMENT) {
try {
String text = reader.getElementText();
System.out.println("Element Local Name:" + reader.getLocalName());
System.out.println("Text:" + text);
} catch (XMLStreamException e) {
System.out.println(e);
}
}
else if(event == XMLStreamConstants.END_ELEMENT){
reader.close();
}
}
} catch (IOException e) {
System.out.println("Error: Unable to Start Server Socket\n\t" + e);
} catch (XMLStreamException e){
System.out.println(e);
}
}
}
example of the xml (of which we receive multiple after each other):
<?xml version="1.0"?>
<!-- The WEATHERDATA-element contains multiple MEASUREMENT-elements -->
<WEATHERDATA>
<MEASUREMENT>
<STN>123456</STN>
<DATE>2009-09-13</DATE>
<TIME>15:59:46</TIME>
<TEMP>-60.1</TEMP>
<DEWP>-58.1</DEWP>
<STP>1034.5</STP>
<SLP>1007.6</SLP>
<VISIB>123.7</VISIB>
<WDSP>10.8</WDSP>
<PRCP>11.28</PRCP>
<SNDP>11.1</SNDP>
<FRSHTT>010101</FRSHTT>
<CLDC>87.4</CLDC>
<WNDDIR>342</WNDDIR>
</MEASUREMENT>
</WEATHERDATA>
We also have a dtd file but I'm not sure if that's helpful.

Using java.util.Scanner may serve as a quick workaround. The disassemble() function skips the XML declaration if present and combines all characters up to and including the next closing </WEATHERDATA> tag into a `String'. The result is then passed to the callback which in this example converts XML into a POJO with JAXB.
What I don't like about Scanner is that it internally buffers the input stream so it is possible to lose the last message when the stream is closed.
public class DisassembleXml {
private static final int port = 8888;
private static final Pattern XML_DECL_PATTERN = Pattern.compile("<\\?xml.*?\\?>");
private static final Pattern DATA_PATTERN =
Pattern.compile(".*?</WEATHERDATA>\\s+", Pattern.DOTALL);
public static void main(String[] args) throws Exception {
final ServerSocket serverSocket = new ServerSocket(port);
System.out.printf("Listening on %d%n", serverSocket.getLocalPort());
final Socket clientSocket = serverSocket.accept();
System.out.printf("Processing from %s%n", clientSocket);
try (Reader sr = new InputStreamReader(clientSocket.getInputStream(), StandardCharsets.ISO_8859_1))
{
disassemble(sr, new ConvertToPojoAndPrint());
}
catch (Exception e)
{
e.printStackTrace();
}
}
private static void disassemble(Reader reader, Consumer<String> xmlConsumer) {
final Scanner sc = new Scanner(reader).useDelimiter("\\Z");
try {
while (true) {
final String xml = sc
.skip(XML_DECL_PATTERN)
.findWithinHorizon(DATA_PATTERN, 0);
if (xml == null || xml.isEmpty())
break;
xmlConsumer.accept(xml);
}
}
catch (Exception e) {
throw new IllegalStateException("cannot interpret stream", e);
}
}
private static class ConvertToPojoAndPrint implements Consumer<String>
{
final JAXBContext jaxbContext;
final Unmarshaller unmarshaller;
ConvertToPojoAndPrint() throws JAXBException {
jaxbContext = JAXBContext.newInstance(WeatherData.class);
unmarshaller = jaxbContext.createUnmarshaller();
}
#Override
public void accept(String xml) {
try {
final WeatherData weatherData = (WeatherData) unmarshaller.unmarshal(new StringReader(xml));
System.out.println("Another sample: " + weatherData);
}
catch (Exception e) {
throw new IllegalStateException(e);
}
}
}
#XmlRootElement(name = "WEATHERDATA")
private static class WeatherData
{
#XmlElement(name = "MEASUREMENT")
Measurement measurement;
#Override
public String toString() { return "WeatherData{" + "measurement=" + measurement + '}'; }
}
private static class Measurement
{
#XmlElement(name = "STN")
String stn;
// ... skipping the rest of elements for brevity
#Override
public String toString() { return "Measurement{" + "stn='" + stn + '\'' + '}'; }
}
}

Jsoup.connect().get() takes only part of html file on Android

So, I try to parse Wikipedia, and my code works well at computer.
All, what I changed - .connect().get is in AsyncTask, but I get only part of html file (no "body", only half of second "script" in "title") and I can't understand why.
This is my code example for Android.
protected String doInBackground(String... params) {
try {
Document doc = Jsoup.connect(params[0]).get();
return doc.toString();
} catch (IOException e) {
//...
e.printStackTrace();
}
return null;
}
And this is simple.
String url = "https://en.wikipedia.org/wiki/Protectorate";
Document doc = null;
try {
doc = Jsoup.connect(url).get();
} catch (IOException e) {
//...
e.printStackTrace();
}
I checked, params[0] is https://en.wikipedia.org/wiki/Protectorate, here's no mistake.
If you need some extra information, I will give it, of course.

Logcat fools us here, since it shortens the message (I assume you checked your string with logcat? See related question)
If you split your result string into chunks, you will see that the whole page was loaded. Try adding something like this logAll function to your AsyncTask class to see the full output:
private class DownloadTask extends AsyncTask<String, Integer, String> {
Document doc = null;
protected String doInBackground(String... params) {
try {
doc = Jsoup.connect(params[0]).get();
return doc.toString();
} catch (Exception e) {
e.printStackTrace();
}
return doc.toString();
}
#Override
protected void onPostExecute(String s) {
super.onPostExecute(s);
logAll("async",doc.toString());
}
void logAll(String TAG, String longString) {
int splitSize = 300;
if (longString.length() > splitSize) {
int index = 0;
while (index < longString.length()-splitSize) {
Log.e(TAG, longString.substring(index, index + splitSize));
index += splitSize;
}
Log.e(TAG, longString.substring(index, longString.length()));
} else {
Log.e(TAG, longString.toString());
}
}
}

How to Parse Big (50 GB) XML Files in Java

Currently im trying to use a SAX Parser but about 3/4 through the file it just completely freezes up, i have tried allocating more memory etc but not getting any improvements.
Is there any way to speed this up? A better method?
Stripped it to bare bones, so i now have the following code and when running in command line it still doesn't go as fast as i would like.
Running it with "java -Xms-4096m -Xmx8192m -jar reader.jar" i get a GC overhead limit exceeded around article 700000
Main:
public class Read {
public static void main(String[] args) {
pages = XMLManager.getPages();
}
}
XMLManager
public class XMLManager {
public static ArrayList<Page> getPages() {
ArrayList<Page> pages = null;
SAXParserFactory factory = SAXParserFactory.newInstance();
try {
SAXParser parser = factory.newSAXParser();
File file = new File("..\\enwiki-20140811-pages-articles.xml");
PageHandler pageHandler = new PageHandler();
parser.parse(file, pageHandler);
pages = pageHandler.getPages();
} catch (ParserConfigurationException e) {
e.printStackTrace();
} catch (SAXException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return pages;
}
}
PageHandler
public class PageHandler extends DefaultHandler{
private ArrayList<Page> pages = new ArrayList<>();
private Page page;
private StringBuilder stringBuilder;
private boolean idSet = false;
public PageHandler(){
super();
}
#Override
public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
stringBuilder = new StringBuilder();
if (qName.equals("page")){
page = new Page();
idSet = false;
} else if (qName.equals("redirect")){
if (page != null){
page.setRedirecting(true);
}
}
}
#Override
public void endElement(String uri, String localName, String qName) throws SAXException {
if (page != null && !page.isRedirecting()){
if (qName.equals("title")){
page.setTitle(stringBuilder.toString());
} else if (qName.equals("id")){
if (!idSet){
page.setId(Integer.parseInt(stringBuilder.toString()));
idSet = true;
}
} else if (qName.equals("text")){
String articleText = stringBuilder.toString();
articleText = articleText.replaceAll("(?s)<ref(.+?)</ref>", " "); //remove references
articleText = articleText.replaceAll("(?s)\\{\\{(.+?)\\}\\}", " "); //remove links underneath headings
articleText = articleText.replaceAll("(?s)==See also==.+", " "); //remove everything after see also
articleText = articleText.replaceAll("\\|", " "); //Separate multiple links
articleText = articleText.replaceAll("\\n", " "); //remove new lines
articleText = articleText.replaceAll("[^a-zA-Z0-9- \\s]", " "); //remove all non alphanumeric except dashes and spaces
articleText = articleText.trim().replaceAll(" +", " "); //convert all multiple spaces to 1 space
Pattern pattern = Pattern.compile("([\\S]+\\s*){1,75}"); //get first 75 words of text
Matcher matcher = pattern.matcher(articleText);
matcher.find();
try {
page.setSummaryText(matcher.group());
} catch (IllegalStateException se){
page.setSummaryText("None");
}
page.setText(articleText);
} else if (qName.equals("page")){
pages.add(page);
page = null;
}
} else {
page = null;
}
}
#Override
public void characters(char[] ch, int start, int length) throws SAXException {
stringBuilder.append(ch,start, length);
}
public ArrayList<Page> getPages() {
return pages;
}
}

Your parsing code is likely working fine, but the volume of data you're loading is probably just too large to hold in memory in that ArrayList.
You need some sort of pipeline to pass the data on to its actual destination without ever
store it all in memory at once.
What I've sometimes done for this sort of situation is similar to the following.
Create an interface for processing a single element:
public interface PageProcessor {
void process(Page page);
}
Supply an implementation of this to the PageHandler through a constructor:
public class Read {
public static void main(String[] args) {
XMLManager.load(new PageProcessor() {
#Override
public void process(Page page) {
// Obviously you want to do something other than just printing,
// but I don't know what that is...
System.out.println(page);
}
}) ;
}
}
public class XMLManager {
public static void load(PageProcessor processor) {
SAXParserFactory factory = SAXParserFactory.newInstance();
try {
SAXParser parser = factory.newSAXParser();
File file = new File("pages-articles.xml");
PageHandler pageHandler = new PageHandler(processor);
parser.parse(file, pageHandler);
} catch (ParserConfigurationException e) {
e.printStackTrace();
} catch (SAXException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
Send data to this processor instead of putting it in the list:
public class PageHandler extends DefaultHandler {
private final PageProcessor processor;
private Page page;
private StringBuilder stringBuilder;
private boolean idSet = false;
public PageHandler(PageProcessor processor) {
this.processor = processor;
}
#Override
public void startElement(String uri, String localName, String qName, Attributes attributes) throws SAXException {
//Unchanged from your implementation
}
#Override
public void characters(char[] ch, int start, int length) throws SAXException {
//Unchanged from your implementation
}
#Override
public void endElement(String uri, String localName, String qName) throws SAXException {
// Elide code not needing change
} else if (qName.equals("page")){
processor.process(page);
page = null;
}
} else {
page = null;
}
}
}
Of course, you can make your interface handle chunks of multiple records rather than just one and have the PageHandler collect pages locally in a smaller list and periodically send the list off for processing and clear the list.
Or (perhaps better) you could implement the PageProcessor interface as defined here and build in logic there that buffers the data and sends it on for further handling in chunks.

Don Roby's approach is somewhat reminiscent to the approach I followed creating a code generator designed to solve this particular problem (an early version was conceived in 2008). Basically each complexType has its Java POJO equivalent and handlers for the particular type are activated when the context changes to that element. I used this approach for SEPA, transaction banking and for instance discogs (30GB). You can specify what elements you want to process at runtime, declaratively using a propeties file.
XML2J uses mapping of complexTypes to Java POJOs on the one hand, but lets you specify events you want to listen on.
E.g.
account/#process = true
account/accounts/#process = true
account/accounts/#detach = true
The essence is in the third line. The detach makes sure individual accounts are not added to the accounts list. So it won't overflow.
class AccountType {
private List<AccountType> accounts = new ArrayList<>();
public void addAccount(AccountType tAccount) {
accounts.add(tAccount);
}
// etc.
};
In your code you need to implement the process method (by default the code generator generates an empty method:
class AccountsProcessor implements MessageProcessor {
static private Logger logger = LoggerFactory.getLogger(AccountsProcessor.class);
// assuming Spring data persistency here
final String path = new ClassPathResource("spring-config.xml").getPath();
ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(path);
AccountsTypeRepo repo = context.getBean(AccountsTypeRepo.class);
#Override
public void process(XMLEvent evt, ComplexDataType data)
throws ProcessorException {
if (evt == XMLEvent.END) {
if( data instanceof AccountType) {
process((AccountType)data);
}
}
}
private void process(AccountType data) {
if (logger.isInfoEnabled()) {
// do some logging
}
repo.save(data);
}
}
Note that XMLEvent.END marks the closing tag of an element. So, when you are processing it, it is complete. If you have to relate it (using a FK) to its parent object in the database, you could process the XMLEvent.BEGIN for the parent, create a placeholder in the database and use its key to store with each of its children. In the final XMLEvent.END you would then update the parent.
Note that the code generator generates everything you need. You just have to implement that method and of course the DB glue code.
There are samples to get you started. The code generator even generates your POM files, so you can immediately after generation build your project.
The default process method is like this:
#Override
public void process(XMLEvent evt, ComplexDataType data)
throws ProcessorException {
/*
* TODO Auto-generated method stub implement your own handling here.
* Use the runtime configuration file to determine which events are to be sent to the processor.
*/
if (evt == XMLEvent.END) {
data.print( ConsoleWriter.out );
}
}
Downloads:
https://github.com/lolkedijkstra/xml2j-core
https://github.com/lolkedijkstra/xml2j-gen
https://sourceforge.net/projects/xml2j/
First mvn clean install the core (it has to be in the local maven repo), then the generator. And don't forget to set up the environment variable XML2J_HOME as per directions in the usermanual.

How to retrieve Message and Task body - Exchange Web Services - Java

I am creating a message using JWebServices, but even though all other fields are retrieved successfully, body does not. message.getBody() returns null. Here are the two methods I call subsequently.
private void createMessage(Service service) throws ParseException {
try {
Message message = new Message();
message.setItemClass(ItemClass.MESSAGE);
message.setSubject("Test");
message.setBody(new Body("Body text"));
message.getToRecipients().add(new Mailbox("John#mydomain.com"));
message.getCcRecipients().add(new Mailbox("Mark#mydomain.com"));
ItemId itemId = service.createItem(message,StandardFolder.SENT_ITEMS);
} catch (ServiceException e) {
System.out.println(e.getMessage());
System.out.println(e.getXmlMessage());
e.printStackTrace();
}
}
private void listItemsInSent(Service service) throws ParseException {
try {
FindItemResponse response = service.findItem(StandardFolder.SENT_ITEMS);
Message m = null;
for (int i = 0; i < response.getItems().size(); i++) {
m = (Message)response.getItems().get(i);
System.out.println(m.getSubject());
System.out.println(m.getItemClass());
System.out.println(m.getLastModifiedTime());
System.out.println(m.getBody());
System.out.println(m.getBodyHtmlText());
System.out.println(m.getBodyPlainText());
System.out.println(m.getItemId());
System.out.println(m.toString());
System.out.println();
}
} catch (ServiceException e) {
System.out.println(e.getMessage());
System.out.println(e.getXmlMessage());
e.printStackTrace();
}
}

Try to replace
m = (Message)response.getItems().get(i);
with
m = service.getMessage(response.getItems().get(i).getItemId());

how to parse some specific nodes of xml using sax or dom based on given condition

i am new to jaxp and i am stuck with this i have to parse this xml using either sax or dom.
i have to ask the user to enter empoyee code and then display the other nodes of that particular employee
for ex:if user enters 101 then i will show
Ename:akshay
ecode:101
dp no.:10
mgr code=201
i tried alot but no results,can somebody help.
<employees>
<employee>
<Ename>akshay</Ename>
<Ecode>101</Ecode>
<EmpSal>2100.0</EmpSal>
<Department_code>10</Department_code>
<Manager_code>201</Manager_code>
</employee>
<employee>
<Ename>rahul</Ename>
<Ecode>102</Ecode>
<EmpSal>21000.0</EmpSal>
<Department_code>20</Department_code>
<Manager_code>202</Manager_code>
</employee>
</employees>
the code i tried to parse this xml
public class ParseUsingDom {
public static void main(String[] args) {
try {
DocumentBuilderFactory factory = DocumentBuilderFactory
.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
Document document = builder.parse("emp.xml");
NodeList list = document.getElementsByTagName("*");
int count = 0;
{
for (int i = 0; i < list.getLength(); i++) {
Element element = (Element) list.item(i);
String nodeName = element.getNodeName();
if (nodeName.equals("employee")) {
count++;
System.out.println("Employee :" + count);
} else if (nodeName.equals("Ename")) {
System.out.println("\tEname:\t"
+ element.getChildNodes().item(0).getNodeValue());
} else if (nodeName.equals("Ecode")) {
System.out.println("\tECode:\t"
+ element.getChildNodes().item(0).getNodeValue());
} else if (nodeName.equals("EmpSal")) {
System.out.println("\tEmpsal:\t"
+ element.getChildNodes().item(0).getNodeValue());
} else if (nodeName.equals("Department_code")) {
System.out.println("\tDepartment_code: "
+ element.getChildNodes().item(0).getNodeValue());
} else if (nodeName.equals("Manager_code")) {
System.out.println("\tManager code:\t"
+ element.getChildNodes().item(0).getNodeValue());
}
}
}} catch (ParserConfigurationException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (SAXException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}

This is pretty straight forward.
Parse the XML as a Document with a DocumentBuilder.
Ask the user for an ID. If this is command line, then a Scanner might help.
Select the relevant data from the document. XPath is probably a good start.
Take these steps individually and figure out how to do them. Put it together and you've got your solution.