We Keep Coding

Digitally sign XML (XAdES ) with timestamping by streaming the XML

I have created an method that digitally signs an xml (XAdES with Timestamping) using xades4j library https://github.com/luisgoncalves/xades4j
Unfortunately the xml's are quite big sometimes (1,8 GB) and I was wondering if there is a way to do that by streaming the XML instead of creating a DOM and loading the whole document in memory. Is there a way? Can I do that with xades4j?
Below is the current code that signs the document using a DOM representation of the xml. The initial method that is called first is the signXml().
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.KeyStoreException;
import javax.annotation.PostConstruct;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.apache.commons.io.FilenameUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import com.safe.AbstractManager;
import com.safe.model.DirectPasswordProvider;
import xades4j.algorithms.ExclusiveCanonicalXMLWithoutComments;
import xades4j.production.Enveloped;
import xades4j.production.SignatureAlgorithms;
import xades4j.production.XadesSigner;
import xades4j.production.XadesTSigningProfile;
import xades4j.providers.KeyingDataProvider;
import xades4j.providers.impl.FileSystemKeyStoreKeyingDataProvider;
import xades4j.providers.impl.HttpTsaConfiguration;
import xades4j.providers.impl.KeyStoreKeyingDataProvider;
import xades4j.utils.DOMHelper;
#Component
public class FileOperationsManager extends AbstractManager {
#Value("${certificates.digital-signature.filepath}")
private String certPath;
#Value("${certificates.digital-signature.password}")
private String certPass;
#Value("${certificates.digital-signature.type}")
private String certType;
private DocumentBuilder db;
private TransformerFactory tf;
#PostConstruct
public void init() throws Exception {
final DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setNamespaceAware(true);
this.db = dbf.newDocumentBuilder();
this.tf = TransformerFactory.newInstance();
}
public Path signXml(final Path xmlFile, final Path targetDir) {
final String baseName = FilenameUtils.getBaseName(xmlFile.getFileName().toString())
.concat("_Signed")
.concat(".")
.concat(FilenameUtils.getExtension(xmlFile.getFileName().toString()));
final Path target = Paths.get(targetDir.toString(), baseName);
try (final FileInputStream fis = new FileInputStream(String.valueOf(xmlFile))) {
final Document doc = this.parseDocument(fis);
final Element elementToSign = doc.getDocumentElement();
final SignatureAlgorithms algorithms = new SignatureAlgorithms()
.withCanonicalizationAlgorithmForTimeStampProperties(new ExclusiveCanonicalXMLWithoutComments("ds", "xades"))
.withCanonicalizationAlgorithmForSignature(new ExclusiveCanonicalXMLWithoutComments());
final KeyingDataProvider kdp = this.createFileSystemKeyingDataProvider(certType, certPath, certPass, true);
final XadesSigner signer = new XadesTSigningProfile(kdp)
.withSignatureAlgorithms(algorithms)
.with(new HttpTsaConfiguration("http://timestamp.digicert.com"))
.newSigner();
new Enveloped(signer).sign(elementToSign);
this.exportDocument(doc, target);
} catch (final FileNotFoundException e) {
throw new RuntimeException();
} catch (final Exception e) {
throw new RuntimeException();
}
return target;
}
private FileSystemKeyStoreKeyingDataProvider createFileSystemKeyingDataProvider(
final String keyStoreType,
final String keyStorePath,
final String keyStorePwd,
final boolean returnFullChain) throws KeyStoreException {
return FileSystemKeyStoreKeyingDataProvider
.builder(keyStoreType, keyStorePath, KeyStoreKeyingDataProvider.SigningCertificateSelector.single())
.storePassword(new DirectPasswordProvider(keyStorePwd))
.entryPassword(new DirectPasswordProvider(keyStorePwd))
.fullChain(returnFullChain)
.build();
}
public Document parseDocument(final InputStream is) {
try {
final Document doc = this.db.parse(is);
final Element elem = doc.getDocumentElement();
DOMHelper.useIdAsXmlId(elem);
return doc;
} catch (final Exception e) {
throw new RuntimeException();
}
}
public void exportDocument(final Document doc, final Path target) {
try (final FileOutputStream out = new FileOutputStream(target.toFile())) {
this.tf.newTransformer().transform(
new DOMSource(doc),
new StreamResult(out));
} catch (final Exception e) {
throw new RuntimeException();
}
}

Unfortunately xades4j doesn't support streaming on the XML document to which the signature will be appended. I don't know if there are other alternative libraries that do.
A possible workaround using xades4j is to use a detached signature instead of an enveloped signature. The signature can be added to an empty XML document and the large XML file is explicitly added as a Reference to that signature.
xades4j delegates the core XML-DSIG handling to Apache Santuario, so if Santuario uses streaming for Reference resolution, this should avoid your issue. I'm not sure, though, but it may be worth testing.
https://github.com/luisgoncalves/xades4j/wiki/DefiningSignedResources
You may need to use a file URI and/or base URIs.

Parse String to XML Document jdom2

I have a problem. I want to convert the string to a xml document.
But this code is throw exception:
Exception in thread "main" java.lang.ClassCastException: com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl cannot be cast to org.jdom2.Document
at ru.unicus.osp.Test.main(Test.java:17)
import java.io.StringReader;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.jdom2.Document;
import org.xml.sax.InputSource;
public class Test {
public static void main(String[] args) throws Exception {
DocumentBuilder db = DocumentBuilderFactory.newInstance().newDocumentBuilder();
InputSource is = new InputSource();
is.setCharacterStream(new StringReader("<root><node1></node1></root>"));
Document doc = (Document) db.parse(is);
}
}
Got any ideas?

You mixed two libraries here. The simplest thing that can be done is to change the import from:
import org.jdom2.Document;
to
import org.w3c.dom.Document;
So the code should look like:
import java.io.StringReader;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import org.w3c.dom.Document;
import org.xml.sax.InputSource;
public class Test {
public static void main(String[] args) throws Exception {
DocumentBuilder db = DocumentBuilderFactory.newInstance().newDocumentBuilder();
InputSource is = new InputSource();
is.setCharacterStream(new StringReader("<root><node1></node1></root>"));
Document doc = (Document) db.parse(is);
}
}

You are mixing two libraries here, DOM, and JDOM.
your current output Document is a JDOM document, it seems, but you are trying to cast a w3 DOM Document instance to be one, and you are failing.
If you want to have a JDOM Document as your output, then your code should likely be:
SAXBuilder builder = new SAXBuilder();
Document doc = builder.build(new StringReader("<root><node1></node1></root>"));
For a DOM document output, just use a DOM Document (org.w3c.dom.Document) instead of org.jdom2.Document

W3C module does not exist

I try to do Y! Weather XML parsing for the sake of learning using BlueJ. When I compiled the class, BlueJ giving me error: package org.w3c does not exist
Here is the code:
import java.net.URL;
import java.io.InputStream;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.DocumentBuilder;
import org.w3c.*;
public class WeatherApp
{
public static void main(String[] args)
{
URL xmlURL = new URL("http://weather.yahooapis.com/forecastrss?w=22722924&u=c");
InputStream xml = xmlURL.openStream();
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder db = dbf.newDocumentBuilder();
Document doc = db.parse(xml);
xml.close();
System.out.println(doc);
}
}
What could go possibly wrong here? And how to fix it? Thanks

Reading XML online and Storing It (Using Java)

I found and followed an example from Stackoverflow (http://stackoverflow.com/questions/2310139/how-to-read-xml-response-from-a-url-in-java) of how to read an XML file from a URL (as you can see in my code pasted below). My only trouble is that now that I got the program to read the XML, how do I get it to store it? For example, could I make it save the information to a XML file built into the project (this would be the best solution for me, if it's possible)? Such as, take for example, I have a blank XML file built into the project. The program runs, reads the XML code off of the URL, and stores it all into the pre-built blank XML file. Could I do this?
If I sound confusing or un-clear about anything, just ask me to clarify what I'm looking for.
And here is my code, if you'd like to look at what I have so far:
package xml.parsing.example;
import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.xml.sax.SAXException;
public class XmlParser {
public static void main (String[] args) throws IOException, ParserConfigurationException, SAXException, TransformerException {
URL url = new URL("http://totheriver.com/learn/xml/code/employees.xml");
URLConnection conn = url.openConnection();
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
Document doc = builder.parse(conn.getInputStream());
TransformerFactory tfactory = TransformerFactory.newInstance();
Transformer xform = tfactory.newTransformer();
// that’s the default xform; use a stylesheet to get a real one
xform.transform(new DOMSource(doc), new StreamResult(System.out));
}
}

Very simply:
File myOutput = new File("c:\\myDirectory\\myOutput.xml");
xform.transform(new DOMSource(doc), new StreamResult(myOutput));

This page has some great examples of how to serialize the DOM object to a neatly formatted XML file.

how to disable dtd at runtime in java's xpath?

I got dtd in file and I cant remove it. When i try to parse it in Java I get "Caused by: java.net.SocketException: Network is unreachable: connect", because its remote dtd. can I disable somehow dtd checking?

You should be able to specify your own EntityResolver, or use specific features of your parser? See here for some approaches.
A more complete example:
<?xml version="1.0"?>
<!DOCTYPE foo PUBLIC "//FOO//" "foo.dtd">
<foo>
<bar>Value</bar>
</foo>
And xpath usage:
import java.io.File;
import java.io.IOException;
import java.io.StringReader;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathFactory;
import org.w3c.dom.Document;
import org.xml.sax.EntityResolver;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
public class Main {
public static void main(String[] args) throws Exception {
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
builder.setEntityResolver(new EntityResolver() {
#Override
public InputSource resolveEntity(String publicId, String systemId)
throws SAXException, IOException {
System.out.println("Ignoring " + publicId + ", " + systemId);
return new InputSource(new StringReader(""));
}
});
Document document = builder.parse(new File("src/foo.xml"));
XPathFactory xpathFactory = XPathFactory.newInstance();
XPath xpath = xpathFactory.newXPath();
String content = xpath.evaluate("/foo/bar/text()", document
.getDocumentElement());
System.out.println(content);
}
}
Hope this helps...

This worked for me:
SAXParserFactory saxfac = SAXParserFactory.newInstance();
saxfac.setValidating(false);
try {
saxfac.setFeature("http://xml.org/sax/features/validation", false);
saxfac.setFeature("http://apache.org/xml/features/nonvalidating/load-dtd-grammar", false);
saxfac.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);
saxfac.setFeature("http://xml.org/sax/features/external-general-entities", false);
saxfac.setFeature("http://xml.org/sax/features/external-parameter-entities", false);
}
catch (Exception e1) {
e1.printStackTrace();
}

I had this problem before. I solved it by downloading and storing a local copy of the DTD and then validating against the local copy. You need to edit the XML file to point to the local copy.
<!DOCTYPE root-element SYSTEM "filename">
Little more info here: http://www.w3schools.com/dtd/dtd_intro.asp
I think you can also manually set some sort of validateOnParse property to "false" in your parser. Depends on what library you're using to parse the XML.
More info here: http://www.w3schools.com/dtd/dtd_validation.asp