I have multiple processes running on different machines which are required to read/write to a shared XML file, for this I am using DOM with Java and FileLocks (While I know that a database would be a more effective approach, this is not viable due to project constraints) .
To make changes to the XML file, the relevant process first creates an exclusively locked channel which is used to read the file, it then attempts to reuse the same channel to write the new version before closing the channel; this way the lock is never down. The issue however is that I am getting a java.nio.channels.ClosedChannelException when attempting to write the result, even though I never explicitly close the channel. I have suspicions that the line of code:
doc = dBuilder.parse(Channels.newInputStream(channel));
closes the channel. If so, how could I force the channel to stay open? My code can be seen below:
[removed code after update]
UPDATE: Placing System.out.println(channel.isOpen()) before and after the suspect line of code confirms that this is where the channel is closed.
UPDATE: Having asked a separate question the code below now prevents the channel from closing during the parse operation. The issue now is that instead of replacing the original xml file, the transformer appends the changed document to the original. In the documentation I cannot find any related options for specifying the output of Transformer.transform (I have searched Transformer/Transformer factory/StreamResult). Am I missing something? Do I need to somehow clear the channel before writing? Thanks.
UPDATE: Finally solved the append issue by truncating the channel to a size of 0. Thank you #JLRishe for the advice. Have posted the working code as an answer.
This is the code which finally works! See question updates for explanations of different parts.
import java.io.*;
import java.nio.channels.*;
import javax.xml.parsers.*;
import javax.xml.transform.*;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import javax.xml.xpath.*;
import org.w3c.dom.*;
import org.xml.sax.SAXException;
public class Test2{
String path = "...Test 2.xml";
public Test2(){
Document doc = null;
DocumentBuilderFactory dbFactory;
DocumentBuilder dBuilder;
NodeList itemList;
Transformer transformer;
FileChannel channel;
Element newElement;
int prevNumber;
TransformerFactory transformerFactory ;
DOMSource source;
StreamResult result;
NonClosingInputStream ncis = null;
try {
channel = new RandomAccessFile(new File(path), "rw").getChannel();
FileLock lock = channel.lock(0L, Long.MAX_VALUE, false);
try {
dbFactory = DocumentBuilderFactory.newInstance();
dBuilder = dbFactory.newDocumentBuilder();
ncis = new NonClosingInputStream(Channels.newInputStream(channel));
doc = dBuilder.parse(ncis);
} catch (SAXException | IOException | ParserConfigurationException e) {
e.printStackTrace();
}
doc.getDocumentElement().normalize();
itemList = doc.getElementsByTagName("Item");
newElement = doc.createElement("Item");
prevNumber = Integer.parseInt(((Element) itemList.item(itemList.getLength() - 1)).getAttribute("Number"));
newElement.setAttribute("Number", (prevNumber + 1) + "");
doc.getDocumentElement().appendChild(newElement);
transformerFactory = TransformerFactory.newInstance();
transformer = transformerFactory.newTransformer();
source = new DOMSource(doc);
channel.truncate(0);
result = new StreamResult(Channels.newOutputStream(channel));
transformer.setOutputProperty("{http://xml.apache.org/xslt}indent-amount", "4");
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "no");
transformer.setOutputProperty(OutputKeys.METHOD, "xml");
transformer.setOutputProperty(OutputKeys.INDENT, "yes");
transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
transformer.transform(source, result);
channel.close();
} catch (IOException | TransformerException e) {
e.printStackTrace();
} finally {
try {
ncis.reallyClose();
} catch (IOException e) {
e.printStackTrace();
}
}
}
class NonClosingInputStream extends FilterInputStream {
public NonClosingInputStream(InputStream it) {
super(it);
}
#Override
public void close() throws IOException {
// Do nothing.
}
public void reallyClose() throws IOException {
// Actually close.
in.close();
}
}
public static void main(String[] args){
new Test2();
}
}
Try this design instead:
Create a new service (a process) which opens a socket and listens to "update commands".
All other processes don't write to the file directly but instead send "update commands" to the new service
That way, you never need to worry about locking. To make the whole thing more reliable, you may want to add buffers to the sending processes so they can continue to live for a while when the service is down.
With this approach, you never have to deal with file locks (which can be unreliable depending on your OS). The socket will also make sure that you can't start the service twice.
Related
I have a web service using JAX-WS API (a soap web service).
import javax.jws.WebService;
#WebService
public class Accounts {
public String getFullName(String userID){
AccountManager GFN = new AccountManager();
return GFN.getFullName(userID);
}
what can I do to change it's encoding to "UTF-8" (for non-English character)?
I fount something like "#Produces("text/html; charset=UTF-8")" but it is for JAX-RS (restful web service) i need something for JAX-WS.
Thanks.
To start, you need to get a hold of the SOAP message as early as possible. The best place to start is a javax.xml.ws.handler.LogicalHandler (as opposed to the more common type of handler, SOAPHandler).
The LogicalHandler intercepts the SOAP payload before the SOAPHandler, so it's the ideal place to do this
Within this handler, you have the liberty to whatever you want with the message, long before the encoding becomes a problem. Your code should look something like this
public class YourHandler implements LogicalHandler{
public boolean handleMessage(LogicalMessageContext context) {
boolean inboundProperty= (boolean)context.get(MessageContext.MESSAGE_INBOUND_PROPERTY);
if (inboundProperty) {
LogicalMessage lm = context.getMessage();
Source payload = lm.getPayload();
Source recodedPayload = modifyEncoding(payload); //This is where you change the encoding. We'll talk more about this
lm.setPayload(recodedPayload) //remember to stuff the payload back in there, otherwise your change will not be registered
}
return true;
}
}
So now you have the message. How to handle the change of encoding can be tricky, but it's completely up to you.
You have the option of setting the encoding on the entire message, or navigating (with xpath) to the field you're interested in and manipulating just that. Even for those two options, there are several ways of accomplishing both. I'm going to go the lazy route: set the encoding on the entire payload:
private Source modifyEncoding(Source payload){
StringWriter sw = new StringWriter();
StreamSource newSource = null;
try {
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8"); //this determines the outcome of the transformation
StreamResult output = new StreamResult(sw);
transformer.transform(source, output);
StringReader sReader = new StringReader(sw.toString());
newSource = new StreamSource(sReader);//Stuff the re-encoded xml back in a Source
} catch(Exception e){
ex.printStackTrace();
}
return newSource;
}
After the LogicalHandler, you now have the SOAPHandler. Here, setting the encoding is more straightforward, but its behaviour is implementation-dependent. Your SOAPHandler can look like this:
public class YourSOAPHandler implements SOAPHandler{
public boolean handleMessage(SOAPMessageContext msgCtxt){
boolean inbound = (boolean)msgCtxt.get(MessageContext.MESSAGE_INBOUND_PROPERTY);
if (inbound){
SOAPMessage msg = msgCtxt.getMessage();
msg.setProperty(javax.xml.soap.SOAPMessage.CHARACTER_SET_ENCODING,"UTF-8");
msgCtxt.Message(msg); //always put the message back where you found it.
}
}
return true;
}
I'm a total Java virgin, and I've been stumbling slowly but surely in developing an IRC bot for my friends. So far, I've gotten nearly all of the features in working order. But, I'm really wracking my brain over this problem here, my bot so far can reply with a link, but every week, I have to change the link in the java file manually and recompile the whole thing. So, I want it to be able to parse the pertinent values from an XML file in the same directory the bot's java files are in, and be able to update those same values through an IRC client.
import org.jibble.pircbot.*;
import javax.xml.parsers.*;
import javax.xml.transform.*;
import javax.xml.transform.dom.*;
import javax.xml.transform.stream.*;
import org.xml.sax.*;
import org.w3c.dom.*;
public class ModBot extends PircBot {
static String inputFile = "./botdata.xml";
static String outputFile = "./botdata.xml";
public ModBot() {
setLogin("ModBot");
this.setName("ModBot");
setVersion(" ");
}
public void onMessage(String channel, String sender, String login, String hostname, String message) {
if (message.equalsIgnoreCase("!lolcat")) {
sendMessage(channel, sender + "http://i.imgur.com/4IX4cUL.jpg");
}
if (message.startsWith("!updatelolcat ")) {
if(login.equals("Mainmod"));
String changelolcat = message.substring(14);
}
}
}
And the XML
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE botdata [
<!ELEMENT botdata (lolcat,partytime,start,end)>
<!ELEMENT lolcat (#PCDATA)>
<!ELEMENT partytime (start,end)>
<!ELEMENT start (#PCDATA)>
<!ELEMENT end (#PCDATA)>
]>
<botdata>
<lolcat>http://i.imgur.com/4IX4cUL.jpg</lolcat>
<partytime>
<start>8:45:30</start>
<end>11:00:00</end>
</partytime>
</botdata>
What I want to do is take whatever "changelolcat" is, and overwrite the current link in the XML, and then a way to read from the same XML to send what's in "lolcat" to anyone replying "!lolcat". I've been going through xpath and jdom and stuff, and I just can't make sense of it. What I've read with methods using xpath looks promising, and I'd prefer to use it because it's prettier to read.
EDIT:
It worked, I put in
try {DocumentBuilderFactory documentBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder documentBuilder = documentBuilderFactory.newDocumentBuilder();
Document document = documentBuilder.parse("botdata.xml");
Node botdata = document.getElementsByTagName("botdata").item(0);
NodeList nodes = botdata.getChildNodes();
for (int i = 0; i < nodes.getLength(); i++) {
Node element = nodes.item(i);
if ("lolcat".equals(element.getNodeName())) {
element.setTextContent(changelolcat);
}
}
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
DOMSource domSource = new DOMSource(document);
StreamResult streamResult = new StreamResult(new File("botdata.xml"));
transformer.transform(domSource, streamResult);
}catch (ParserConfigurationException pce) {
pce.printStackTrace();
} catch (TransformerException tfe) {
tfe.printStackTrace();
} catch (IOException ioe) {
ioe.printStackTrace();
} catch (SAXException sae) {
sae.printStackTrace();
}
after String changelolcat = message.substring(14);
EDIT: I figured out how to parse from my XML to send what's in a node as a message, does this look right? I feel like I'm not supposed to keep copying the doc builder over and over in different methods
if (message.equalsIgnoreCase("!lolcat")) {
try {
DocumentBuilderFactory documentBuilderFactory =
DocumentBuilderFactory.newInstance();
DocumentBuilder documentBuilder =
documentBuilderFactory.newDocumentBuilder();
Document document = documentBuilder.parse(botxml);
XPathFactory xpathFactory = XPathFactory.newInstance();
XPath xpath = xpathFactory.newXPath();
String lolcat = xpath.evaluate("//lolcat", document);
sendMessage(channel, sender + lolcat);
} catch (Exception e) {
e.printStackTrace();
}
}
One way to do this would be to use the javax.xml.parsers.DocumentBuilder to parse your xml file into javax.swing.text.Document, which provides a great interface for getting and modifying individual elements and their values. You can then write the modified document back to the original file, overwriting it with the new version.
Here are some links to the relevant javadocs:
Document
DocumentBuilder
You might also want to look at the javadocs for the DocumentBuilderFactory object, which is definitely the best way to generate DocumentBuilders.
I have researched on the subject but couldn't find any relevant info regarding that
Do we need to take any security measurements to secure javax.xml.transform.Transformer against XML external entity attacks?
I did the following and it seems to expand the dtd.
String fileData = "<!DOCTYPE acunetix [ <!ENTITY sampleVal SYSTEM \"file:///media/sample\">]><username>&sampleVal;</username>";
TransformerFactory transformerFactory = TransformerFactory.newInstance();
transformerFactory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
Transformer transformer = transformerFactory.newTransformer();
StringWriter buff = new StringWriter();
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
transformer.transform(new StreamSource(new StringReader(fileData)), new StreamResult(buff));
System.out.println(buff.toString());
output contains the value from the file
<username>test</username>
Your code seems correct. When I run this slightly modified JUnit test case:
#Test
public void test() throws TransformerException, URISyntaxException {
File testFile = new File(getClass().getResource("test.txt").toURI());
assertTrue(testFile.exists());
String fileData = "<!DOCTYPE acunetix [ <!ENTITY foo SYSTEM \"file://" +
testFile.toString() +
"\">]><xxe>&foo;</xxe>";
TransformerFactory transformerFactory = TransformerFactory.newInstance();
System.out.println(transformerFactory.getClass().getName());
transformerFactory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
Transformer transformer = transformerFactory.newTransformer();
StringWriter buff = new StringWriter();
transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");
transformer.transform(new StreamSource(new StringReader(fileData)), new StreamResult(buff));
assertEquals("<xxe>&foo;</xxe>", buff.toString());
}
I get the following output:
com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl
[Fatal Error] :1:182: External Entity: Failed to read external document 'test.txt', because 'file' access is not allowed due to restriction set by the accessExternalDTD property.
ERROR: 'External Entity: Failed to read external document 'test.txt', because 'file' access is not allowed due to restriction set by the accessExternalDTD property.'
From the setFeature JavaDocs:
All implementations are required to support the XMLConstants.FEATURE_SECURE_PROCESSING feature. When the feature is:
true: the implementation will limit XML processing to conform to implementation limits and behave in a secure fashion as defined by the implementation. Examples include resolving user defined style sheets and functions. If XML processing is limited for security reasons, it will be reported via a call to the registered ErrorListener.fatalError(TransformerException exception). See setErrorListener(ErrorListener listener).
That error goes away if I comment out transformerFactory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true); and then the test fails because the entity is resolved.
Try adding an ErrorListener to both the TransformerFactory and Transformer:
transformerFactory.setErrorListener(new ErrorListener() {
#Override
public void warning(TransformerException exception) throws TransformerException {
System.out.println("In Warning: " + exception.toString());
}
#Override
public void error(TransformerException exception) throws TransformerException {
System.out.println("In Error: " + exception.toString());
}
#Override
public void fatalError(TransformerException exception) throws TransformerException {
System.out.println("In Fatal: " + exception.toString());
}
});
Transformer transformer = transformerFactory.newTransformer();
transformer.setErrorListener(transformerFactory.getErrorListener());
I see the following new console output now:
In Error: javax.xml.transform.TransformerException: External Entity: Failed to read external document 'test.txt', because 'file' access is not allowed due to restriction set by the accessExternalDTD property.
Maybe your implementation is treating it as a warning? Otherwise, maybe it's the implementation you're using? It looks like the JavaDoc spec isn't precise, so one implementation might do something different than another. I'd be interested to know faulty implementations!
I know that this is an old post but for those who find themselves here, I hope is helps :)
After applying the solution below, SonarQube still complained with 'Disable access to external entities in XML parsing' security issue :(
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_DTD, "");
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "");
Eventually I landed on the solution below which finally fixed the issue for me.
TransformerFactory factory = TransformerFactory.newInstance();
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_DTD, "");
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "");
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_STYLESHEET, "");
factory.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
I'm facing an issue with JDK (both 1.6 and 1.7) XSLT transformations.
The thing is that I want to process simple WSDL that is using xsd:import for its XSD (that lies in same location) with my XSLT transformation.
public static void main(String[] args) throws Exception {
InputStream xmlStream = new FileInputStream("/home/d1x/temp/xslt/test.wsdl");
String xmlSystemId = "file:///home/d1x/temp/xslt/test.wsdl";
InputStream xsltStream = XsltTransformation.class.getResourceAsStream("wsdl-viewer.xsl");
OutputStream outputStream = new FileOutputStream("/home/d1x/temp/xslt/output.html");
new XsltTransformation().transform(xmlStream, xmlSystemId, xsltStream, outputStream);
}
public void transform(InputStream xmlStream, String xmlSystemId, InputStream xsltStream, OutputStream outputStream) {
Source xmlSource = new StreamSource(xmlStream, xmlSystemId);
Source xsltSource = new StreamSource(xsltStream);
TransformerFactory transFact = TransformerFactory.newInstance();
try {
Transformer trans = transFact.newTransformer(xsltSource);
trans.transform(xmlSource, new StreamResult(outputStream));
} catch (TransformerConfigurationException e) {
e.printStackTrace();
} catch (TransformerException e) {
e.printStackTrace();
}
}
When I run my code, I get this exception that is kinda hard to debug. When I remove the import, everything works fine.
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at com.sun.org.apache.xml.internal.utils.SuballocatedIntVector.elementAt(SuballocatedIntVector.java:438)
at com.sun.org.apache.xml.internal.dtm.ref.DTMDefaultBase._firstch(DTMDefaultBase.java:524)
at com.sun.org.apache.xalan.internal.xsltc.dom.SAXImpl.access$200(SAXImpl.java:76)
at com.sun.org.apache.xalan.internal.xsltc.dom.SAXImpl$NamespaceChildrenIterator.next(SAXImpl.java:1433)
at com.sun.org.apache.xalan.internal.xsltc.dom.StepIterator.next(StepIterator.java:111)
at com.sun.org.apache.xalan.internal.xsltc.dom.StepIterator.next(StepIterator.java:111)
at com.sun.org.apache.xalan.internal.xsltc.dom.DupFilterIterator.setStartNode(DupFilterIterator.java:96)
at com.sun.org.apache.xalan.internal.xsltc.dom.UnionIterator$LookAheadIterator.setStartNode(UnionIterator.java:78)
at com.sun.org.apache.xalan.internal.xsltc.dom.MultiValuedNodeHeapIterator.setStartNode(MultiValuedNodeHeapIterator.java:212)
at com.sun.org.apache.xalan.internal.xsltc.dom.CurrentNodeListIterator.setStartNode(CurrentNodeListIterator.java:153)
at com.sun.org.apache.xalan.internal.xsltc.dom.CachedNodeListIterator.setStartNode(CachedNodeListIterator.java:55)
at GregorSamsa.topLevel()
... etc...
WSDL itself is very simple and is using the import:
...<types>
<xsd:schema>
<xsd:import namespace="http://mytest.com" schemaLocation="test.xsd"/>
</xsd:schema>
</types>...
Used XSLT can be found at: http://tomi.vanek.sk/xml/wsdl-viewer.xsl
I managed to solve this issue by switching to Saxon implementation of JAXP instead of built-in Java implementation. The only code change was:
TransformerFactory transFact = net.sf.saxon.TransformerFactoryImpl.newInstance();
I have a Java maven project which includes XSLT transformations. I load the stylesheet as follows:
TransformerFactory tFactory = TransformerFactory.newInstance();
DocumentBuilderFactory dFactory = DocumentBuilderFactory
.newInstance();
dFactory.setNamespaceAware(true);
DocumentBuilder dBuilder = dFactory.newDocumentBuilder();
ClassLoader cl = this.getClass().getClassLoader();
java.io.InputStream in = cl.getResourceAsStream("xsl/stylesheet.xsl");
InputSource xslInputSource = new InputSource(in);
Document xslDoc = dBuilder.parse(xslInputSource);
DOMSource xslDomSource = new DOMSource(xslDoc);
Transformer transformer = tFactory.newTransformer(xslDomSource);
The stylesheet.xsl has a number of statements. These appear to be causing problems, when I try to run my unit tests I get the following errors:
C:\Code\workspace\app\dummy.xsl; Line #0; Column #0; Had IO Exception with stylesheet file: footer.xsl
C:\Code\workspace\app\dummy.xsl; Line #0; Column #0; Had IO Exception with stylesheet file: topbar.xsl
The include statements in the XSLT are relative links
xsl:include href="footer.xsl"
xsl:include href="topbar.xsl"
I have tried experimenting and changing these to the following - but I still get the error.
xsl:include href="xsl/footer.xsl"
xsl:include href="xsl/topbar.xsl"
Any ideas? Any help much appreciated.
Solved my problem using a URIResolver.
class MyURIResolver implements URIResolver {
#Override
public Source resolve(String href, String base) throws TransformerException {
try {
ClassLoader cl = this.getClass().getClassLoader();
java.io.InputStream in = cl.getResourceAsStream("xsl/" + href);
InputSource xslInputSource = new InputSource(in);
Document xslDoc = dBuilder.parse(xslInputSource);
DOMSource xslDomSource = new DOMSource(xslDoc);
xslDomSource.setSystemId("xsl/" + href);
return xslDomSource;
} catch (...
And assigning this with the TransformerFactory
tFactory.setURIResolver(new MyURIResolver());
URIResolver can also be used in a more straightforward way as below:
class XsltURIResolver implements URIResolver {
#Override
public Source resolve(String href, String base) throws TransformerException {
try{
InputStream inputStream = this.getClass().getClassLoader().getResourceAsStream("xslts/" + href);
return new StreamSource(inputStream);
}
catch(Exception ex){
ex.printStackTrace();
return null;
}
}
}
Use the URIResolver with TransformerFactory as shown below:
TransformerFactory transFact = TransformerFactory.newInstance();
transFact.setURIResolver(new XsltURIResolver());
Or with a lambda expression:
transFact.setURIResolver((href, base) -> {
final InputStream s = this.getClass().getClassLoader().getResourceAsStream("xslts/" + href);
return new StreamSource(s);
});
Set your DocumentBuilder object with an EntityResolver.
You'll have to extend EntityResolver class to resolve your external entities (footer.xsl and topbar.xsl).
I had a problem similar to this once with relative paths in the XSLT.
If you can, try to put absolute paths in the XSLT - that should resolve the error.
An absolute path probably isn't preferable for the final version of the XSLT, but it should get you past the maven problem. Perhaps you can have two versions of the XSLT, one with absolute paths for maven and one with relative paths for whatever other tool it's being used with.