We Keep Coding

How to spock integration test with standalone tomcat runner?

Our project is not currently using a spring framework.
Therefore, it is being tested based on the standalone tomcat runner.
However, since integration-enabled tests such as #SpringBootTest are not possible, Tomcat is operated in advance and the HTTP API test is carried out using Spock.
Is there a way to turn this like #SpringBootTest?
TomcatRunner
private Tomcat tomcat = null;
private int port = 8080;
private String contextPath = null;
private String docBase = null;
private Context rootContext = null;
public Tomcat8Launcher(){
init();
}
public Tomcat8Launcher(int port, String contextPath, String docBase){
this.port = port;
this.contextPath = contextPath;
this.docBase = docBase;
init();
}
private void init(){
tomcat = new Tomcat();
tomcat.setPort(port);
tomcat.enableNaming();
if(contextPath == null){
contextPath = "";
}
if(docBase == null){
File base = new File(System.getProperty("java.io.tmpdir"));
docBase = base.getAbsolutePath();
}
rootContext = tomcat.addContext(contextPath, docBase);
}
public void addServlet(String servletName, String uri, HttpServlet servlet){
Tomcat.addServlet(this.rootContext, servletName, servlet);
rootContext.addServletMapping(uri, servletName);
}
public void addListenerServlet(ServletContextListener listener){
rootContext.addApplicationListener(listener.getClass().getName());
}
public void startServer() throws LifecycleException {
tomcat.start();
tomcat.getServer().await();
}
public void stopServer() throws LifecycleException {
tomcat.stop();
}
public static void main(String[] args) throws Exception {
System.setProperty("java.util.logging.manager", "org.apache.logging.log4j.jul.LogManager");
System.setProperty(javax.naming.Context.INITIAL_CONTEXT_FACTORY, "org.apache.naming.java.javaURLContextFactory");
System.setProperty(javax.naming.Context.URL_PKG_PREFIXES, "org.apache.naming");
Tomcat8Launcher tomcatServer = new Tomcat8Launcher();
tomcatServer.addListenerServlet(new ConfigInitBaseServlet());
tomcatServer.addServlet("restServlet", "/rest/*", new RestServlet());
tomcatServer.addServlet("jsonServlet", "/json/*", new JsonServlet());
tomcatServer.startServer();
}
Spock API Test example
class apiTest extends Specification {
//static final Tomcat8Launcher tomcat = new Tomcat8Launcher()
static final String testURL = "http://localhost:8080/api/"
#Shared
def restClient
def setupSpec() {
// tomcat.main()
restClient = new RESTClient(testURL)
}
def 'findAll user'() {
when:
def response = restClient.get([path: 'user/all'])
then:
with(response){
status == 200
contentType == "application/json"
}
}
}
The test will not work if the annotations are removed from the annotations below.
// static final Tomcat8Launcher tomcat = new Tomcat8Launcher()
This line is specified API Test at the top.
// tomcat.main()
This line is specified API Test setupSpec() method
I don't know why, but only logs are recorded after Tomcat has operated and the test method is not executed.
Is there a way to fix this?

I would suggest to create a Spock extension to encapsulate everything you need. See writing custom extensions of the Spock docs as well as the built-in extensions for inspiration.

JUnit4 - How to test for readonly/write-protected directory if run with docker

We have an integration test setup for testing the behavior of missing but required configuration properties. Among one of these properties is a directory where failed uploads should be written to for later retries. The general behavior for this property should be that the application doesn't even start up and fail immediately when certain constraints are violated.
The properties are managed by Spring via certain ConfigurationProperties among these we have a simple S3MessageUploadSettings class
#Getter
#Setter
#ConfigurationProperties(prefix = "s3")
#Validated
public class S3MessageUploadSettings {
#NotNull
private String bucketName;
#NotNull
private String uploadErrorPath;
...
}
In the respective Spring configuration we now perform certain validation checks, like whether the path exists, is writable and a directory, and throw respective RuntimeExceptions when certain assertions aren't met:
#Slf4j
#Import({ S3Config.class })
#Configuration
#EnableConfigurationProperties(S3MessageUploadSettings.class)
public class S3MessageUploadSpringConfig {
#Resource
private S3MessageUploadSettings settings;
...
#PostConstruct
public void checkConstraints() {
String sPath = settings.getUploadErrorPath();
Path path = Paths.get(sPath);
...
log.debug("Probing path '{}' for existence', path);
if (!Files.exists(path)) {
throw new RuntimeException("Required error upload directory '" + path + "' does not exist");
}
log.debug("Probig path '{}' for being a directory", path);
if (!Files.isDirectory(path)) {
throw new RuntimeException("Upload directory '" + path + "' is not a directoy");
}
log.debug("Probing path '{}' for write permissions", path);
if (!Files.isWritable(path)) {
throw new RuntimeException("Error upload path '" + path +"' is not writable);
}
}
}
Our test setup now looks like this:
public class StartupTest {
#ClassRule
public static TemporaryFolder testFolder = new TemporaryFolder();
private static File BASE_FOLDER;
private static File ACCESSIBLE;
private static File WRITE_PROTECTED;
private static File NON_DIRECTORY;
#BeforeClass
public static void initFolderSetup() throws IOException {
BASE_FOLDER = testFolder.getRoot();
ACCESSIBLE = testFolder.newFolder("accessible");
WRITE_PROTECTED = testFolder.newFolder("writeProtected");
if (!WRITE_PROTECTED.setReadOnly()) {
fail("Could not change directory permissions to readonly")
}
if (!WRITE_PROTECTED.setWritable(false)) {
fail("Could not change directory permissions to writable(false)");
}
NON_DIRECTORY = testFolder.newFile("nonDirectory");
}
#Configuration
#Import({
S3MessageUploadSpringConfig.class,
S3MockConfig.class,
...
})
static class BaseContextConfig {
// common bean definitions
...
}
#Configuration
#Import(BaseContextConfig.class)
#PropertySource("classpath:ci.properties")
static class NotExistingPathContextConfig {
#Resource
private S3MessageUploadSettings settings;
#PostConstruct
public void updateSettings() {
settings.setUploadErrorPath(BASE_FOLDER.getPath() + "/foo/bar");
}
}
#Configuration
#Import(BaseContextConfig.class)
#PropertySource("classpath:ci.properties")
static class NotWritablePathContextConfig {
#Resource
private S3MessageUploadSettings settings;
#PostConstruct
public void updateSettings() {
settings.setUploadErrorPath(WRITE_PROTECTED.getPath());
}
}
...
#Configuration
#Import(BaseContextConfig.class)
#PropertySource("classpath:ci.properties")
static class StartableContextConfig {
#Resource
private S3MessageUploadSettings settings;
#PostConstruct
public void updateSettings() {
settings.setUploadErrorPath(ACCESSIBLE.getPath());
}
}
#Test
public void shouldFailStartupDueToNonExistingErrorPathDirectory() {
ApplicationContext context = null;
try {
context = new AnnotationConfigApplicationContext(StartupTest.NotExistingPathContextConfig.class);
fail("Should not have started the context");
} catch (Exception e) {
e.printStackTrace();
assertThat(e, instanceOf(BeanCreationException.class));
assertThat(e.getMessage(), containsString("Required error upload directory '" + BASE_FOLDER + "/foo/bar' does not exist"));
} finally {
closeContext(context);
}
}
#Test
public void shouldFailStartupDueToNonWritablePathDirectory() {
ApplicationContext context = null;
try {
context = new AnnotationConfigApplicationContext(StartupTest.NotWritablePathContextConfig.class);
fail("Should not have started the context");
} catch (Exception e) {
assertThat(e, instanceOf(BeanCreationException.class));
assertThat(e.getMessage(), containsString("Error upload path '" + WRITE_PROTECTED + "' is not writable"));
} finally {
closeContext(context);
}
}
...
#Test
public void shouldStartUpSuccessfully() {
ApplicationContext context = null;
try {
context = new AnnotationConfigApplicationContext(StartableContextConfig.class);
} catch (Exception e) {
e.printStackTrace();
fail("Should not have thrown an exception of type " + e.getClass().getSimpleName() + " with message " + e.getMessage());
} finally {
closeContext(context);
}
}
private void closeContext(ApplicationContext context) {
if (context != null) {
// check and close any running S3 mock as this may have negative impact on the startup of a further context
closeS3Mock(context);
// stop a running Spring context manually as this might interfere with a starting context of an other test
((ConfigurableApplicationContext) context).stop();
}
}
private void closeS3Mock(ApplicationContext context) {
S3Mock s3Mock = null;
try {
if (context != null) {
s3Mock = context.getBean("s3Mock", S3Mock.class);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (null != s3Mock) {
s3Mock.stop();
}
}
}
}
When run locally, everything looks fine and all tests pass. Though our CI runs these tests inside a docker container and for some reason changing file permissions seem to end up in a NOOP returning true on the method invocation though not changing anything in regards of the file permission itself.
Neiter File.setReadOnly(), File.setWritable(false) nor Files.setPosixFilePermissions(Path, Set<PosixFilePermission>) seem to have an effect on the actual file permissions in the docker container.
I've also tried to change the directories to real directories, i.e. /root or /dev/pts that are write protected, though as the CI runs the tests as root these directories are writable by the application and the test fails again.
I also considered using an in-memory file system (such as JimFS) though here I'm not sure how to convince the test to make use of the custom filesystem. AFAIK JimFS does not support the constructor needed for declaring it as default filesystem.
Which other possibilities exist from within Java to change a directories permission to readonly/write-protected when run inside a docker container or test successfully for such a directory?

I assume this is due to the permissions and policies of the JVM, and you cannot do anything from your code if the OS has blocked some permissions for your JVM.
You can try to edit java.policy file and set appropriate file permissions.
Perhaps these will be some given files to which write privileges will be set, for example:
grant {
permission java.io.FilePermission "/dev/pts/*", "read,write,delete";
};
More examples in docs: https://docs.oracle.com/javase/8/docs/technotes/guides/security/spec/security-spec.doc3.html.

Embedded Tomcat with Servlet 3.0 - How to skip certain jars when scanning?

Here is a simple method for an embedded Tomcat server that scans for Servlet 3.0 initializers:
public static void main(String[] args) throws ServletException, LifecycleException {
Tomcat tomcat = new Tomcat();
tomcat.setPort(8080);
StandardContext ctx = (StandardContext) tomcat.addWebapp("/", new File("web").getAbsolutePath());
//declare an alternate location for your "WEB-INF/classes" dir:
File additionWebInfClasses = new File("target/classes");
VirtualDirContext resources = new VirtualDirContext();
resources.setExtraResourcePaths("/WEB-INF/classes=" + additionWebInfClasses);
ctx.setResources(resources);
tomcat.start();
System.out.println("Started");
tomcat.getServer().await();
}
I know that the property tomcat.util.scan.DefaultJarScanner.jarsToSkip in catalina.properties allows you to limit which jars are scanned to speed deployment time. How would I incorporate this same idea into the embedded Tomcat code here?
BONUS: Is there a way to specify which jars to include instead of which jars to skip?

While I don't have a specific way of loading individual catalina.properties properties, I found it convenient enough to just overload the JarScanner itself on the Tomcat container contexts like so:
Answer is in Groovy (sorry, I get lazy)
public static void main(String[] args) throws ServletException, LifecycleException {
Tomcat tomcat = new TomcatWithFastJarScanner()
tomcat.setPort(8080)
StandardContext ctx = (StandardContext) tomcat.addWebapp("/", new File("web").getAbsolutePath())
//declare an alternate location for your "WEB-INF/classes" dir:
VirtualDirContext resources = new VirtualDirContext()
resources.setExtraResourcePaths("/WEB-INF/classes=" + new File("target/classes"))
ctx.setResources(resources)
tomcat.start()
println "Started server on port 8080"
tomcat.getServer().await()
}
private static class TomcatWithFastJarScanner extends Tomcat {
#Override
public void start() throws LifecycleException {
getServer().findServices().each { service ->
service.getContainer().findChildren().each { container ->
container.findChildren().each { c ->
((Context) c).setJarScanner(new FastJarScanner())
}
}
}
super.start()
}
}
private static class FastJarScanner extends StandardJarScanner {
def jarsToInclude = [ 'spring-web.*' ]
#Override
public void scan(ServletContext context, ClassLoader classloader,
JarScannerCallback callback, Set<String> jarsToSkip) {
jarsToSkip = new HashSet<String>();
((URLClassLoader) classloader.getParent()).getURLs().each {
def jar = it.path.find(/[^\/]+\.jar$/)
if(!jar) return
for(String inclusionPattern : jarsToInclude) {
if(jar.find(inclusionPattern))
println "including jar: " + jar
else jarsToSkip.add(jar)
}
}
super.scan(context, classloader, callback, jarsToSkip);
}
}
The basic idea is that we are looking at all the jars which the classloader can see, and excluding all of the jars that we don't want to include.
Tomcat embedded starts up real fast like this!

The following is speculative, only reading the specs. One could use a web-fragment, with its own META-INF/web.xml with metadata-complete=true signaling no annotation scanning.

You can extend Tomcat setting scanClassPath to false, something like - this is on top of what user jkschneider here has pointed out):
new Tomcat() {
#Override
public void start() throws LifecycleException {
for (final org.apache.catalina.Service service : getServer().findServices()) {
for (final Container container : service.getContainer().findChildren()) {
for (final Container subContainer : container.findChildren()) {
final StandardJarScanner jarScanner = (StandardJarScanner) ((Context) subContainer).getJarScanner();
jarScanner.setScanClassPath(false);
}
}
}
super.start();
}
};
btw better approach may be as follows:
final Tomcat tomcat = new Tomcat();
final StandardHost tomcatHost = (StandardHost) tomcat.getHost();
tomcatHost.setContextClass("com.fullclassname....FastStandardContext");
where FastStandardContext is
public class FastStandardContext extends StandardContext {
public FastStandardContext() {
((StandardJarScanner) getJarScanner()).setScanClassPath(false);
}
}
This will cover cases where new webapps are added after tomcat is running.

How to change .properties file using apache.commons.configuration

I have an app where I filter messages according to some rules(existing some keywords or regexps). These rules are to be stored in .properties file(as they must be persistent). I've figured out how to read data from this file. here is the part of the code:
public class Config {
private static final Config ourInstance = new Config();
private static final CompositeConfiguration prop = new CompositeConfiguration();
public static Config getInstance() {
return ourInstance;
}
public Config(){
}
public synchronized void load() {
try {
prop.addConfiguration(new SystemConfiguration());
System.out.println("Loading /rules.properties");
final PropertiesConfiguration p = new PropertiesConfiguration();
p.setPath("/home/mikhail/bzrrep/DLP/DLPServer/src/main/resources/rules.properties");
p.load();
prop.addConfiguration(p);
} catch (ConfigurationException e) {
e.printStackTrace();
}
final int processors = prop.getInt("server.processors", 1);
// If you don't see this line - likely config name is wrong
System.out.println("Using processors:" + processors);
}
public void setKeyword(String customerId, String keyword){
}
public void setRegexp(String customerId, String regexp)
{}
}
as you see I'm going to add values to some properties. Here is the .properties file itself:
users = admin, root, guest
users.admin.keywords = admin
users.admin.regexps = test-5, test-7
users.root.keywords = root
users.root.regexps = *
users.guest.keywords = guest
users.guest.regexps =
I have a GUI for user to add keywords and regexps to this config. so, how to implement methods setKeyword and setRegexp?

The easyest way I found is to read the current values of the property to the String[], add there a new value and set property.
props.setProperty(fieldName, values);

How to load XMLCatalog from classpath resources (inside a jar), reliably?

Below are some code fragments that indicate what I am trying at the moment, but its unreliable. Princiaply I think
because you can only register a protocol handler once, and occasionally other libraries may be doing this first.
import org.apache.xerces.util.XMLCatalogResolver;
public static synchronized XMLCatalogResolver getResolver() {
String c[] = {"classpath:xml-catalog.xml"};
if (cr==null) {
log.debug("Registering new protcol handler for classpath");
ConfigurableStreamHandlerFactory configurableStreamHandlerFactory = new ConfigurableStreamHandlerFactory("classpath", new org.fao.oek.protocols.classpath.Handler(XsdUtils.class.getClassLoader()));
configurableStreamHandlerFactory.addHandler("http", new sun.net.www.protocol.http.Handler());
URL.setURLStreamHandlerFactory(configurableStreamHandlerFactory);
log.debug("Creating new catalog resolver");
cr = new XMLCatalogResolver(c);
}
return cr;
}
xml-catalog.xml contains:
<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
<group prefer="public" xml:base="classpath:org/me/myapp/xsd/" >
<uri name="http://www.w3.org/XML/1998/namespace" uri="xml.xsd"/>
<uri name="http://www.w3.org/1999/xlink" uri="xlink.xsd" />
<uri name="http://www.w3.org/2001/XMLSchema" uri="XMLSchema.xsd" />
<uri name="http://purl.org/dc/elements/1.1/" uri="dc.xsd" />
<uri name="http://www.loc.gov/mods/v3" uri="mods-3.3.xsd" />
</group>
</catalog>
Obviously - the xsd files exist at the right place in the classpath.

The resolver acted properly with the following minimum set of code:
public class XsdUtils {
static {
System.setProperty("java.protocol.handler.pkgs", "org.fao.oek.protocols");
}
private static XMLCatalogResolver cr;
public static synchronized XMLCatalogResolver getResolver() {
if (cr == null) {
cr = new XMLCatalogResolver(new String[] { "classpath:xml-catalog.xml" });
}
return cr;
}
public static void main(String[] args) throws MalformedURLException, IOException {
XMLCatalogResolver resolver = getResolver();
URL url0 = new URL("classpath:xml-catalog.xml");
URL url1 = new URL(resolver.resolveURI("http://www.loc.gov/mods/v3"));
url0.openConnection();
url1.openConnection();
}
}
You can alternatively specify java.protocol.handler.pkgs as a JVM argument:
java -Djava.protocol.handler.pkgs=org.fao.oek.protocols ...
The Handler class was implemented as follows:
package org.fao.oek.protocols.classpath;
import java.io.IOException;
import java.net.URL;
import java.net.URLConnection;
public class Handler extends java.net.URLStreamHandler {
#Override
protected URLConnection openConnection(URL u) throws IOException {
String resource = u.getPath();
if (!resource.startsWith("/")) resource = "/" + resource;
System.out.println(getClass().getResource(resource));
return getClass().getResource(resource).openConnection();
}
}
It is important to have the forward slash ("/") when requesting the resource, as answered by this Stack Overflow question: "open resource with relative path in java."
Note the main method in XsdUtils. The output to the program when xml-catalog.xml and mods-3.3.xsd are on the classpath but not in a JAR is:
file:/workspace/8412798/target/classes/xml-catalog.xml
file:/workspace/8412798/target/classes/org/me/myapp/xsd/mods-3.3.xsd
The output to the program when the files are in a JAR is:
jar:file:/workspace/8412798/target/stackoverflow.jar!/xml-catalog.xml
jar:file:/workspace/8412798/target/stackoverflow.jar!/org/me/myapp/xsd/mods-3.3.xsd
With respect to this code in the original question:
new org.fao.oek.protocols.classpath.Handler(XsdUtils.class.getClassLoader())
your Handler does not need a specific class loader unless you have configured your application to use a special class loader, like one extended from URLClassLoader.
"A New Era for Java Protocol Handlers" is a good resource about protocol handlers.
Just to bring everything full circle, the following class uses XsdUtils.getResolver() to parse XML. It validates against the schemas specified in the XMLCatalogResolver:
public class SampleParser {
public static void main(String[] args) throws Exception {
String xml = "<?xml version=\"1.0\"?>" + //
"<mods ID=\"id\" version=\"3.3\" xmlns=\"http://www.loc.gov/mods/v3\">" + //
"<titleInfo></titleInfo>" + //
"</mods>";
ByteArrayInputStream is = new ByteArrayInputStream(xml.getBytes());
XMLReader parser = XMLReaderFactory.createXMLReader(org.apache.xerces.parsers.SAXParser.class.getName());
parser.setFeature("http://xml.org/sax/features/validation", true);
parser.setFeature("http://apache.org/xml/features/validation/schema", true);
parser.setFeature("http://apache.org/xml/features/validation/schema-full-checking", true);
parser.setProperty("http://apache.org/xml/properties/internal/entity-resolver", XsdUtils.getResolver());
parser.setErrorHandler(new ErrorHandler() {
#Override
public void error(SAXParseException exception) throws SAXException {
System.out.println("error: " + exception);
}
#Override
public void fatalError(SAXParseException exception) throws SAXException {
System.out.println("fatalError: " + exception);
}
#Override
public void warning(SAXParseException exception) throws SAXException {
System.out.println("warning: " + exception);
}
});
parser.parse(new InputSource(is));
}
}