We Keep Coding

Unable to load application.properties on Hadoop jar (NullPointerException)

I've looked at all kinds of answers for this problem. None of them work.
I have the following code:
import java.util.Properties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.FileNotFoundException;
import java.io.IOException;
public class ApplicationConfig {
private static Logger LOG;
private String appConfigFileLocation = "application.properties";
private Properties appConfig;
private static ApplicationConfig instance;
public static ApplicationConfig getInstance() {
if(instance == null) {
instance = new ApplicationConfig();
}
return instance;
}
private ApplicationConfig() {
LOG = LoggerFactory.getLogger(this.getClass().getSimpleName());
appConfig = new Properties();
try {
LOG.info("Reading config from " + appConfigFileLocation);
appConfig.load(ClassLoader.getSystemResourceAsStream(appConfigFileLocation));
LOG.info("Done reading config from " + appConfigFileLocation);
} catch (FileNotFoundException e) {
LOG.error("Encountered FileNotFoundException while reading configuration: " + e.getMessage());
throw new RuntimeException(e);
} catch (IOException e) {
LOG.error("Encountered IOException while reading configuration: " + e.getMessage());
throw new RuntimeException(e);
}
}
}
I created a JAR file. The JAR file has application.properties at the root. I also copied the application.properties file in /etc/hadoop/conf and in the target/classes/ directory.
I use the hadoop jar command to execute the code.
But I keep getting the error: java.lang.NullPointerException at java.util.Properties$LineReader.readLine(Properties.java:434)
Please help me at resolving this frustrating error!

Found the error.
hadoop jar checks in the Hadoop classpath. Even though the file was there in the Hadoop classpath, it didn't have read permissions from the Hadoop user.
A simple sudo chmod a+r /etc/hadoop/conf/application.properties did the trick!

Spring Boot Application Could not initialize class org.apache.logging.log4j.util.PropertiesUtil

I am trying to implement the hello world web application using spring boot, gradle and tomcat by following "Building a RESTful Web Service" but have been unable to run make it run so far.
The code is pretty much the same as the one provided on the website, I have wasted hours debugging it thinking there was a bug in the provided code but I still can't figure out what's wrong.
I am using Eclipse Java EE IDE for Web Developers, Version: Neon.3 Release (4.6.3), Build id: 20170314-1500
Any idea what could be the issue?
build.gradle
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:2.0.2.RELEASE")
}
}
apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'
bootJar {
baseName = 'gs-rest-service'
version = '0.1.0'
}
repositories {
mavenCentral()
}
sourceCompatibility = 1.8
targetCompatibility = 1.8
dependencies {
compile("org.springframework.boot:spring-boot-starter-web")
testCompile('org.springframework.boot:spring-boot-starter-test')
}
Greeting.java
package App;
public class Greeting {
private final long id;
private final String content;
public Greeting(long id, String content) {
this.id = id;
this.content = content;
}
public long getId() {
return id;
}
public String getContent() {
return content;
}
}
GreetingController.java
package App;
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
#RestController
public class GreetingController {
private static final String template = "Hello, %s!";
private final AtomicLong counter = new AtomicLong();
#RequestMapping("/greeting")
public Greeting greeting(#RequestParam(value="name", defaultValue="World") String name) {
return new Greeting(counter.incrementAndGet(),
String.format(template, name));
}
}
Application.java
package App;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class Application {
public static void main(String[] args) {
System.getProperties().put("server.port", 8486);
SpringApplication.run(Application.class, args);
}
}
Stacktrace
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.logging.log4j.util.PropertiesUtil
at org.apache.logging.log4j.status.StatusLogger.<clinit>(StatusLogger.java:71)
at org.apache.logging.log4j.LogManager.<clinit>(LogManager.java:60)
at org.apache.commons.logging.LogFactory$Log4jLog.<clinit>(LogFactory.java:199)
at org.apache.commons.logging.LogFactory$Log4jDelegate.createLog(LogFactory.java:166)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:109)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:99)
at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:198)
at App.Application.main(Application.java:9)

When using the authorisation manager, this message can also be the result of a missing grant (or file) in the catalina.policy or security.policy (-Djava.security.policy).
For example add:
grant codeBase "file:${catalina.base}/webapps/${APPLICATION_NAME}/-" {
permission java.security.AllPermission;
};

Apparently setting the port number using System.getProperties().put("server.port", 8486); the NoClassDefFoundError exception.
However creating a application.properties file mentioned by #Nitishkumar Singh in the resources folder to specify the port number to use solved the issue.

Add this dependency:
compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.11.0'
And try again.

Could not initialize class org.apache.logging.log4j.util.PropertiesUtil.
I met the same problem as you. Automated deployment to Tomcat by using Jenkins always occurs. Probably some jar don't exclusion dependecy 'org.apache.logging.log4j'. But springbooot 2.0.1.RELEASE uses logback by default. On org.springframework.boot.web.servlet.support.SpringBootServletInitializer:
public void onStartup(ServletContext servletContext) throws ServletException {
this.logger = LogFactory.getLog(this.getClass());
WebApplicationContext rootAppContext = this.createRootApplicationContext(servletContext);
if (rootAppContext != null) {
servletContext.addListener(new ContextLoaderListener(rootAppContext) {
public void contextInitialized(ServletContextEvent event) {
}
});
} else {
this.logger.debug("No ContextLoaderListener registered, as createRootApplicationContext() did not return an application context");
}
}
public static Log getLog(String name) {
switch(logApi) {
case LOG4J:
return LogFactory.Log4jDelegate.createLog(name);
case SLF4J_LAL:
return LogFactory.Slf4jDelegate.createLocationAwareLog(name);
case SLF4J:
return LogFactory.Slf4jDelegate.createLog(name);
default:
return LogFactory.JavaUtilDelegate.createLog(name);
}
}

Apache beam: No Runner was specified and the DirectRunner was not found on the classpath

I am building a gradle java project (please refer below) using Apache Beam code and executing on Eclipse Oxygen.
package com.xxxx.beam;
import java.io.IOException;
import org.apache.beam.runners.spark.SparkContextOptions;
import org.apache.beam.runners.spark.SparkPipelineResult;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineRunner;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.transforms.MapElements;
import org.apache.beam.sdk.transforms.SimpleFunction;
import org.apache.beam.sdk.values.KV;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.beam.sdk.io.FileIO;
import org.apache.beam.sdk.io.FileIO.ReadableFile;
public class ApacheBeamTestProject {
public void modelExecution(){
SparkContextOptions options = (SparkContextOptions) PipelineOptionsFactory.create();
options.setSparkMaster("xxxxxxxxx");
JavaSparkContext sc = options.getProvidedSparkContext();
JavaLinearRegressionWithSGDExample.runJavaLinearRegressionWithSGDExample(sc);
Pipeline p = Pipeline.create(options);
p.apply(FileIO.match().filepattern("hdfs://path/to/*.gz"))
// withCompression can be omitted - by default compression is detected from the filename.
.apply(FileIO.readMatches())
.apply(MapElements
// uses imports from TypeDescriptors
.via(
new SimpleFunction <ReadableFile, KV<String,String>>() {
private static final long serialVersionUID = -5715607038612883677L;
#SuppressWarnings("unused")
public KV<String,String> createKV(ReadableFile f) {
String temp = null;
try{
temp = f.readFullyAsUTF8String();
}catch(IOException e){
}
return KV.of(f.getMetadata().resourceId().toString(), temp);
}
}
))
.apply(FileIO.write())
;
SparkPipelineResult result = (SparkPipelineResult) p.run();
result.getState();
}
public static void main(String[] args) throws IOException {
System.out.println("Test log");
PipelineOptions options = PipelineOptionsFactory.create();
Pipeline p = Pipeline.create(options);
p.apply(FileIO.match().filepattern("hdfs://path/to/*.gz"))
// withCompression can be omitted - by default compression is detected from the filename.
.apply(FileIO.readMatches())
.apply(MapElements
// uses imports from TypeDescriptors
.via(
new SimpleFunction <ReadableFile, KV<String,String>>() {
private static final long serialVersionUID = -5715607038612883677L;
#SuppressWarnings("unused")
public KV<String,String> createKV(ReadableFile f) {
String temp = null;
try{
temp = f.readFullyAsUTF8String();
}catch(IOException e){
}
return KV.of(f.getMetadata().resourceId().toString(), temp);
}
}
))
.apply(FileIO.write());
p.run();
}
}
I am observing the following error when executing this project in Eclipse.
Test log
Exception in thread "main" java.lang.IllegalArgumentException: No Runner was specified and the DirectRunner was not found on the classpath.
Specify a runner by either:
Explicitly specifying a runner by providing the 'runner' property
Adding the DirectRunner to the classpath
Calling 'PipelineOptions.setRunner(PipelineRunner)' directly
at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:291)
at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:281)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:591)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:532)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155)
at org.apache.beam.sdk.options.PipelineOptionsValidator.validate(PipelineOptionsValidator.java:95)
at org.apache.beam.sdk.options.PipelineOptionsValidator.validate(PipelineOptionsValidator.java:49)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:44)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
This project doesn't contain pom.xml file. Gradle has setup for all the links.
I am not sure how to fix this error? Could someone advise?

It seems that you are trying to use the DirectRunner and it is not on the classpath of your application. You can supply it by adding beam-runners-direct-java dependency to your application:
https://mvnrepository.com/artifact/org.apache.beam/beam-runners-direct-java
EDIT (answered in comment): you are trying to run this code on spark, but didn't specify it in PipelineOptions. Beam by default tries to run the code on DirectRunner, so I think this is why you get this error. Specifying:
options.setRunner(SparkRunner.class); before creating the pipeline sets the correct runner and fixes the issue.

Downloading the beam-runners-direct-java-x.x.x.jar and adding it to the project classpath worked for me. Please refer to this maven repository to download the DirectRunner jar file.
Furthermore, if you need a specific beam runner for your project, you can pass the runner name as a program argument (eg: --runner=DataflowRunner) and add the corresponding jar to the project classpath.

Import "someObj.proto" was not found or had errors

I need to import a .proto file defined in a jar in our central maven.
Here's my build.gralde:
apply plugin: 'com.google.protobuf'
buildscript {
repositories {
maven {
url 'http://abc.def.com/content/groups/public/'
}
maven {
url 'http://abc.def.com/content/repositories/snapshot/'
}
}
dependencies {
classpath 'def.abc.someObj:someObj-proto:1.2'
}
}
protobuf {
generatedFilesBaseDir="$projectDir/src/"
}
sourceSets {
main {
// this tells the plugin where your project specific
// protofiles are located
proto {
srcDir 'src/main/resources/proto/'
}
java {
srcDir 'src/main/java'
}
}
}
Here's my other.proto file:
syntax = "proto3";
option java_multiple_files = true;
option java_package = "com.def.abc";
import "google/protobuf/any.proto";
import "someObj.proto";
whenever I try to compile, it's always complaining at
Import "someObj.proto" was not found or had errors.
I've downloaded the jar file from this maven, and clearly saw that this someObj.proto is in there.
Any ideas please?
Thanks!

How to run Junit TestSuites from gradle?

I am trying to migrate from Ant build to Gradle in my project. There are a bunch of test cases (subclasses of junit.framework.TestCase) and few test suites (subclasses of junit.framework.TestSuite). Gradle automatically picked up all test cases(subclasses of junit.framework.TestCase) to be run, but not the suites (subclasses of junit.framework.TestSuite).
I probably could work around by calling ant.junit to run it. But, I feel there should be a native easy way to force gradle to pick them and run. I couldn't find anything in the document . Am I missing something?

This was hard for me to figure out, but here is an example:
// excerpt from https://github.com/djangofan/WebDriverHandlingMultipleWindows
package webdriver.test;
import http.server.SiteServer;
import java.io.File;
import java.io.IOException;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
#RunWith(Suite.class)
#Suite.SuiteClasses({ TestHandleCacheOne.class, TestHandleCacheThree.class, TestHandleCacheThree.class })
public class SuiteOne extends MultiWindowUtils {
public static SiteServer fs;
#BeforeClass
public static void setUpSuiteOne() {
File httpRoot = new File("build/resources/test");
System.out.println("Server root directory is: " + httpRoot.getAbsolutePath() );
int httpPort = Integer.parseInt("8080");
try {
fs = new SiteServer( httpPort , httpRoot );
} catch (IOException e) {
e.printStackTrace();
}
initializeBrowser( "firefox" );
System.out.println("Finished setUpSuiteOne");
}
#AfterClass
public static void tearDownSuiteOne() {
closeAllBrowserWindows();
System.out.println("Finished tearDownSuiteOne");
}
}
And a build.gradle similar to this:
apply plugin: 'java'
apply plugin: 'eclipse'
group = 'test.multiwindow'
ext {
projTitle = 'Test MultiWindow'
projVersion = '1.0'
}
repositories {
mavenCentral()
}
dependencies {
compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '2.+'
compile group: 'junit', name: 'junit', version: '4.+'
compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.+'
}
task testGroupOne(type: Test) {
//include '**/*SuiteOne.*'
include '**/SuiteOne.class'
reports.junitXml.destination = "$buildDir/test-results/SuiteOne")
reports.html.destination = "$buildDir/test-results/SuiteOne")
}
task testGroupTwo(type: Test) {
//include '**/*SuiteTwo.*'
include '**/SuiteTwo.class'
reports.junitXml.destination = "$buildDir/test-results/SuiteTwo")
reports.html.destination = "$buildDir/test-results/SuiteTwo")
}