We Keep Coding

Is it possible to partition across single file in spring batch?

I have read about partitioning in spring-batch I've found an example which demonstrates partitioning. The example reads persons from CSV files, does some processing and insert data into the database. So at this example 1 partitioning = 1 file and so partitioner implementation looks like this:
public class MultiResourcePartitioner implements Partitioner {
private final Logger logger = LoggerFactory.getLogger(MultiResourcePartitioner.class);
public static final String FILE_PATH = "filePath";
private static final String PARTITION_KEY = "partition";
private final Collection<Resource> resources;
public MultiResourcePartitioner(Collection<Resource> resources) {
this.resources = resources;
}
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> map = new HashMap<>(gridSize);
int i = 0;
for (Resource resource : resources) {
ExecutionContext context = new ExecutionContext();
context.putString(FILE_PATH, getPath(resource)); //Depends on what logic you want to use to split
map.put(PARTITION_KEY + i++, context);
}
return map;
}
private String getPath(Resource resource) {
try {
return resource.getFile().getPath();
} catch (IOException e) {
logger.warn("Can't get file from from resource {}", resource);
throw new RuntimeException(e);
}
}
}
But what if I have single 10TB file? Does spring batch allow to partition it in some way?
update:
I tried following approach to achieve what I want:
make 2 steps - first step to divide file into pieces and second step to process pieces we got after the first step:
#Configuration
public class SingleFilePartitionedJob {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private ToLowerCasePersonProcessor toLowerCasePersonProcessor;
#Autowired
private DbPersonWriter dbPersonWriter;
#Autowired
private ResourcePatternResolver resourcePatternResolver;
#Value("${app.file-to-split}")
private Resource resource;
#Bean
public Job splitFileProcessingJob() throws IOException {
return jobBuilderFactory.get("splitFileProcessingJob")
.incrementer(new RunIdIncrementer())
.flow(splitFileIntoPiecesStep())
.next(csvToDbLowercaseMasterStep())
.end()
.build();
}
private Step splitFileIntoPiecesStep() throws IOException {
return stepBuilderFactory.get("splitFile")
.tasklet(new FileSplitterTasklet(resource.getFile()))
.build();
}
#Bean
public Step csvToDbLowercaseMasterStep() throws IOException {
MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
partitioner.setResources(resourcePatternResolver.getResources("split/*.csv"));
return stepBuilderFactory.get("csvReaderMasterStep")
.partitioner("csvReaderMasterStep", partitioner)
.gridSize(10)
.step(csvToDataBaseSlaveStep())
.taskExecutor(jobTaskExecutorSplitted())
.build();
}
#Bean
public Step csvToDataBaseSlaveStep() throws MalformedURLException {
return stepBuilderFactory.get("csvToDatabaseStep")
.<Person, Person>chunk(50)
.reader(csvPersonReaderSplitted(null))
.processor(toLowerCasePersonProcessor)
.writer(dbPersonWriter)
.build();
}
#Bean
#StepScope
public FlatFileItemReader csvPersonReaderSplitted(#Value("#{stepExecutionContext[fileName]}") String fileName) throws MalformedURLException {
return new FlatFileItemReaderBuilder()
.name("csvPersonReaderSplitted")
.resource(new UrlResource(fileName))
.delimited()
.names(new String[]{"firstName", "lastName"})
.fieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
setTargetType(Person.class);
}})
.build();
}
#Bean
public TaskExecutor jobTaskExecutorSplitted() {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
taskExecutor.setMaxPoolSize(30);
taskExecutor.setCorePoolSize(25);
taskExecutor.setThreadNamePrefix("cust-job-exec2-");
taskExecutor.afterPropertiesSet();
return taskExecutor;
}
}
tasklet:
public class FileSplitterTasklet implements Tasklet {
private final Logger logger = LoggerFactory.getLogger(FileSplitterTasklet.class);
private File file;
public FileSplitterTasklet(File file) {
this.file = file;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
int count = FileSplitter.splitTextFiles(file, 100);
logger.info("File was split on {} files", count);
return RepeatStatus.FINISHED;
}
}
logic for splitting file:
public static int splitTextFiles(File bigFile, int maxRows) throws IOException {
int fileCount = 1;
try (BufferedReader reader = Files.newBufferedReader(Paths.get(bigFile.getPath()))) {
String line = null;
int lineNum = 1;
Path splitFile = Paths.get(bigFile.getParent() + "/" + fileCount + "split.txt");
BufferedWriter writer = Files.newBufferedWriter(splitFile, StandardOpenOption.CREATE);
while ((line = reader.readLine()) != null) {
if (lineNum > maxRows) {
writer.close();
lineNum = 1;
fileCount++;
splitFile = Paths.get("split/" + fileCount + "split.txt");
writer = Files.newBufferedWriter(splitFile, StandardOpenOption.CREATE);
}
writer.append(line);
writer.newLine();
lineNum++;
}
writer.close();
}
return fileCount;
}
So I put all file pieces to the special directory.
But this doesn't work because on the moment of context initialization folder /split does not exist yet.
update
I've generated workaround which works:
public class MultiResourcePartitionerWrapper implements Partitioner {
private final MultiResourcePartitioner multiResourcePartitioner = new MultiResourcePartitioner();
private final ResourcePatternResolver resourcePatternResolver;
private final String pathPattern;
public MultiResourcePartitionerWrapper(ResourcePatternResolver resourcePatternResolver, String pathPattern) {
this.resourcePatternResolver = resourcePatternResolver;
this.pathPattern = pathPattern;
}
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
try {
Resource[] resources = resourcePatternResolver.getResources(pathPattern);
multiResourcePartitioner.setResources(resources);
return multiResourcePartitioner.partition(gridSize);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
But it looks ugly. Is it a correct solution?

Spring batch allow you to partition, but it's up to you how to do it.
You can simply split your 10TB file in the partitioner class (by number or by max rows), and each partion reads one splitted file. You can find a lot of example of how to split large file in java.
split very large text file by max rows

log4j2 -version 2.5 - How to Configure log4j2 using a java.util.properties object

I am trying to configure log4j2 version 2.5 using a properties object. Reason for doing this is migration from version 1.2.17. I cannot directly use the properties file. We do some modifications to it programmaticaly.
Here's what I have tried:
LogTest.java
public class LogTest {
static {
System.setProperty("log4j.configurationFactory", "logsample.common.util.LogsampleConfigurationFactory");
}
private static Logger logger = LogManager.getLogger(LogTest.class.getName());
public static void main(String[] args) throws Exception
{
logger.debug("First log");
logger.info("Infoed");
}
}
trace.properties
name = PropertiesConfig
property.filename = D:/rolling/rollingtest.log
appenders = file
appender.file.type = File
appender.file.name = LOGFile
appender.file.fileName = ${filename}
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d %p %C{1.} [%t] %m%n
loggers = file
logger.file.name = logware.common.util
logger.file.level = debug
logger.file.appenderRefs = file
logger.file.appenderRef.file.ref = LOGFile
LogwareConfigurationFactory.java
public class LogsampleConfigurationFactory extends ConfigurationFactory {
#Override
protected String[] getSupportedTypes() {
return new String[]{".properties", "*"};
}
#Override
public Configuration getConfiguration(ConfigurationSource source) {
return new PropertiesConfiguration(createConfigurationSource(), null);
}
#Override
public Configuration getConfiguration(String name, URI configLocation) {
return new PropertiesConfiguration(createConfigurationSource(), null);
}
private ConfigurationSource createConfigurationSource()
{
Properties p = new Properties();
ByteArrayOutputStream out = new ByteArrayOutputStream();
InputStream in = null;
try {
p.load(new FileInputStream("D:/log4jSample/properties/trace.properties"));
p.store(out, null);
} catch (IOException e) {
e.printStackTrace();
}
in = new ByteArrayInputStream(out.toByteArray());
ConfigurationSource configSrc = null;
try {
configSrc = new ConfigurationSource(in);
}
catch (IOException i)
{
}
return configSrc;
}
}
When I run the LogTest class, it fails to get the LogContext with a null pointer.
Exception Stack
java.lang.ExceptionInInitializerError
Caused by: java.lang.NullPointerException
at org.apache.logging.log4j.core.config.builder.impl.BuiltConfiguration.<init>(BuiltConfiguration.java:58)
at org.apache.logging.log4j.core.config.properties.PropertiesConfiguration.<init>(PropertiesConfiguration.java:36)
at logware.common.util.LogwareConfigurationFactory.getConfiguration(LogwareConfigurationFactory.java:46)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:427)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:256)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:561)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:578)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:235)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:167)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:522)
at logware.common.util.LogTest.<clinit>(LogTest.java:58)
So, its the rootComponent that I have not put in the PropertiesConfiguration constructor. But I have no idea what it should be.
Any guidance or a clue in this regard would be great.

With 2.5 I would recommend you do:
#Override
public Configuration getConfiguration(ConfigurationSource source) {
PropertiesConfigurationFactory factory = new PropertiesConfigurationFactory();
return factory.getConfiguration(source);
}
With the latest release you will have to modify this to do:
#Override
public Configuration getConfiguration(LoggerContext ctx, ConfigurationSource source) {
PropertiesConfigurationFactory factory = new PropertiesConfigurationFactory();
return factory.getConfiguration(ctx, source);
}

Spring configuration : make a string attribute

2 questions :
do you know a "best" way to use JNDI resources on spring
configuration?
how can I transform the static attribut configPath to a "value" I can
use in this 2 methods but also on other bean?
#Configuration
#ComponentScan(basePackages = {"com.fdilogbox.report.serveur"})
#EnableAspectJAutoProxy
public class SpringConfig {
public static final String JNDI_RESOURCES = "java:comp/env/report/config";
public static String configPath;
#PostConstruct
public void initConfig() throws IllegalArgumentException {
try {
final JndiTemplate template = new JndiTemplate();
configPath = (String) template.lookup(JNDI_RESOURCES);
Log4jConfigurer.initLogging(configPath + "/log4j.xml");
} catch (NamingException | FileNotFoundException ex) {
throw new IllegalArgumentException("Unable to start logging, because non-file resource!", ex);
}
}
#Bean
public static PropertySourcesPlaceholderConfigurer properties() {
try {
final JndiTemplate template = new JndiTemplate();
configPath = (String) template.lookup(JNDI_RESOURCES);
} catch (NamingException ex) {
throw new IllegalArgumentException("JNDI resource not found : " + JNDI_RESOURCES, ex);
}
PropertySourcesPlaceholderConfigurer pspc = new PropertySourcesPlaceholderConfigurer ();
Resource[] resources = new FileSystemResource[]{new FileSystemResource(configPath + "/application.properties")};
pspc.setLocations(resources);
pspc.setIgnoreUnresolvablePlaceholders(true);
return pspc;
}
}

RestFul web service which reads properties file

I have got couple of Jersey REST Web services say SendPassword and ResetPassword whose purpose is to send email .
For sending email , i have configured a properties file under tomcat and all this works fine
The code of the SendPassword.java is somewhat this way
#Path("/sendpassword")
public class SendPassword {
#GET
#Produces("application/json")
public String sendPasswordToEmail(#QueryParam("empid") String empid)
throws JSONException
{
try {
SendEmailUtility.sendmail("weqw","2312");
}
catch(Exception e)
{
}
}
SendEmailUtility.java
public class SendEmailUtility
{
public static String sendmail(String sendemalto,String generatedpwd) throws IOException {
String result = "fail";
File configDir = new File(System.getProperty("catalina.base"), "conf");
File configFile = new File(configDir, "email.properties");
InputStream stream = new FileInputStream(configFile);
Properties props_load = new Properties();
props_load.load(stream);
final String username = props_load.getProperty("username");
final String password = props_load.getProperty("password");
Properties props_send = new Properties();
props_send.put("mail.smtp.auth","true");
props_send.put("mail.smtp.starttls.enable","true");
Transport.send(message);
// Some code to send email
result = "success";
} catch (MessagingException e) {
result = "fail";
e.printStackTrace();
}
return result;
}
}
As you can see i am reading the properties file for every call of the websercice
(As reading operation is somewhat costly) , Is there any way to resolve this ??
Could you please let me know whats the best approach to handle this.
Thanks in advance .

There are few ways to do this the one way of doing it is to make the props_load a private static member of the class and call it like this
public class SendEmailUtility
{
private static Properties props;
private static Properties getProperties() {
if (props == null) {
File configDir = new File(System.getProperty("catalina.base"), "conf");
File configFile = new File(configDir, "email.properties");
InputStream stream = new FileInputStream(configFile);
props = new Properties();
props.load(stream);
}
return props;
}
public static String sendmail(String sendemalto,String generatedpwd) throws IOException {
String result = "fail";
Properties props_load = getProperties();
final String username = props_load.getProperty("username");
final String password = props_load.getProperty("password");
Properties props_send = new Properties();
props_send.put("mail.smtp.auth","true");
props_send.put("mail.smtp.starttls.enable","true");
Transport.send(message);
// Some code to send email
result = "success";
} catch (MessagingException e) {
result = "fail";
e.printStackTrace();
}
return result;
}
}
The other design I would suggest is to make an email service class like EmailSender or EmailService and then inject it into SendPassword class
#Path("/sendpassword")
public class SendPassword {
#Autowired
EmailService emailService;
#GET
#Produces("application/json")
public String sendPasswordToEmail(#QueryParam("empid") String empid)
throws JSONException
{

I would recommend using resource bundle, which does not need InputStream
create a properties file and put directly inside your packages along with your java code
example folder structure
com
- preethi
-SendPassword.java
-email.properties
Then you can code like
ResourceBundle props_load = ResourceBundle.getBundle("com.preethi.email");
final String username = props_load.getString("username");
This way you don't have to worry about opening and closing the stream or file path

You could use a lazy-getter to fetch and cache the Properties object.
private static Properties props;
private static Properties getProperties() {
if (props == null) {
File configDir = new File(System.getProperty("catalina.base"), "conf");
File configFile = new File(configDir, "email.properties");
InputStream stream = new FileInputStream(configFile);
props = new Properties();
props.load(stream);
}
return props;
}
Each time you want to use the Properties, call getProperties(). It will cache it the first time it's called. Each subsequent call will just return the cached object.
Note: This example does not catch any exceptions.

loading the properties file

I wanted to read some properties file.
For that I created a small program which reads, writes and also updates this properties file.
Now some people are saying the properties file should be read only once, that means when the class is loaded it should read once, not multiple times for each key.
So I have to read the properties file inside a static block.
Now my doubt if I make any new entry to the properties file, will it be loaded the new entry ?
Please suggest me which is the correct way to design the loading of properties file.
public class Parser {
private String path;
private static Properties prop = new Properties();
public Parser(String path) throws IOException{
this.path = path;
load();
}
public Model readPropertiesFile(){
Model model = new Model();
model.setName(prop.getProperty("name"));
return model ;
}
public void createOrUpdatePropertiesFile(Model model){
prop.setProperty("name", model.getName());
}
public void setPath(String path){
this.path = path;
}
public String getPath(){
return path ;
}
public void load() throws IOException{
File file = new File(path);
if(!file.exists()){
file.createNewFile();
System.out.println("File created..");
}
prop.load(new FileInputStream(file));
}

You can try this ways;
Load properties file from classpath
public static Properties load(String propsName) throws Exception {
Properties props = new Properties();
URL url = ClassLoader.getSystemResource(propsName);
props.load(url.openStream());
return props;
}
Load a properties file
public static Properties load(File propsFile) throws IOException {
Properties props = new Properties();
FileInputStream fis = new FileInputStream(propsFile);
props.load(fis);
fis.close();
return props;
}