I've written a springboot application to perform etl from data source to another data lake every 15 mins. I've scheduled the execution using #Scheduled annotation to a function.
I had created jar and was executing directly through java -jar ingest.jar. It works fine for some days (3-4 days). And just pauses without any exception. To resume, I have to go and press any key to make it active again.
#Scheduled(initialDelayString = "${ingest.initialdelay.in.seconds}000", fixedRateString = "${ingest.rate.in.seconds}000")
public void ingestData(){
// Ingestion Logic
}
Because the problem persisted, I created war and deployed to the tomcat server. But the problem still remains.
Can somebody point me what am I missing here? The same application works fine if I deploy to cloudfoundry.
IO Streams - FileInputStream and FileOutputStream
Helper Functions for IO
public static void saveLastSuccessfulDate(String filepath, String propertyName, Date dateTime) {
Properties prop = new Properties();
OutputStream output = null;
try {
String lastDate = getDateInFormat(dateTime);
log.info("Saving: " + lastDate);
output = new FileOutputStream(filepath);
prop.setProperty(propertyName, lastDate);
prop.store(output, null);
} catch (IOException io) {
io.printStackTrace();
} finally {
if (output != null) {
try {
output.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
//Helper to Write to properties file
public static String checkLastSuccessfulDateAsString(String filepath, String propName) {
Properties prop = new Properties();
InputStream input = null;
try {
input = new FileInputStream(filepath);
// load a properties file
prop.load(input);
String lastSuccesfulDate = prop.getProperty(propName);
log.info("Last Successful Date: "+lastSuccesfulDate);
return lastSuccesfulDate;
} catch (FileNotFoundException f) {
log.error("checkLastSuccessfulDateAsString: File Not Found: " + f.getMessage());
} catch (IOException ex) {
log.error(ex.getMessage());
ex.printStackTrace();
} finally {
if (input != null) {
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return null;
}
Regards
In default, spring #Scheduled use single thread. So, if one task was blocking, the next task won't run.
You can make your task class implements SchedulingConfigurer.It will use multithread to run task and avoid blocking. Code like it:
#Component
public class TaskService implements SchedulingConfigurer {
#Override
public void configureTasks(ScheduledTaskRegistrar scheduledTaskRegistrar) {
scheduledTaskRegistrar.setScheduler(taskExecutor());
}
#Bean(destroyMethod = "shutdown")
public ScheduledExecutorService taskExecutor() {
return Executors.newScheduledThreadPool(100);
}
// your code
#Scheduled(initialDelayString = "${ingest.initialdelay.in.seconds}000", fixedRateString = "${ingest.rate.in.seconds}000")
public void ingestData(){
// Ingestion Logic
}
}
May help you.
Related
I am making a program that communicates data through a resource folder within a network. There are two "types" of this same program, a runner and a viewer. The runner saves data and the viewer loads data. I am using OOS and OIS to do this. The runner saves the data to the resource every few seconds, and the viewer loads the data from the folder every few seconds as well. After some time between 10-80 minutes (as I have found), I get an EOFException when using a shared network, but in the Eclipse IDE, it runs smoothly for an indefinite time.
I don't know why this is. The first thing I did was move the load function to a separate thread, but that didn't help. Then I tried resetting the Thread every 10 minutes, that didn't work either. Would this solution even theoretically work or is there another glaring error in my logic.
Please ask if you need any more information.
Code for loading data through resetting Thread:
LinkedList<Thread> threads = new LinkedList<>();
Timer resetTimer = new Timer();
resetTimer.schedule(new TimerTask() {
#Override
public void run() {
try {
threads.clear();
threads.add(new Thread(new Runnable(){
public void run(){
Timer loadDataClock = new Timer();
loadDataClock.schedule(new TimerTask() {
#Override
public void run() {
try {
loadData();
} catch (Exception e) {
advancedStackTrace(e, 1);
}
}
}, 0, 5000);
}
}));
threads.get(0).start();
} catch (Exception e) {
e.printStackTrace();
}
}
}, 0, 10 * 60 * 1000);
(This works by adding Threads to a linked list every 10 minutes and clearing the LinkedList every 10 minutes and replacing it with a new Thread)
Code for loading in saving data within a DataHandler class:
public static void save(Serializable data, String filename) throws IOException {
haltLoad = true;
try {
try (ObjectOutputStream oos = new ObjectOutputStream(Files.newOutputStream(Paths.get(filename + "_fir")))) {
oos.writeObject(data);
try {
Path source = Paths.get(filename + "_fir");
Path target = Paths.get(filename);
Files.copy(source, target, StandardCopyOption.REPLACE_EXISTING);
} catch (IOException e) {
e.printStackTrace();
}
}
} catch (Exception e) {
e.printStackTrace();
}
haltLoad = false;
}
public static Object load(String filename) throws ClassNotFoundException, IOException {
if (!haltLoad) {
try (ObjectInputStream ois = new ObjectInputStream(Files.newInputStream(Paths.get(filename)))) {
return ois.readObject();
} catch (Exception e) {
return null;
}
} else {
return null;
}
}
(As a note, the variable haltLoad prevents loading while the save method is being called)
I've an already works code to read an excel file. At first, the API in the controller clase receive the file as MultipartFile type. Then, because of some reasons, I need to convert the MultipartFile into File type. Here is the code:
private static File convert(MultipartFile file) throws IOException {
try {
File convertedFile = new File(file.getOriginalFilename());
convertedFile.createNewFile();
FileOutputStream fos = new FileOutputStream(convertedFile);
fos.write(file.getBytes());
fos.close();
return convertedFile;
} catch (IOException e) {
e.printStackTrace();
throw new IOException("Error in converting the file with error message: " + e.getMessage());
}
}
Here is the service class which called in the controller, which call the convert method above:
public void create(MultipartFile file) throws Exception {
try {
File newFile = convert(file);
// rest of code
} catch (Exception e) {
// rest of code
}
}
Before I try to call the service in a new thread, code above works fine. But, when I try to call the service in a new thread, like below code, it says java.io.FileNotFoundException (The system cannot find the file specified), and the main problem is in this line fos.write(file.getBytes());. Here is how I create the new Thread in the controller:
#RequestMapping(method = RequestMethod.POST, value = "uploadfile")
public ResponseEntity<?> create(#RequestParam (value = "file", required = false) MultipartFile file) throws Exception {
try {
// ...other process
// ================================== code below not work
Thread create;
create = new Thread() {
public void run() {
try {
service.create(file);
} catch (Exception e) {
e.printStackTrace();
}
}
};
create.start();
// ================================== code below not work
Thread t1 = new Thread(new Runnable() {
public void run() {
try {
service.create(file);
} catch (Exception e) {
e.printStackTrace();
}
}
});
t1.start();
// ================================== code below not work
new Thread(() -> {
try {
service.create(file);
} catch (Exception e) {
e.printStackTrace();
}
}){{start();}};
// ...rest of code
} catch (Exception e) {
// ...rest of code
}
}
above is several way how I try to make the thread but none of them works, result with same execption.
Objective: In short, I want to make the file reading in background and immediately send response to client after the service called.
Spring supports async way by return a Callable object. the pseudo code is like:
#RequestMapping(method = RequestMethod.POST, value = "uploadfile")
public Callable<ResponseEntity<?>> create(#RequestParam (value = "file", required = false) MultipartFile file) throws Exception {
return () -> {
try {
service.create(file);
return ResponseEntity.ok()
} catch (Exception e) {
return ResponseEntity.error( /*some error*/
e.printStackTrace();
}
};
}
Here's a tutorial reference : https://niels.nu/blog/2016/spring-async-rest.html .
another reference: How to make a async rest with Spring?
I am writing a routine that will retrieve a list of URLs from a file, get the content for each URL using JSoup, find certain patterns and write the findings in output files (one for each URL that was analyzed).
I have a WebPageAnalysisTask (which implements Callable) and by now it is returning null, but it will return an object that holds the results of the processing (to be done):
public WebPageAnalyzerTask(String targetUrl, Pattern searchPattern) {
this.targetUrl = targetUrl;
this.searchPattern = searchPattern;
}
#Override
public WebPageAnalysisTaskResult call() throws Exception {
long startTime = System.nanoTime();
String htmlContent = this.getHtmlContentFromUrl();
List<String> resultContent = this.getAnalysisResults(htmlContent);
try (BufferedWriter bw = Files.newBufferedWriter(Paths.get("c:/output", UUID.randomUUID().toString() + ".txt"),
StandardCharsets.UTF_8, StandardOpenOption.WRITE)) {
bw.write(parseListToLine(resultContent));
}
long endTime = System.nanoTime();
return null;
}
I am writing the file using NIO and try-with-resources.
The code that will use the task is the following:
/**
* Starts the analysis of the Web Pages retrieved from the input text file using the provided pattern.
*/
public void startAnalysis() {
List<String> urlsToBeProcessed = null;
try (Stream<String> stream = Files.lines(Paths.get(this.inputPath))) {
urlsToBeProcessed = stream.collect(Collectors.toList());
if (urlsToBeProcessed != null && urlsToBeProcessed.size() > 0) {
List<Callable<WebPageAnalysisTaskResult>> pageAnalysisTasks = this
.buildPageAnalysisTasksList(urlsToBeProcessed);
ExecutorService executor = Executors.newFixedThreadPool(THREAD_POOL_SIZE);
List<Future<WebPageAnalysisTaskResult>> results = executor.invokeAll(pageAnalysisTasks);
executor.shutdown();
} else {
throw new NoContentToProcessException();
}
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* Builds a list of tasks in which each task will be filled with data required for the analysis processing.
* #param urlsToBeProcessed The list of URLs to be processed.
* #return A list of tasks that must be handled by an executor service for asynchronous processing.
*/
private List<Callable<WebPageAnalysisTaskResult>> buildPageAnalysisTasksList(List<String> urlsToBeProcessed) {
List<Callable<WebPageAnalysisTaskResult>> tasks = new ArrayList<>();
UrlValidator urlValidator = new UrlValidator(ALLOWED_URL_SCHEMES);
urlsToBeProcessed.forEach(urlAddress -> {
if (urlValidator.isValid(urlAddress)) {
tasks.add(new WebPageAnalyzerTask(urlAddress, this.targetPattern));
}
});
return tasks;
}
The file holding the URLs list is being read once. The ExecutorService creates the task for each URL and will analyze and write the file with results asynchronously.
By now the file is being read and the content of HTML for each URL is being analyzed and saved in a String. However the task is not writing the file. So I wonder to know what could be happening there.
Can somebody tell me if I am missing something?
Thanks in advance.
Probably you're getting an exception in the following try block
try (BufferedWriter bw = Files.newBufferedWriter(Paths.get("c:/output", UUID.randomUUID().toString() + ".txt"),
StandardCharsets.UTF_8, StandardOpenOption.WRITE)) {
bw.write(parseListToLine(resultContent));
}
Try to add a catch block to it and print the exception if it actually happens to see what causes it
catch (IOException e) {
// Replace with logger or some kind of error handling in production code
e.printStackTrace();
}
Because of the task will run error at the method call() in class WebPageAnalyzerTask,so you should check the result of List<Future<WebPageAnalysisTaskResult>> results = executor.invokeAll(pageAnalysisTasks); and determine what error occur when the task is running.
for (Future<WebPageAnalysisTaskResult> future : results ) {
try {
future.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
}
I need to read a properties file containing some configuration data in a JSF web application.
Right now the code looks like this
private Properties getConfig() {
Properties properties = new Properties();
InputStream inputStream = null;
try {
inputStream = this.getClass().getResourceAsStream("/config.properties");
try {
properties.load(inputStream);
} catch (IOException e) {
logger.error("Error while reading config properties", e);
}
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
logger.error(e.getMessage(), e);
}
}
}
return properties;
}
Is it safe to do it this way or can I run into concurrency issues when multiple threads are calling getConfig()?
No, that should be perfectly safe. I can't see any concurrency issues in there.
However, the exception handling might not be ideal - is it valid to return an empty properties object if you fail to load the config, or should you propagate the exception out of getConfig()? Up to you, really....
I'm trying to enter some value in external application using Java.
Java application looks like this:
Runtime runtime = Runtime.getRuntime();
// ... str build ...
proc = runtime.exec(str);
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(proc.getOutputStream()));
bw.write(value);
bw.flush();
bw.close();
if (proc.waitFor() != 0)
// error msg
// the end
Application hangs at waitFor method.
External application looks like this:
welcome banner
please enter 8 character input:
Welcome banner is printed using printf and input is taken with SetConsoleMode/ReadConsoleInput. ReadConsoleInput reads one char and they are masked with * character.
Help
you can use:
proc.getOutputStream().write("some date".getBytes())
keep in mind that you HAVE to read everything the app send to stdout and stderr, else it might get stuck writing there.
I use a generic class to read it in a different thread.
usage is like:
InputStreamSucker inSucker = new InputStreamSucker(proc.getInputStream());
InputStreamSucker errSucker = new InputStreamSucker(proc.getErrorStream());
proc.waitFor();
int exit = process.exitValue();
inSucker.join();
errSucker.join();
InputStreamSucker code is here:
public class InputStreamSucker extends Thread
{
static Logger logger = Logger.getLogger(InputStreamSucker.class);
private final BufferedInputStream m_in;
private final ByteArrayOutputStream m_out;
private final File m_outFile;
public InputStreamSucker(InputStream in) throws FileNotFoundException
{
this(in, null);
}
public InputStreamSucker(InputStream in, File outFile) throws FileNotFoundException
{
m_in = new BufferedInputStream(in, 4096);
m_outFile = outFile;
m_out = new ByteArrayOutputStream();
start();
}
#Override
public void run()
{
try
{
int c;
while ((c = m_in.read()) != -1)
{
m_out.write(c);
}
}
catch (IOException e)
{
logger.error("Error pumping stream", e);
}
finally
{
if (m_in != null)
{
try
{
m_in.close();
}
catch (IOException e)
{
}
}
try
{
m_out.close();
}
catch (IOException e)
{
logger.error("Error closing out stream", e);
}
if (m_outFile != null)
{
byte data[] = m_out.toByteArray();
if (data.length > 0)
{
FileOutputStream fo = null;
try
{
fo = new FileOutputStream(m_outFile);
fo.write(data);
}
catch (IOException e)
{
logger.error("Error writing " + m_outFile);
}
finally
{
try
{
if (fo != null) fo.close();
}
catch (IOException e)
{
logger.error("Error closing " + m_outFile);
}
}
}
}
}
}
public String getOutput()
{
return new String(m_out.toByteArray());
}
}
Got the answer! The trick is to use WriteConsoleInput() API because program expects keyboard event, not text ... That's why the waitFor() waited forever! :)