Non-blocking FileWalkTree file search with Webflux

Non-blocking FileWalkTree file search with Webflux - java

I need to create a non-blocking functionality where I search text files within a given folder and it returns the count of search terms found in it.
I am able to execute the test in a blocking manner. I am wondering if anyone could help me transform it into a non-blocking task so that whenever a file is finished being scanned the result delivered without waiting for all the files to be scanned.
The main point is: I don't want to wait for all files to be scanned to start delivering the result to the client (Angular app).
public interface SearchService {
List<SearchResponse> search(SearchRequest searchRequest);
}
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.ToString;
import java.io.Serializable;
#ToString
#Getter
#RequiredArgsConstructor(staticName = "of")
public class SearchResponse implements Serializable {
private final String server;
private final String filePath;
private final long count;
private final boolean errorReadingFile;
}
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.ToString;
import javax.validation.constraints.NotNull;
import java.io.Serializable;
#ToString
#Getter
#RequiredArgsConstructor(staticName = "of")
public class SearchRequest implements Serializable {
#NotNull
private final String server;
#NotNull
private final String rootPath;
#NotNull
private final String searchTerm;
}
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import java.io.IOException;
import java.nio.file.*;
import java.util.ArrayList;
import java.util.EnumSet;
import java.util.List;
#Slf4j
#Service
public class FileSearchService implements SearchService {
#Override
public List<SearchResponse> search(SearchRequest searchRequest) {
Path start = Paths.get(searchRequest.getRootPath());
EnumSet<FileVisitOption> opts = EnumSet.of(FileVisitOption.FOLLOW_LINKS);
int maxDepth = Integer.MAX_VALUE;
SearchTermFileVisitor visitor = new SearchTermFileVisitor(searchRequest, new ArrayList<>());
try {
Files.walkFileTree(start,opts,maxDepth, visitor);
return visitor.getSearchResponseList();
} catch (IOException e) {
System.out.println(e.getMessage());
}
}
}
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import java.io.BufferedReader;
import java.io.IOException;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Stream;
#Slf4j
#Getter
#RequiredArgsConstructor
public class SearchTermFileVisitor extends SimpleFileVisitor<Path> {
private final SearchRequest searchRequest;
private final List<SearchResponse> searchResponseList;
private SearchResponse searchFileContent(Path path, SearchRequest searchRequest) {
SearchResponse response;
try (BufferedReader br = Files.newBufferedReader(path)) {
response = SearchResponse.of(
searchRequest.getServer(),
Paths.get(path.toUri()).toString(),
countWordsInFile(searchRequest.getSearchTerm(), br.lines()),
false);
} catch (Exception e) {
response = SearchResponse.of(
searchRequest.getServer(),
path.toString(),
0,
true);
}
log.debug(response.toString());
return response;
}
private int countWordsInFile(String searchTerm, Stream<String> linesStream) {
return linesStream
.parallel()
.map(line -> countWordsInLine(line, searchTerm))
.reduce(0, Integer::sum);
}
private int countWordsInLine(String line, String searchTerm) {
Pattern pattern = Pattern.compile(searchTerm.toLowerCase());
Matcher matcher = pattern.matcher(line.toLowerCase());
int count = 0;
int i = 0;
while (matcher.find(i)) {
count++;
i = matcher.start() + 1;
}
return count;
}
private boolean isTextFile(Path path) throws IOException {
String type = Files.probeContentType(path);
if (type == null) {
//type couldn't be determined, assume binary
return false;
} else if (type.startsWith("text")) {
return true;
} else {
//type isn't text
return false;
}
}
#Override
public FileVisitResult postVisitDirectory(Path dir, IOException exc) {
log.debug("Visited: " + (Path) dir);
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
if (attrs.isRegularFile()
&& !attrs.isDirectory()
&& !attrs.isSymbolicLink()
&& isTextFile(file)) {
searchResponseList.add(searchFileContent(file, searchRequest));
}
return FileVisitResult.CONTINUE;
}
#Override
public FileVisitResult visitFileFailed(Path file, IOException exc) {
return FileVisitResult.CONTINUE;
}
}
The test case:
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.net.URISyntaxException;
import java.nio.file.Paths;
import java.util.List;
class FileSearchServiceTest {
private SearchService searchService = new FileSearchService();
#Test
void test_search_window_root_c_path() {
SearchRequest sr = SearchRequest.of("localhost", "c:\\", "a");
final List<SearchResponse> searchResult = searchService.search(sr);
Assertions.assertNotNull(searchResult.size());
}
}
I want to use WebFlux to receive the results one by one without waiting for all the files to be scanned.

Consider (1) performing the search in a separate thread, (2) using the observer pattern for receiving intermediate results back to the thread created code (references below), and (3) join the search thread to the implementing code so you can return the result list once it is finished. It means you need to pass 'this' to the thread so it has a reference to call back to a separate method for intermediate results. The references below contain sample code.
Think of this like a GUI. You can run the GUI in a separate thread, and for every button click it calls back to the controller code (including a 'finish' once Exit is clicked).
Re: https://dzone.com/articles/the-observer-pattern-using-modern-java
https://en.wikipedia.org/wiki/Observer_pattern

Related

How to make spring batch step execution parallel with configurable thread count?

I am having following spring-batch application
SpringBatchApplication.java
package com.spbt.job.sample;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class SpringBatchApplication {
public static void main(String[] args) {
SpringApplication.run(SpringBatchApplication.class, args);
}
}
TraverseJob.java
package com.spbt.job.sample;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
#Configuration
#EnableBatchProcessing
public class TraverseJob {
#Autowired
protected JobBuilderFactory jobBuilderFactory;
#Autowired
protected StepBuilderFactory stepBuilderFactory;
private String inputFolderPath = "/tmp/inputFolder";
#Bean("TraverseJob")
public Job job() {
return jobBuilderFactory.get("TraverseJob")
.incrementer(new RunIdIncrementer())
.start(traverseStep())
.build();
}
#Bean("TraverseStep")
public Step traverseStep() {
return stepBuilderFactory.get("TraverseStep")
.tasklet(traverseJobTasklet(null))
.build();
}
#Bean("TraverseJobTasklet")
#StepScope
public TraverseJobTasklet traverseJobTasklet(#Value("#{jobParameters[date]}") String date) {
TraverseJobTasklet tasklet = new TraverseJobTasklet();
tasklet.setJobDate(date);
tasklet.setJobDirPath(inputFolderPath);
return tasklet;
}
}
TraverseJobTasklet.java
package com.spbt.job.sample;
import org.springframework.batch.core.StepContribution;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.beans.factory.annotation.Autowired;
import java.io.File;
public class TraverseJobTasklet implements Tasklet {
private String jobDirPath;
private String jobDate;
#Autowired
private RemoteFilePush remoteFilePush;
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception {
try {
traverseDir(new File(jobDirPath));
} catch (Exception ex) {
throw ex;
}
return RepeatStatus.FINISHED;
}
private void traverseDir(File filePath) throws Exception {
try {
File[] files = filePath.listFiles();
if (files != null) {
for (File file : files) {
String name = file.getName();
if (file.isDirectory()) {
if (remoteFilePush.isRemoteDirExist(name)) {
continue;
} else {
remoteFilePush.createRemoteDir(name);
traverseDir(file);
}
} else {
remoteFilePush.pushFile(file.getPath());
}
}
} else {
throw new Exception("empty/null dir -> " + filePath.getName());
}
} catch (Exception ex) {
throw ex;
}
}
public String getJobDirPath() {
return jobDirPath;
}
public void setJobDirPath(String jobDirPath) {
this.jobDirPath = jobDirPath;
}
public String getJobDate() {
return jobDate;
}
public void setJobDate(String jobDate) {
this.jobDate = jobDate;
}
}
RemoteFilePushLogic.java
package com.spbt.job.sample;
import org.springframework.stereotype.Component;
#Component
public class RemoteFilePush {
public boolean isRemoteDirExist(String name) throws InterruptedException {
boolean isRemoteDirExist = false;
// code to check dir on remote server
return isRemoteDirExist;
}
public void createRemoteDir(String name) throws InterruptedException {
// code to create dir on remote server
}
public void pushFile(String path) throws InterruptedException {
// code to push file on remote server
System.out.println("Pushed");
}
}
I want to do parallel traversal and execution in traverseDir method in TraverseJobTasklet, by keeping my RemoteFilePush Logic intact, my inputFolderPath can contain multiple child directories each of which contains some files in it.
I have tried to follow link for spring-batch version which I am using, But its xml based and I do not seem to get how can I create multiple step out of single traverseStep I have?

input a sub-folder string path per worker step is where i am hitting wall with spring code, if you can point me to some ref. it will be helpful, most of the example on net is xml based.
Here is a quick self-contained example with Java config:
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.JobParametersBuilder;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.StepContribution;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepScope;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.partition.support.Partitioner;
import org.springframework.batch.core.scope.context.ChunkContext;
import org.springframework.batch.core.step.tasklet.Tasklet;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.repeat.RepeatStatus;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.task.SimpleAsyncTaskExecutor;
#Configuration
#EnableBatchProcessing
public class PartitionJobSample {
private final JobBuilderFactory jobs;
private final StepBuilderFactory steps;
public PartitionJobSample(JobBuilderFactory jobs, StepBuilderFactory steps) {
this.jobs = jobs;
this.steps = steps;
}
#Bean
public Step managerStep() {
return steps.get("masterStep")
.partitioner(workerStep().getName(), partitioner(null))
.step(workerStep())
.gridSize(4)
.taskExecutor(taskExecutor())
.build();
}
#Bean
public SimpleAsyncTaskExecutor taskExecutor() {
return new SimpleAsyncTaskExecutor();// TODO useful for testing, use a more robust task executor in production
}
#Bean
#StepScope
public Partitioner partitioner(#Value("#{jobParameters['rootFolder']}") String rootFolder) {
List<String> subFolders = getSubFolders(rootFolder);
return new Partitioner() {
#Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> map = new HashMap<>(gridSize);
for (String folder : subFolders) {
ExecutionContext executionContext = new ExecutionContext();
executionContext.put("filePath", folder);
map.put("partition-for-" + folder, executionContext);
}
return map;
}
};
}
private List<String> getSubFolders(String rootFolder) {
// TODO implement this
return Arrays.asList("/data/folder1", "/data/folder2");
}
#Bean
public Step workerStep() {
return steps.get("workerStep")
.tasklet(getTasklet(null))
.build();
}
#Bean
#StepScope
public Tasklet getTasklet(#Value("#{stepExecutionContext['filePath']}") String filePath) {
return new TraverseJobTasklet(filePath);
}
#Bean
public Job job() {
return jobs.get("job")
.start(managerStep())
.build();
}
public static void main(String[] args) throws Exception {
ApplicationContext context = new AnnotationConfigApplicationContext(PartitionJobSample.class);
JobLauncher jobLauncher = context.getBean(JobLauncher.class);
Job job = context.getBean(Job.class);
JobParameters jobParameters = new JobParametersBuilder()
.addString("rootFolder", "/data")
.toJobParameters();
jobLauncher.run(job, jobParameters);
}
class TraverseJobTasklet implements Tasklet {
private String filePath;
public TraverseJobTasklet(String filePath) {
this.filePath = filePath;
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
// TODO call traversePath for filePath which is a sub-folder here
System.out.println(Thread.currentThread().getName() + " processing sub-folder " + filePath);
return RepeatStatus.FINISHED;
}
}
}
It passes the root directory as a job parameter and executes a partitioned step where each worker processes a sub-folder (calling your tasklet).
If you run it, you should see something like:
SimpleAsyncTaskExecutor-2 processing sub-folder /data/folder1
SimpleAsyncTaskExecutor-1 processing sub-folder /data/folder2
I will let you adapt it to your situation accordingly.

Mock an instance method call of another class that takes a URL and File object and writes a file to a HTTP connection

I'm trying to write UTs for a file called DocumentStoreAccessor.java. Here is the class below:
package com.company.main.accessor;
import com.company.main.dagger.component.AccessorComponent;
import com.company.main.dagger.component.DaggerAccessorComponent;
import com.company.main.util.aws.s3.AWSS3Util;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
#Slf4j
#RequiredArgsConstructor
public class DocumentStoreAccessor {
private final DocumentStore documentStoreClient; //Comes from Dagger
public DocumentStoreAccessor() {
AccessorComponent accessorComponent = DaggerAccessorComponent.create();
this.documentStoreClient = accessorComponent.provideDocumentStoreClient();
}
private int putContentsIntoS3(CreateUploadS3UrlResult createUploadS3UrlResponse,
#NonNull File file) {
int uploadStatusCode = 0;
try {
URL url = new URL(createUploadS3UrlResponse.getS3Url());
uploadStatusCode = new AWSS3Util().upload(url, file); //Instance comes from a util class
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return uploadStatusCode;
}
public String uploadFile(File file, DocumentFileExtension fileExtension) throws Exception {
String documentId = null;
CreateUploadS3UrlResult createUploadS3Urlresult = documentStoreClient.createUploadS3Url(new CreateUploadS3UrlRequest());
int putContentsStatusCode = putContentsIntoS3(createUploadS3Urlresult, file);
if (putContentsStatusCode == 200) {
CreateDocumentRequest createDocumentRequest = new CreateDocumentRequest()
CreateDocumentResult document = documentStoreClient.createDocument(createDocumentRequest);
documentId = document.getDocumentId();
} else {
throw new Exception("Status code is: " + putContentsStatusCode);
}
return documentId;
}
}
Inside this file I do new AWSS3Util().upload(url, file).
And here is the AWSS3Util.java
package com.company.main.util.aws.s3;
import com.company.main.exception.DataAccessException;
import com.company.main.exception.RetriableDataAccessException;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.NonNull;
import lombok.extern.slf4j.Slf4j;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.Reader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
#Slf4j
#AllArgsConstructor
#Builder
public class AWSS3Util {
private static final String PUT_REQUEST_METHOD = "PUT";
public int upload(#NonNull final URL url, #NonNull final File file) throws IOException {
final InputStream inputStream = new FileInputStream(file);
final Reader fileReader = new InputStreamReader(inputStream, StandardCharsets.UTF_8);
final BufferedReader br = new BufferedReader(fileReader);
HttpURLConnection connection = null;
int responseCode = 0;
try {
// Create the connection and use it to upload the new object using the pre-signed URL.
connection = create(url);
connection.setDoOutput(true);
connection.setRequestMethod(PUT_REQUEST_METHOD);
final OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream(), StandardCharsets.UTF_8);
String st;
while ((st = br.readLine()) != null) {
out.write(st);
}
out.close();
responseCode = connection.getResponseCode();
} catch (IOException e) {
throw new RetriableDataAccessException(String.format("S3 upload request failed with request: %s", url.toString()), e);
} finally {
inputStream.close();
fileReader.close();
br.close();
}
return responseCode;
}
private HttpURLConnection create(URL url) throws IOException {
return (HttpURLConnection) url.openConnection();
}
}
I want to make new AWSS3Util().upload(url, file) return a 200..
I'm unable to do so.. I keep getting a NullPointerException. Here is what I have for the past day:
package com.company.main.accessor;
import com.company.main.util.aws.s3.AWSS3Util;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLStreamHandler;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
#ExtendWith(MockitoExtension.class)
public class DocumentStoreAccessorTest {
#Mock
private DocumentStore mockDocumentStoreClient;
#Mock
private HttpURLConnection httpURLConnection;
#Mock
private OutputStream outputStream;
#InjectMocks
private DocumentStoreAccessor classUnderTest;
private URL url;
private AWSS3Util awss3Util;
private CreateUploadS3UrlRequest dummyCreateUploadS3UrlRequest;
private CreateUploadS3UrlResult dummyCreateUploadS3UrlResult;
private CreateDocumentRequest dummyCreateDocumentRequest;
private CreateDocumentResult dummyCreateDocumentResult;
#BeforeEach
void setUp() throws IOException {
awss3Util = new AWSS3Util();
url = getMockUrl(httpURLConnection);
dummyCreateUploadS3UrlRequest = new CreateUploadS3UrlRequest();
dummyCreateUploadS3UrlResult = new CreateUploadS3UrlResult().withS3Url("http://foo.io://:99");
dummyCreateDocumentRequest = new CreateDocumentRequest();
dummyCreateDocumentResult = new CreateDocumentResult().withDocumentId("foo");
}
#Test
void uploadSuccess() throws IOException {
when(mockDocumentStoreClient.createUploadS3Url(dummyCreateUploadS3UrlRequest)).thenReturn(dummyCreateUploadS3UrlResult);
AWSS3Util aSpy = Mockito.spy(awss3Util);
Mockito.when(aSpy.upload(url, getDataToUpload())).thenReturn(200);
when(mockDocumentStoreClient.createDocument(dummyCreateDocumentRequest)).thenReturn(dummyCreateDocumentResult);
String id = classUnderTest.uploadFile(getDataToUpload(), DocumentFileExtension.XLSX);
assertEquals(id, "foo");
verify(mockDocumentStoreClient, times(1)).createUploadS3Url(dummyCreateUploadS3UrlRequest);
}
private File getDataToUpload() {
return new File("TestFileName.xlsx");
}
/**
* We cannot directly use Mockito to mock URL. This helper method, helps us to create the mock url.
* <p>
* https://stackoverflow.com/questions/565535/mocking-a-url-in-java
*/
private URL getMockUrl(HttpURLConnection httpURLConnection) throws IOException {
final URLStreamHandler handler = new URLStreamHandler() {
#Override
protected URLConnection openConnection(final URL arg0)
throws IOException {
return httpURLConnection;
}
};
final URL url = new URL("http://foo.io", "foo.io", 80, "", handler);
return url;
}
}
I'm unable to mock the URL class, so I followed a another Stackoverflow post..
The AWSS3Util cannot come through Dagger, it is a util class that we're all using so this must not change.
I'm not sure if I'm going in the right direction.. I've tried "spying" on that AWSS3Util class. I want this method to return a 200 or any status code of my choice to cover them in UTs by asserting a String return as seen in the example below
I can change the the upload method inside AWSS3Util to static if it helps UTs
Cannot use PowerMockito (this is a last resort)
Try incorporating Yan's comments:
#Test
void verifyCreateUploadS3UrlInvocation() throws Exception {
when(mockDocumentStoreClient.createUploadS3Url(dummyCreateUploadS3UrlRequest)).thenReturn(dummyCreateUploadS3UrlResult);
when(httpURLConnection.getOutputStream()).thenReturn(outputStream);
when(httpURLConnection.getResponseCode()).thenReturn(200);
when(awss3Util.upload(url, getDataToUpload())).thenReturn(200);
when(mockDocumentStoreClient.createDocument(dummyCreateDocumentRequest)).thenReturn(dummyCreateDocumentResult);
String id = classUnderTest.uploadFile(getDataToUpload(), DocumentFileExtension.XLSX);
assertEquals(id, "foo");
verify(mockDocumentStoreClient, times(1)).createUploadS3Url(dummyCreateUploadS3UrlRequest);
}
I get a 405 thrown when I specifically want it to send a 200, and therefore java.lang.Exception: Status code is: 405 thrown from my function.

Firstly I would change a little bit AWSS3Util, because reading binary files as string can lead to very interesting results.
import lombok.NonNull;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
public class AWSS3Util {
private static final String PUT_REQUEST_METHOD = "PUT";
public int upload(#NonNull final URL url, #NonNull final File file) throws IOException {
HttpURLConnection connection = create(url);
try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file))) {
connection.setDoOutput(true);
connection.setRequestMethod(PUT_REQUEST_METHOD);
try (BufferedOutputStream bos = new BufferedOutputStream(connection.getOutputStream())) {
byte[] buffer = new byte[1024];
int len;
while ((len = bis.read(buffer)) > 0) {
bos.write(buffer, 0, len);
}
}
return connection.getResponseCode();
}
}
private HttpURLConnection create(URL url) throws IOException {
return (HttpURLConnection) url.openConnection();
}
}
Test for upload method could be like this:
import org.junit.jupiter.api.Test;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.net.URLStreamHandler;
import java.nio.file.Files;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class MyTest {
#Test
public void test() throws IOException {
final HttpURLConnection mockUrlCon = mock(HttpURLConnection .class);
URLStreamHandler stubUrlHandler = new URLStreamHandler() {
#Override
protected URLConnection openConnection(URL u) throws IOException {
return mockUrlCon;
}
};
URL url = new URL("http://foo.io", "foo.io", 80, "", stubUrlHandler);
when(mockUrlCon.getResponseCode()).thenReturn(200);
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
when(mockUrlCon.getOutputStream()).thenReturn(outputStream);
File file = new File("c:\\anyFile.png");
int responseCode = new AWSS3Util().upload(url, file);
assertEquals(200, responseCode);
byte[] expectedBytes = Files.readAllBytes(file.toPath());
byte[] actualBytes = outputStream.toByteArray();
assertArrayEquals(expectedBytes, actualBytes);
}
}

I had to violate the second rule - I had to DependencyInject it via Dagger
package com.company.main.dagger.module;
import com.company.main.util.aws.s3.AWSS3Util;
import dagger.Module;
import dagger.Provides;
import javax.inject.Singleton;
#Module
public class AWSS3UtilModule {
#Provides
#Singleton
public static AWSS3Util provideAwss3Util() {
return new AWSS3Util();
}
}
and then in my component
import javax.inject.Singleton;
#Singleton
#Component(modules = {
ShipDocsDMSAccessorModule.class,
AWSS3UtilModule.class
})
public interface AccessorComponent {
ShipDocsDMS provideShipDocsDMSClient();
AWSS3Util provideAwss3Util();
}
and then in my main class
#Slf4j
#RequiredArgsConstructor
public class ShipDocsDMSAccessor {
private final ShipDocsDMS shipDocsDMSClient;
private final AWSS3Util awss3Util;
public ShipDocsDMSAccessor() {
AccessorComponent accessorComponent = DaggerAccessorComponent.create();
this.shipDocsDMSClient = accessorComponent.provideShipDocsDMSClient();
this.awss3Util = accessorComponent.provideAwss3Util();
}
.
.
.
.
and then in my test, I can freely mock AWSS3Util dependency and carry ahead and force out the required response.

Process file (which sit on Server A) on Server B

Since Server A and server B doesn't have SFTP, and I am try to implement a web service on Server B, that takes the file on Server A and process it. I tried using Spring boot to do this, like save the file first, and the process. But this way it seems like async, which means when the code try to process the file, the file is not ready yet, (confirm when I print the file location, it return null). What is the good way to handle this?
Current Code I have for the controller is following:
package com.example.filedemo.controller;
import com.example.filedemo.payload.UploadFileResponse;
import com.example.filedemo.service.FileStorageService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.io.Resource;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.servlet.support.ServletUriComponentsBuilder;
import javax.servlet.http.HttpServletRequest;
import javax.xml.ws.Response;
import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
#RestController
public class FileController {
private static final Logger logger = LoggerFactory.getLogger(FileController.class);
private static final String PYTHON_FILE = "V:/speechRecognition/audio_transcribe.py";
#Autowired
private FileStorageService fileStorageService;
#PostMapping("/uploadFile")
public UploadFileResponse uploadFile(#RequestParam("file") MultipartFile file) {
String fileName = fileStorageService.storeFile(file);
String fileDownloadUri =
ServletUriComponentsBuilder.fromCurrentContextPath().path("/downloadFile/").path(fileName).toUriString();
return new UploadFileResponse(fileName, fileDownloadUri, file.getContentType(), file.getSize());
}
#PostMapping("/processFile")
public ResponseEntity<String> processFile(#RequestParam("file") MultipartFile file) throws IOException {
String filename = uploadFile(file).getFileName();
File actualFile = new File("E:\\Audio\\uploads\\" + filename);
String fetching = "python " + PYTHON_FILE + " " + actualFile.getAbsolutePath();
String[] command = new String[] {"cmd.exe", "/c", fetching};
Process p = Runtime.getRuntime().exec(command);
String pythonPath = System.getProperty("PYTHON_PATH");
System.out.println("pythonPath is " + pythonPath);
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String ret = in.readLine();
System.out.println(ret);
return new ResponseEntity<String>("Success", HttpStatus.OK);
}
}
FileStorageService.java
package com.example.filedemo.service;
import com.example.filedemo.exception.FileStorageException;
import com.example.filedemo.exception.MyFileNotFoundException;
import com.example.filedemo.property.FileStorageProperties;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.io.Resource;
import org.springframework.core.io.UrlResource;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import org.springframework.web.multipart.MultipartFile;
import java.io.IOException;
import java.net.MalformedURLException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
#Service
public class FileStorageService {
private final Path fileStorageLocation;
#Autowired
public FileStorageService(FileStorageProperties fileStorageProperties) {
this.fileStorageLocation = Paths.get(fileStorageProperties.getUploadDir())
.toAbsolutePath().normalize();
try {
Files.createDirectories(this.fileStorageLocation);
} catch (Exception ex) {
throw new FileStorageException("Could not create the directory where the uploaded files will be stored.", ex);
}
}
public String storeFile(MultipartFile file) {
// Normalize file name
String fileName = StringUtils.cleanPath(file.getOriginalFilename());
try {
// Check if the file's name contains invalid characters
if(fileName.contains("..")) {
throw new FileStorageException("Sorry! Filename contains invalid path sequence " + fileName);
}
// Copy file to the target location (Replacing existing file with the same name)
Path targetLocation = this.fileStorageLocation.resolve(fileName);
long numberOfByte = Files.copy(file.getInputStream(), targetLocation, StandardCopyOption.REPLACE_EXISTING);
System.out.println("Copy byte " + numberOfByte);
return fileName;
} catch (IOException ex) {
throw new FileStorageException("Could not store file " + fileName + ". Please try again!", ex);
}
}
public Resource loadFileAsResource(String fileName) {
try {
Path filePath = this.fileStorageLocation.resolve(fileName).normalize();
Resource resource = new UrlResource(filePath.toUri());
if(resource.exists()) {
return resource;
} else {
throw new MyFileNotFoundException("File not found " + fileName);
}
} catch (MalformedURLException ex) {
throw new MyFileNotFoundException("File not found " + fileName, ex);
}
}
}
UploadFileResponse
package com.example.filedemo.payload;
public class UploadFileResponse {
private String fileName;
private String fileDownloadUri;
private String fileType;
private long size;
public UploadFileResponse(String fileName, String fileDownloadUri, String fileType, long size) {
this.fileName = fileName;
this.fileDownloadUri = fileDownloadUri;
this.fileType = fileType;
this.size = size;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public String getFileDownloadUri() {
return fileDownloadUri;
}
public void setFileDownloadUri(String fileDownloadUri) {
this.fileDownloadUri = fileDownloadUri;
}
public String getFileType() {
return fileType;
}
public void setFileType(String fileType) {
this.fileType = fileType;
}
public long getSize() {
return size;
}
public void setSize(long size) {
this.size = size;
}
}

I would recommend storing the file in the database, then operating on it from there. Obviously, be careful the file size doesn't overrun the column size.
If your operations require an actual file on disk (rather than, say, a stream of bytes), then you could create a sync service that polls the database for new files and saves a copy on the local disk when found. Then your processing code can use the local disk file handle.

How to mock a void method which acquires the lock in Zookeepr and we can also verify the locks

I have to create a unit testcases for a method which acquires the lock with zookeeper and data is processed with CompletableFuture are used
Below is the high level code :
import lombok.Data;
#Data
public class ConfigurationsIntegrationModel {
public enum InteractionType {
TEST,
DEV;
}
private InteractionType interactionType;
private String lockName;
}
import org.springframework.stereotype.Service;
import java.util.Arrays;
import java.util.List;
#Service("configurationsIntegrationService")
public interface ConfigurationsIntegrationService {
public default List<ConfigurationsIntegrationModel> getRecords(ConfigurationsIntegrationModel.InteractionType integrationType) {
return Arrays.asList(getDynamicIntegrationConfigurationMock(integrationType));
}
private static ConfigurationsIntegrationModel getDynamicIntegrationConfigurationMock(ConfigurationsIntegrationModel.InteractionType integrationType) {
ConfigurationsIntegrationModel configurationsIntegration = new ConfigurationsIntegrationModel();
configurationsIntegration.setLockName("Test_Lock");
configurationsIntegration.setInteractionType(integrationType);
return configurationsIntegration;
}
}
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.locks.InterProcessLock;
import org.apache.curator.framework.recipes.locks.InterProcessSemaphoreMutex;
import org.apache.curator.retry.RetryNTimes;
import java.util.concurrent.TimeUnit;
public class DistributedLockProcessor {
private CuratorFramework client;
private String path;
public DistributedLockProcessor(String host, String path) {
RetryPolicy retryPolicy = new RetryNTimes(5, 90);
client = CuratorFrameworkFactory.newClient(host, retryPolicy);
client.start();
}
public InterProcessLock acquire(String lockName) throws Exception {
InterProcessSemaphoreMutex sharedLock = new InterProcessSemaphoreMutex(client, path + "/" + lockName);
if (!sharedLock.acquire(0, TimeUnit.SECONDS)) {
return null;
}
return sharedLock;
}
public boolean release(InterProcessLock sharedLock) throws Exception {
sharedLock.release();
return true;
}
}
import org.apache.curator.framework.recipes.locks.InterProcessLock;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;
import java.util.function.Consumer;
public class LockingExecutorProcessor<T> {
private Executor executor = null;
private DistributedLockProcessor distributedLock = null;
public LockingExecutorProcessor(String host, String path, int executorCount) {
executor = Executors.newFixedThreadPool(executorCount);
distributedLock = new DistributedLockProcessor(host, path);
}
public void process(List<String> locks, List<T> items, Consumer<T> consumer) throws ExecutionException, InterruptedException {
final List<CompletableFuture<Void>> completableFutures = new ArrayList<>();
for (int i = 0; i < locks.size(); i++) {
final int record = i;
CompletableFuture<Void> future =
CompletableFuture.runAsync(
() -> {
InterProcessLock interProcessLock = null;
try {
interProcessLock = distributedLock.acquire(locks.get(record));
} catch (Exception e) {
e.printStackTrace();
}
if (interProcessLock != null) {
consumer.accept(items.get(record));
}
}, executor);
completableFutures.add(future);
}
CompletableFuture<Void> completable = CompletableFuture.allOf(completableFutures.toArray(new CompletableFuture[completableFutures.size()]));
completable.get();
}
}
import org.springframework.stereotype.Service;
import java.util.Arrays;
import java.util.List;
#Service("messageService")
public interface MessageService {
public default List<String> getMessagesList(ConfigurationsIntegrationModel.InteractionType integrationType) {
return Arrays.asList("Message1", "Message2", "Message3","Message4");
}
}
import com.google.common.util.concurrent.RateLimiter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
#Component("sampleJob")
public class SampleJob {
#Autowired
private LockingExecutorProcessor<ConfigurationsIntegrationModel> lockingExecutorProcessor;
#Autowired
private ConfigurationsIntegrationService configurationsIntegrationService;
#Autowired
private RateLimiter rateLimiter;
#Autowired
private MessageService messageService;
private List<String> getLockNames(List<ConfigurationsIntegrationModel> integrationConfigurations) {
List<String> lockNames = new ArrayList<>();
for (ConfigurationsIntegrationModel integrationConfiguration : integrationConfigurations) {
lockNames.add(integrationConfiguration.getLockName());
}
return lockNames;
}
#Scheduled(fixedRateString = "100")
public void executeJob() throws ExecutionException, InterruptedException {
List<ConfigurationsIntegrationModel> testRecords = configurationsIntegrationService.getRecords(ConfigurationsIntegrationModel.InteractionType.TEST);
List<String> lockNames = getLockNames(testRecords);
lockingExecutorProcessor.process(
lockNames,
testRecords,
recordsConfig -> {
final List<CompletableFuture<Void>> completableFutures = new ArrayList<>();
List<String> msgList = messageService.getMessagesList(recordsConfig.getInteractionType());
for (String message : msgList) {
completableFutures.add(
CompletableFuture.runAsync(
() -> {
System.out.println("Message is #### "+ message);
}, Executors.newFixedThreadPool(10)));
}
});
}
}
Below is the testcase which i tried so far :
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.MockitoAnnotations;
import org.mockito.junit.MockitoJUnitRunner;
import java.util.concurrent.ExecutionException;
#RunWith(MockitoJUnitRunner.class)
public class SampleJobTest {
#InjectMocks
private SampleJob sampleJob = new SampleJob();
#Mock
private ConfigurationsIntegrationService configurationsIntegrationService;
#Mock
private MessageService messageService;
#Mock
private LockingExecutorProcessor<ConfigurationsIntegrationModel> lockingExecutorProcessor;
#Test
public void testSampleJob() throws ExecutionException, InterruptedException {
Mockito.doCallRealMethod().when(lockingExecutorProcessor).process(Mockito.any(), Mockito.any(), Mockito.any());
Mockito.doCallRealMethod().when(configurationsIntegrationService).getRecords(Mockito.any());
Mockito.doCallRealMethod().when(messageService).getMessagesList(Mockito.any());
sampleJob.executeJob();
}
}
When i debug the code then it breaks at the line CompletableFuture.runAsync of LockingExecutorProcessor and throws null pointer error; the reason is distributedLockProcessor object is null;
How can we mock it and how to connect to test zookeeper server instead of original to ensure locking is working fine

This is done. Now the testcases are working fine. I have used curator-test dependency and also used reflection to mock the private objects.

Jackson xml map attribute value to property

I am integrating with an old system and have a need to parse the following xml into my object. I am trying to do this with jackson but I can't get the mapping to work. Anyone know how to map the following xml to the pojo?
#JacksonXmlRootElement(localName = "properties")
#Data
public class Example {
private String token;
private String affid;
private String domain;
}
xml example:
<properties>
<entry key="token">rent</entry>
<entry key="affid">true</entry>
<entry key="domain">checking</entry>
</properties>
I have tried adding
#JacksonXmlProperty(isAttribute = true, localName = "key")
to the properties but this of course doesn't work and I do not see another way to get this to work. Any ideas?
I am using the mapper like so...
ObjectMapper xmlMapper = new XmlMapper();
dto = xmlMapper.readValue(XML_STRING, Example .class);
I am using the following dependencies
compile('org.springframework.boot:spring-boot-starter-web')
runtime('org.springframework.boot:spring-boot-devtools')
compileOnly('org.projectlombok:lombok')
testCompile('org.springframework.boot:spring-boot-starter-test')
compile('org.apache.commons:commons-lang3:3.5')
compile('com.fasterxml.jackson.dataformat:jackson-dataformat-xml')
compile('com.squareup.okhttp3:okhttp:3.10.0')

This does work.
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlProperty;
import com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlText;
import javax.xml.stream.XMLInputFactory;
import javax.xml.stream.XMLStreamException;
import javax.xml.stream.XMLStreamReader;
import java.io.IOException;
import java.io.StringReader;
import java.util.List;
public class XmlParserDemo {
public static void main(String[] args) throws IOException, XMLStreamException {
String xmlString = "<properties>\n" +
" <entry key=\"token\">rent</entry>\n" +
" <entry key=\"affid\">true</entry>\n" +
" <entry key=\"domain\">checking</entry>\n" +
"</properties>";
XMLStreamReader sr = null;
sr = XMLInputFactory.newFactory().createXMLStreamReader(new StringReader(xmlString));
sr.next();
XmlMapper mapper = new XmlMapper();
List<Entry> entries = mapper.readValue(sr, new TypeReference<List<Entry>>() {
});
sr.close();
entries.forEach(e ->
System.out.println(e.key + ":" + e.value));
}
public static class Entry {
#JacksonXmlProperty(isAttribute = true, localName = "key")
private String key;
#JacksonXmlText
private String value;
public String getKey() {
return key;
}
public void setKey(String key) {
this.key = key;
}
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
}
}
Output is:
token:rent
affid:true
domain:checking

I have looked through Jackson thoroughly and it doesn't seem that there is a way to accomplish this. However, I will share my solution here in case it is useful to someone else.
package com.example.config;
import com.example.dto.Example;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.http.HttpInputMessage;
import org.springframework.http.HttpOutputMessage;
import org.springframework.http.MediaType;
import org.springframework.http.converter.AbstractHttpMessageConverter;
import org.springframework.http.converter.HttpMessageNotReadableException;
import org.springframework.http.converter.HttpMessageNotWritableException;
import org.springframework.http.converter.StringHttpMessageConverter;
import org.w3c.dom.Node;
import org.xml.sax.InputSource;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
public class Converter extends AbstractHttpMessageConverter<Example> {
private static final XPath XPATH_INSTANCE = XPathFactory.newInstance().newXPath();
private static final StringHttpMessageConverter MESSAGE_CONVERTER = new StringHttpMessageConverter();
#Override
protected boolean supports(Class<?> aClass) {
return aClass == Example.class;
}
#Override
protected Example readInternal(Class<? extends LongFormDTO> aClass, HttpInputMessage httpInputMessage) throws IOException, HttpMessageNotReadableException {
String responseString = MESSAGE_CONVERTER.read(String.class, httpInputMessage);
Reader xmlInput = new StringReader(responseString);
InputSource inputSource = new InputSource(xmlInput);
Example dto = new Example();
Node xml;
try {
xml = (Node) XPATH_INSTANCE.evaluate("/properties", inputSource, XPathConstants.NODE);
} catch (XPathExpressionException e) {
log.error("Unable to parse response", e);
return dto;
}
log.info("processing populate application response={}", responseString);
dto.setToken(getString("token", xml));
dto.setAffid(getInt("affid", xml, 36));
dto.domain(getString("domain", xml));
xmlInput.close();
return dto;
}
private String getString(String propName, Node xml, String defaultValue) {
String xpath = String.format("//entry[#key='%s']/text()", propName);
try {
String value = (String) XPATH_INSTANCE.evaluate(xpath, xml, XPathConstants.STRING);
return StringUtils.isEmpty(value) ? defaultValue : value;
} catch (XPathExpressionException e) {
log.error("Received error retrieving property={} from xml", propName, e);
}
return defaultValue;
}
private String getString(String propName, Node xml) {
return getString(propName, xml, null);
}
private int getInt(String propName, Node xml, int defaultValue) {
String stringValue = getString(propName, xml);
if (!StringUtils.isEmpty(stringValue)) {
try {
return Integer.parseInt(stringValue);
} catch (NumberFormatException e) {
log.error("Attempted to parse value={} as integer but received error", stringValue, e);
}
}
return defaultValue;
}
private int getInt(String propName, Node xml) {
return getInt(propName, xml,0);
}
private boolean getBoolean(String propName, Node xml) {
String stringValue = getString(propName, xml );
return Boolean.valueOf(stringValue);
}
#Override
protected void writeInternal(Example dto, HttpOutputMessage httpOutputMessage) throws IOException, HttpMessageNotWritableException {
throw new UnsupportedOperationException("Responses of type=" + MediaType.TEXT_PLAIN_VALUE + " are not supported");
}
}
I chose to hide this in a message converter so I don't have to look at it again but you can apply these steps where you see fit. If you choose this route, you will need to configure a rest template to use this converter. If not, it is important to cache the xml into a Node object as regenerating each time will be very costly.
package com.example.config;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.http.MediaType;
import org.springframework.http.client.OkHttp3ClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
#Configuration
public class RestConfig {
#Bean
#Primary
public RestTemplate restTemplate() {
return new RestTemplate(new OkHttp3ClientHttpRequestFactory());
}
#Bean
public RestTemplate restTemplateLe(RestTemplateBuilder builder) {
List<HttpMessageConverter<?>> messageConverters = new ArrayList<>();
ExampleConverter exampleConverter = new ExampleConverter();
exampleConverter.setSupportedMediaTypes(Collections.singletonList(MediaType.TEXT_PLAIN));
messageConverters.add(exampleConverter);
return builder.messageConverters(messageConverters)
.requestFactory(new OkHttp3ClientHttpRequestFactory())
.build();
}
}

We Keep Coding

Java is a programming language and computing platform first released by Sun Microsystems in 1995.

Non-blocking FileWalkTree file search with Webflux - java

Related

How to make spring batch step execution parallel with configurable thread count?

Mock an instance method call of another class that takes a URL and File object and writes a file to a HTTP connection

Process file (which sit on Server A) on Server B

How to mock a void method which acquires the lock in Zookeepr and we can also verify the locks

Jackson xml map attribute value to property

Categories

Resources