Currently I'm dealing with a TarArchiveInputStream as this :
private Mono<Employee> createEmployeeFromArchiveFile() {
return Mono.fromCallable(() -> {
return new Employee();
})
.flatMap(employee -> {
try {
TarArchiveInputStream tar =
new TarArchiveInputStream(new GzipCompressorInputStream(new FileInputStream(new File("/tmp/myarchive.tar.gz"))));
TarArchiveEntry entry;
tar.read();
while ((entry = tar.getNextTarEntry()) != null) {
if (entry.getName().equals("data1.txt")) {
// process data
String data1 = IOUtils.toString(tar, String.valueOf(StandardCharsets.UTF_8));
if (data1.contains("age")) {
employee.setAge(4);
} else {
return Mono.error(new Exception("Missing age"));
}
}
if (entry.getName().equals("data2.txt")) {
// a lot more processing => put that in another function for clarity purpose
String data2 = IOUtils.toString(tar, String.valueOf(StandardCharsets.UTF_8));
employee = muchProcessing(employee, data2);
}
}
tar.close();
} catch (Exception e) {
return Mono.error(new Exception("Error while streaming archive"));
}
return Mono.just(employee);
});
}
private Employee muchProcessing(Employee employee, String data2) {
if (data2.contains("name")) {
employee.setName(4);
} else {
// return an error ?
}
return employee;
}
Firstly, is this a correct way to process the archive file with Reactor ? It works fine, but it seems like synchronous business inside a flatMap. I haven't found a better way.
Secondly, I don't know how to handle the function muchProcessing(tar). If that function triggers errors, how would it return them in order to be dealt appropriately as a Mono.error ? Since I want this function to return me an employee.
Thanks!
You can handle the task inside the flatMap as a CompletableFuture and convert it to a Mono. Here's a link on how to do that:
How to create a Mono from a completableFuture
Then, you can abstract it out as:
.flatMap(this::processEmployee).doOnError(this::logError).onErrorResume(getFallbackEmployee())
Related
I am developing a custom Spring cloud gateway predicate factory that will take decision based on some value in the request body.
I have written the following code.
public class OperatorIdPredicateFactory extends AbstractRoutePredicateFactory<OperatorIdPredicateFactory.Config> {
public OperatorIdPredicateFactory(Class<Config> configClass) {
super(configClass);
}
#Override
public Predicate<ServerWebExchange> apply(Config config) {
return (ServerWebExchange webExchange) -> {
return webExchange.getRequest().getBody().as((Flux<DataBuffer> body) ->{
var returnValue = new Object() {boolean value = false;}; // wrapper anonymous class for return value;
body.subscribe((DataBuffer buffer) ->{
CharBuffer charBuffer = StandardCharsets.UTF_8.decode(buffer.asByteBuffer());
DataBufferUtils.release(buffer);
String requestJson = charBuffer.toString();
ObjectMapper mapper = new ObjectMapper();
RouteRequestPayload requestPayload = null;
try {
requestPayload = mapper.readValue(requestJson, RouteRequestPayload.class);
} catch (JsonProcessingException e) {
e.printStackTrace();
}
if(config.getPtoId().equalsIgnoreCase(requestPayload.getRouteRequest().getOperatorId()))
returnValue.value = true;
});
return returnValue.value;
});
};
}
// config class
}
But the problem is that the method is returning before the subscribe method finishes.
How can I make sure that the subscribe method finished before returning the value.
List<CompletableFuture<GreetHolder>> completableFutures = langList.stream()
.map(lang -> getGreeting(lang))
.collect(Collectors.toList());
CompletableFuture<Void> allFutures = CompletableFuture.allOf(completableFutures.toArray(new CompletableFuture[completableFutures.size()]));
CompletableFuture<List<GreetHolder>> allCompletableFuture = allFutures.thenApply(future -> {
return completableFutures.stream().map(completableFuture -> completableFuture.join()).collect(Collectors.toList());
});
CompletableFuture completableFuture = allCompletableFuture.thenApply(greets -> {
return greets.stream().map(GreetHolder::getGreet).collect(Collectors.toList());
});
completableFuture.get();
private CompletableFuture<GreetHolder> getGreeting (String lang){
return CompletableFuture.supplyAsync(() -> {
try {
logger.info("Task execution started.");
Thread.sleep(2000);
logger.info("Task execution stopped.");
} catch (InterruptedException e) {
e.printStackTrace();
}
return new GreetHolder(getGreet(lang));
}, pool);
}
}
I want to read the List of greeting values returned from either from allCompletableFuture or completableFutureCompleted. I am not sure how to read the values returned by getGreeting method and output the list to the screen. Please feel free to modify the code if needed in the last step.
Your greetings are returned by completableFuture.get(); but you aren't capturing them in your example.
Assuming "getGreet" returns a string, does this work?
List<String> greetings = completableFuture.get();
I've created a Flowable (RxJava v3) that parses a file. I'd like it to support backpressure. This is important because the files can be quite large, and I don't want them to be loaded into memory at once. Here is my first attempt:
public Flowable<Byte> asFlowable(InputStream is) {
return Flowable.create(new FlowableOnSubscribe<Byte>() {
#Override
public void subscribe(FlowableEmitter<Byte> emitter) throws Exception {
try (DataInputStream inputStream = new DataInputStream(is)){
if (inputStream.readInt() != SOME_CONSTANT) {
throw new IllegalArgumentException("illegal file format");
}
if (inputStream.readInt() != SOME_OTHER_CONSTANT) {
throw new IllegalArgumentException("illegal file format");
}
final int numItems = inputStream.readInt();
for(int i = 0; i < numItems; i++) {
if(emitter.isCancelled()) {
return;
}
emitter.onNext(inputStream.readByte());
}
emitter.onComplete();
} catch (Exception e) {
emitter.onError(e);
}
}
}, BackpressureStrategy.BUFFER);
}
The reason I used Flowable.create instead of Flowable.generateis because I need to validate the file, and throw errors if some magic numbers at the beginning of the file are wrong or not found. This didn't fit well with the Flowable.generate lambdas (but if you know of a better way please post it).
Ok let's assume the cold Flowable supported backpressure. Now I'd like to process it in a console-like application.
Question:
I want to request a new Byte from the Flowable and print it to console each time the user presses space (similar to what more or less do in Linux). What would the best way of doing it? I intend to observe the flowable directly in the public static void main method, since I need to read and write using the console.
I've been reading the Backpressure section in RxJAva's Wiki and found this snippet:
someObservable.subscribe(new Subscriber<t>() {
#Override
public void onStart() {
request(1);
}
#Override
public void onCompleted() {
// gracefully handle sequence-complete
}
#Override
public void onError(Throwable e) {
// gracefully handle error
}
#Override
public void onNext(t n) {
// do something with the emitted item "n"
// request another item:
request(1);
}
});
But this confused me even more as the request method doesn't seem to exist in RxJava 3.
Use generate, blockingSubscribe and read a line from the console:
class State {
DataInputStream inputStream;
int count;
int i;
}
BufferedReader bin = new BufferedReader(new InputStreamReader(System.in));
Flowable.generate(() -> {
State s = new State();
s.inputStream = new DataInputStream(is);
try {
if (s.inputStream.readInt() != SOME_CONSTANT) {
throw new IllegalArgumentException("illegal file format");
}
if (s.inputStream.readInt() != SOME_OTHER_CONSTANT) {
throw new IllegalArgumentException("illegal file format");
}
s.count = s.inputStream.readInt();
} catch (IOException ex) {
s.inputStream.close();
throw ex;
}
return s;
}, (state, emitter) -> {
if (state.i < s.count) {
emitter.onNext(state.inputStream.readByte());
s.i++;
}
if (state.i >= s.count) {
emitter.onComplete();
}
}, state -> {
state.inputStream.close();
})
.subscribeOn(Schedulers.io())
.blockingSubscribe(b -> {
System.out.println(b);
bin.readLine();
}, Flowable.bufferSize());
I have been trying to implement a DAO method for delete operation for Azure Storage entities. Delete using TableOperation was ok.
TableOperation deleteEntity = TableOperation.delete(entity);
But when I tried it using Batch Operation, It was not supported.
Any suggestions to overcome this issue is highly appreciated.
But when I tried it using Batch Operation, It was not supported.
I assumed that you could group your items for deleting by partition key, then execute the TableBatchOperation.
Here I wrote a helper class via C# language for achieving this purpose, you could refer to it:
public class TableBatchHelper<T> where T : ITableEntity
{
const int batchMaxSize = 100;
public static IEnumerable<TableBatchOperation> GetBatchesForDelete(IEnumerable<T> items)
{
var list = new List<TableBatchOperation>();
var partitionGroups = items.GroupBy(arg => arg.PartitionKey).ToArray();
foreach (var group in partitionGroups)
{
T[] groupList = group.ToArray();
int offSet = batchMaxSize;
T[] entities = groupList.Take(offSet).ToArray();
while (entities.Any())
{
var tableBatchOperation = new TableBatchOperation();
foreach (var entity in entities)
{
tableBatchOperation.Add(TableOperation.Delete(entity));
}
list.Add(tableBatchOperation);
entities = groupList.Skip(offSet).Take(batchMaxSize).ToArray();
offSet += batchMaxSize;
}
}
return list;
}
public static async Task BatchDeleteAsync(CloudTable table, IEnumerable<T> items)
{
var batches = GetBatchesForDelete(items);
await Task.WhenAll(batches.Select(table.ExecuteBatchAsync));
}
}
Then, you could you execute the batch deleting as follows:
await TableBatchHelper<ClassName>.BatchDeleteAsync(cloudTable,items);
Or
var batches = TableBatchHelper<ClassName>.GetBatchesForDelete(entities);
Parallel.ForEach(batches, new ParallelOptions()
{
MaxDegreeOfParallelism = 5
}, (batchOperation) =>
{
try
{
table.ExecuteBatch(batchOperation);
Console.WriteLine("Writing {0} records", batchOperation.Count);
}
catch (Exception ex)
{
Console.WriteLine("ExecuteBatch throw a exception:" + ex.Message);
}
});
No, That was the code without using block operation. Following is the code that includes block operation. Sorry for not mentioning that
TableBatchOperation batchOperation = new TableBatchOperation();
List<TableBatchOperation> list = new ArrayList<>();
if (partitionQuery != null) {
for (AzureLocationData entity : cloudTable.execute(partitionQuery)) {
batchOperation.add(TableOperation.delete(entity));
list.add(batchOperation); //exception thrown line
}
try {
cloudTable.execute((TableOperation) batchOperation);
} catch (StorageException e) {
e.printStackTrace();
}
}
public void deleteLocationsForDevice(String id) {
logger.info("Going to delete location data for Device [{}]", id);
// Create a filter condition where the partition key is deviceId.
String partitionFilter = TableQuery.generateFilterCondition(
PARTITION_KEY,
TableQuery.QueryComparisons.EQUAL,
id);
// Specify a partition query, using partition key filter.
TableQuery<AzureLocationData> partitionQuery =
TableQuery.from(AzureLocationData.class)
.where(partitionFilter);
if (partitionQuery != null) {
for (AzureLocationData entity : cloudTable.execute(partitionQuery)) {
TableOperation deleteEntity = TableOperation.delete(entity);
try {
cloudTable.execute(deleteEntity);
logger.info("Successfully deleted location records with : " + entity.getPartitionKey());
} catch (StorageException e) {
e.printStackTrace();
}
}
} else {
logger.debug("No records to delete!");
}
// throw new UnsupportedOperationException("AzureIotLocationDataDao Delete Operation not supported");
}
I have a method implementation that calls a functional interface (Java 8) and I am trying to write a unit test for it. Here is the method I want to test:
// Impl
public Optional<ExternalRelease> getStagedRelease(AccessParams accessParams) throws DigitalIngestionException {
String resourcesPath=getReleaseResourcesPath(accessParams).orElseThrow(() -> new ValidationException(WRONG_ACCESS_CONTEXT_EXCEPTION));
String ddexFilePath=getReleaseDdexPath(accessParams).get();
if( stageDataManager.directoryExists(resourcesPath) ) {
List<Track> tracks = getTracks(accessParams).
orElse(new ArrayList<>());
ExternalRelease externalRelease = null;
//Verify if lite XML already exists
if( stageDataManager.fileExists(ddexFilePath) ) {
//Load externalRelease values
String liteDdex = stageDataManager.loadFileContent(ddexFilePath).
orElseThrow(() -> new ProcessException("Lite DDEX content couldn't be read."));
externalRelease = ddexManagerExecutor( (ddexManager) -> ddexManager.getExternalReleaseFromLiteDdex(liteDdex) ).
orElseThrow(() -> new ProcessException("External release couldn't be parsed from Lite DDEX."));
externalRelease.setTracks(tracks);
return Optional.ofNullable(externalRelease);
} else {
//Create lite ddex if it doesn't exist
ExternalRelease releaseFromTracks=getReleaseFromTracks(tracks);
String liteDdex = ddexManagerExecutor( (ddexManager) -> ddexManager.getLiteDdexFromAccessParamsAndExternalRelease(accessParams, releaseFromTracks)).
orElseThrow(() -> new ProcessException("Lite DDEX content couldn't be generated."));
stageDataManager.writeStringInFile(ddexFilePath, liteDdex);
return Optional.ofNullable(releaseFromTracks);
}
} else {
return Optional.empty();
}
}
Note the line within the second IF
externalRelease = ddexManagerExecutor( (ddexManager) -> ddexManager.getExternalReleaseFromLiteDdex(liteDdex) ).orElseThrow(() -> new ProcessException("External release couldn't be parsed from Lite DDEX."));`
The ddexManagerExecutor is a functional interface using java 8 features:
public interface ddexManagerConsumer<R> {
R process(DdexManager ddexManager) throws ProcessException;
}
private <R> R ddexManagerExecutor(ddexManagerConsumer<R> action) throws ProcessException {
DdexManager ddexManager = null;
try {
ddexManager = (DdexManager) ddexManagerPool.getTarget();
return action.process(ddexManager);
} catch (Exception e) {
throw new ProcessException("Error while accessing to ddexManager pool.");
} finally {
try {
ddexManagerPool.releaseTarget(ddexManager);
} catch (Exception e) {
log.error("Error while releasing ddexManager instance.", e);
}
}
}
So in my unit tests, which I am trying to do the following,
#Test
public void getStagedReleaseTest_whenFileDoesNoExist() throws Exception{
AccessParams accessParams = getDefaultAccessParams();
File file = new File("src/test/resources/testTrack.mp3");
String xml="<xml/>";
accessParams.setFileName("testTrack.mp3");
accessParams.setReleaseSlug("test-release-slug");
Optional<List<File>> mockFileList = Optional.of(Arrays.asList(file));
when(mockedStageDataManager.directoryExists(any())).thenReturn(true);
when(mockedStageDataManager.getResources(anyString(), any())).thenReturn(mockFileList);
when(mockedStageDataManager.fileExists(anyString())).thenReturn(false);
when(mockedStageDataManager.loadFileContent(anyString())).thenReturn(Optional.of(xml));
when(mockedDdexManager.getLiteDdexFromAccessParamsAndExternalRelease(any(), any())).thenReturn(Optional.of(xml));
DigitalIngestionServiceImpl serviceSpy = spy(service);
Optional<ExternalRelease> externalRelease = serviceSpy.getStagedRelease(accessParams);
Assert.assertNotNull(externalRelease);
}
This breaks my unit test since I am not mocking the call for the private functional interface. I am simply accounting for mocking the following inner line:
ddexManager.getExternalReleaseFromLiteDdex(liteDdex)
But none for the functional interface call that calls the above method. Any clue how to achieve that ?
You can mock ddexManagerPool.getTarget() in order to return a mocked DdexManager that will return what you want when calling getExternalReleaseFromLiteDdex