We have a requirements to add a header and footer to a output avro file, but seems like the SDK doesn't support it. For TextIO writer, it seems to have that capability withHeader and withFooter.
That been said, what's the best way to do it without creating a separate pipeline? I tried add another step after the writer, but seems like the pipeline assumed to be ended after the writer.
Finally found a solution by extending FileBasedSink.
public class CustomAvroSink extens FileBasedSink{
...
#Override
public WriteOperation<DestinationT, GenericRecord> createWriteOperation() {
return new CustomAvroWriteOperation(this, this.genericRecords, this.header, this.footer);
....
}
private static class CustomAvroWriteOperation<DestinationT, OutputT> extends
WriteOperation<DestinationT, OutputT> {
private final DynamicAvroDestinations<?, DestinationT, OutputT> dynamicDestinations;
private final boolean genericRecords;
private final OutputT header;
private final OutputT footer;
private CustomAvroWriteOperation(HeaderFooterAvroSink<?, DestinationT, OutputT> sink,
boolean genericRecords, OutputT header, OutputT footer) {
super(sink);
this.dynamicDestinations = sink.getDynamicDestinations();
this.genericRecords = genericRecords;
this.header = header;
this.footer = footer;
}
public Writer<DestinationT, OutputT> createWriter() throws Exception {
return new CustomAvroWriter<>(this, this.dynamicDestinations, this.genericRecords,
this.header, this.footer);
}
}
...
private static class CustomAvroWriter<DestinationT, OutputT> extends
Writer<DestinationT, OutputT> {
#Override
protected void writeHeader() throws Exception {
if (this.header != null) {
this.dataFileWriter.append(this.header);
}
}
#Override
protected void writeFooter() throws Exception {
if (this.footer != null) {
this.dataFileWriter.append(this.footer);
}
}
}
}
Then I can just do myPCollection.apply("header footer",WriteFiles.to(new CustomAvroSink(...header, footer...)))
Related
I'm working on a class that will get a list of strings and process them asynchronously using CompletableFutures. Each string is processed by invoking another class that will perform several operations and return a response or throw an exception if there is an error.
I would like to aggregate the responses that I get, whether they have a valid response or an exception and return them as a list to the caller. I would like the caller to be able to expect a list of SomeResponse and be able to interpret them using polymorphism.
However, I'm stuck on determining if this can be done using polymorphism at all, given that the fields for the success and error response are completely different. I have added some pseudo code below on one alternative I have thought of. Basically have SomeResponse be an interface with an isSuccess method. This will allow the caller to know if it's an error or not. However, the caller would still have to cast it to the correct implementation in order to get the value or the error. Is there a better way to approach this? My requirement is being able to return both a success and error response for each given request in the list. If there is an exception, we don't want to abort the entire operation.
public MyProcessorClass {
private final SomeOtherClass someOtherClass;
public List<SomeResponse> process(List<String> requestList) {
return requestList.stream().map(this::procesRequest)
.collectors(Collect.tolist()):
}
private processRequest(String request) {
CompletableFuture completableFuture = CompletableFuture
.supplyAsync(() => {
return new SomeSuccessResponse(someOtherClass.execute(request));
})
.exceptionally(e -> {
return new SomeErrorResponse(e.getCause);
});
return completableFuture.get();
}
}
public interface SomeResponse {
boolean isSuccess();
}
public class SomeSuccessResponse implements SomeResponse {
private final String value;
#Getter
private final boolean success;
public SomeSuccessResponse(String value) {
this.value = value;
this.success = true;
}
}
public class SomeErrorResponse implements SomeResponse {
private final Throwable error;
#Getter
private final boolean success;
public SomeErrorResponse(Throwable error) {
this.error = error;
this.success = false;
}
}
What you want is the visitor pattern https://en.wikipedia.org/wiki/Visitor_pattern
public class Main {
interface IResponse {
void acceptHandler(IResponseHandler handler);
}
static class ResponseA implements IResponse {
#Override
public void acceptHandler(IResponseHandler handler) {
handler.handle(this);
}
}
static class ResponseB implements IResponse {
#Override
public void acceptHandler(IResponseHandler handler) {
handler.handle(this);
}
}
public interface IResponseHandler {
void handle(ResponseA response);
void handle(ResponseB responseB);
}
public static void main(String[] args) {
final IResponseHandler handler = new IResponseHandler() {
#Override
public void handle(ResponseA response) {
System.out.println("Handle ResponseA");
}
#Override
public void handle(ResponseB responseB) {
System.out.println("Handle ResponseB");
}
};
final IResponse someResponse = new ResponseA();
someResponse.acceptHandler(handler);
}
}
I have pipe delimited data, which I want to transform that into xml. That transformation needs to be done based on the content of pipe delimited data. I'm trying to apply factory design pattern to design my model classes.It is throwing following errors.
what is the best way to solve the problem for the content based transformation.
Source.txt
0191155154|0000000001|0000001234|US|0000001101|2117565242|00029|00001|03000|
0191155154|0000000001|0000002342|US|0000001101|2117565242|00029|00001|03030|
PartIfd.java
#CsvRecord(separator = "\\|",skipField = true)
public class PartIfd {
#DataField(pos = 3)
private Integer WHSID;
#Link
private CntrlSeg cntrlSeg;
}
CntrlSeg.java
public class CntrlSeg {
#DataField(pos = 5)
private Integer index;
#Link
private PartSeg partSeg;
}
PartIfdFactory.java
public PartSeg getInstance(String string){
if(string.equals("03000")){
return new PartSeg3000();
}else
return new PartSeg3030();
}
PartSeg.java
public interface PartSeg {
public void recordNum();
}
PartSeg3000.java
public class PartSeg3000 implements PartSeg {
#DataField(pos = 9)
private Integer recordNum;
#Override
public void recordNum() {
System.out.println("3000");
}
}
PartSeg3030.java
public class PartSeg3030 implements PartSeg {
#DataField(pos = 7)
private Integer recordNum;
#Override
public void recordNum() {
System.out.println("3030");
}
}
ConverterRoute.java
public class ConverterRoute implements RoutesBuilder {
private static final String SOURCE_INPUT_PATH = "file://inbox?fileName=3000.txt";
private static final String SOURCE_OUTPUT_PATH = "file://outbox?fileName=file_$simple{date:now:yyyyMMddHHmmssSSS}.xml";
public void addRoutesToCamelContext(CamelContext context) throws Exception {
context.addRoutes(new RouteBuilder() {
public void configure() {
try {
DataFormat bindyFixed = new BindyCsvDataFormat(PartIfd.class);
XStreamDataFormat xStreamDataFormat = new XStreamDataFormat();
xStreamDataFormat.setAliases(Collections.singletonMap("PART_INB_IFD",PartIfd.class.getCanonicalName()));
from(SOURCE_INPUT_PATH).
split().tokenize(System.lineSeparator()).
unmarshal(bindyFixed).
marshal(xStreamDataFormat).
to(SOURCE_OUTPUT_PATH).log("Finished Transformation").end();
} catch (Exception e) {
e.printStackTrace();
}
}
});
}
}
Stack-trace
Caused by: java.lang.InstantiationException: com.test.domain.PartSeg
at java.lang.Class.newInstance(Class.java:427) ~[na:1.8.0_191]
at org.apache.camel.util.ObjectHelper.newInstance(ObjectHelper.java:1734) ~[camel-core-2.24.1.jar:2.24.1]
... 32 common frames omitted
Caused by: java.lang.NoSuchMethodException: com.test.domain.PartSeg.<init>()
I am using a Item with the inferface ResourceAware implemented to get the current resource filename. I get it in the ItemProcessor and this is working fine to log a complete information about the error.
Anyway, I want get the fileName if the moment of the FieldSetMapper throw an Exception, to register the line content, the line number and the file name but I don't found the correct way.
This is my current code:
public class MyCustomLineMapper<T> implements LineMapper<BaseDTO>, InitializingBean {
private LineTokenizer tokenizer;
private BeanWrapperFieldSetMapper<BaseDTO> fieldSetMapper;
public BaseDTO mapLine(String line, int lineNumber) throws Exception {
try {
BaseDTO r = fieldSetMapper.mapFieldSet(tokenizer.tokenize(line));
r.setLineNumber(lineNumber);
r.setLineContent(line);
return r;
} catch(Exception ex){
BaseDTO r = new BaseDTO();
r.setError(String.format(Keys.LINE_PARSE_ERROR, lineNumber, "FILE_NAME", line));
return r;
/// I want use the current filename in this moment.
}
}
public void setLineTokenizer(LineTokenizer tokenizer) {
this.tokenizer = tokenizer;
}
public void setFieldSetMapper(BeanWrapperFieldSetMapper<BaseDTO> fieldSetMapper) {
this.fieldSetMapper = fieldSetMapper;
}
public void afterPropertiesSet() {
Assert.notNull(tokenizer, "The LineTokenizer must be set");
Assert.notNull(fieldSetMapper, "The FieldSetMapper must be set");
}
}
In my item "BaseDTO" I implemented ResourceAware like this:
public class BaseDTO implements ResourceAware {
protected String itemClassifier;
protected String error;
protected int lineNumber;
protected String lineContent;
protected Resource res;
protected String fileName;
#Override
public void setResource(Resource res) {
this.res = res;
this.fileName = res.getFilename();
}
But in case of an exception reading a file (IncorrecNumberOfTokens by Example) I need get my item from FieldSetMapper with this field filled but the exception stop this. How can I achieve this?
Sorry for my english. Thanks in advance for all the help.
The context is:
I'm making a upload csv service and when i do some verification on the content i want to throw a custom runtime exception who contain the error message and the line to the problem.
What i got for now is:
public class NotValidCsvException extends RuntimeException {
private static final long serialVersionUID = 1L;
private Integer _line;
public NotValidCsvException(final String message, final Integer line) {
super(formatMessage(message, line));
}
public static String formatMessage(String message, Integer line) {
return new StringBuilder()
.append("{message: ")
.append("'")
.append(message)
.append("'")
.append(", ligne: ")
.append("'")
.append(line)
.append("'")
.append("}").toString();
}
}
But the problem is that i got a response like:
{"timestamp":1511785651810,"status":500,"error":"Internal Server Error","exception":"com.sstrn.pa.service.impl.exception.NotValidCsvException","message":"{"message": "message_import_csv_undefined_thing", "line": 0}"}
But i'd like to have something like:
{"timestamp":1511785651810,"status":500,"error":"Internal Server Error","exception":"com.sstrn.pa.service.impl.exception.NotValidCsvException","message":"message_import_csv_undefined_thing", "line": 0}
So to do this i created and interface AttributeLineEnabled whose implemented by my NotValidCsvException.
public interface AttributeLineEnabled {
public Integer getLine();
}
public class NotValidCsvException extends RuntimeException implements AttributeLineEnabled {
private static final long serialVersionUID = 1L;
private Integer _line;
public NotValidCsvException(final String message) {
super(message);
}
public NotValidCsvException(final String message, Integer line) {
super(message);
_line = line;
}
public Integer getLine() {
return _line;
}
public void setLine(final Integer line) {
_line = line;
}
}
Then i created a controller who extends DefaultErrorAttributes and implement ErrorAttributes.
This controller override the getErrorAttributes method who set the jsonResponse.
public class BaseErrorAttributes extends DefaultErrorAttributes implements ErrorAttributes {
private static final String LINE = "line";
public BaseErrorAttributes() {
super();
}
public Map<String, Object> getErrorAttributes(RequestAttributes requestAttributes, boolean includeStackTrace) {
Map<String, Object> errorAttributes = super.getErrorAttributes(requestAttributes, includeStackTrace);
Throwable error = getError(requestAttributes);
if (error instanceof AttributeLineEnabled) {
AttributeLineEnabled attributeLineEnabled= (AttributeLineEnabled) error;
errorAttributes.put(LINE, attributeLineEnabled.getLine());
}
return errorAttributes;
}
}
And i inject it into my controller who make the importation process:
private BaseErrorAttributes _baseErrorAttributes;
// getter and setter too
Main.java
package com.example.decorator;
public class Main {
public static void main(String[] args) {
Response response = new Response();
View head = new View("<title>Hello, world!</title>");
View body = new View("<h1>Hello, world!</h1>");
response.setContent(new HtmlLayout(head, body));
response.render();
}
}
Response.java
package com.example.decorator;
public class Response {
private Response content;
public Response () {}
public <T extends Response> void setContent(T content) {
this.content = content;
}
public void render() {
this.content.render();
};
}
View.java
package com.example.decorator;
public class View extends Response {
private String content;
public View(String content) {
this.content = content;
}
public void setContent(String content) {
this.content = content;
}
public String getContent() {
return this.content;
}
public void render() {
System.out.println(this.content);
}
}
Layout.java
package com.example.decorator;
public class Layout extends Response {
private Response view;
public <T extends Response> Layout(T view) {
this.view = view;
}
public void render() {
this.view.render();
}
}
HtmlLayout.java
package com.example.decorator;
public class HtmlLayout extends Response {
private Response head;
private Response body;
public <T extends Response> HtmlLayout(T head, T body) {
this.head = head;
this.body = body;
}
public void render() {
System.out.println("<!doctype html>");
System.out.println("<html>");
System.out.println("<head>");
this.head.render();
System.out.println("</head>");
System.out.println("<body>");
this.body.render();
System.out.println("</body>");
System.out.println("</html>");
}
}
Decorator pattern is used when you want an object of type(interface) A to do more than it does currently. An example would be: Web page(logical screen) that does fit your physical screen does not need scroll bar. however, if the page(logical screen) does not fit the physical screen, you have to decorate it with scroll bar.
In GOF words: The intent of Decorator is to attach additional responsibilities to an object dynamically.
In code that would look like:
interface LogicalScreen {
void render(String physical );
}
An implementation:
class SimpleScreen implements LogicalScreen {
public void render(String physical) {
// render itself
}
}
Implementation of decorator:
class ScreenWithScrollbar implements LogicalScreen {
private final LogicalScreen decoratd;
public ScreenWithScrollbar(LogicalScreen decorated) {
this.decoratd = decorated;
}
public void render(String physical) {
// render scroll bar
// ...
// render the decorated
decoratd.render(physical);
// eventually do some more stuff
}
public doScroll() {}
}
How is wired:
public class WhatIsDecorator {
public static void main(String[] args) {
LogicalScreen l1 = new SimpleScreen();
LogicalScreen ds = new ScreenWithScrollbar(l1);
ds.render("MyMonitor");
}
}
You can chain like this as many as you need. Decorator2(Decorator1(Simple)) ...
As far as I've looked over for short:
Change your Response to an interface
Your View.java does not match a decorator pattern really, so remove the extends of Response e.g.
Best regards