We Keep Coding

Unable to create Kafka Redis Sink with Single Message Transformations

I am trying to create a Kafka Redis sink that deletes a particular key in Redis. One of the ways is to create a Record or Message in Kafka with a specific key and Value as null. But as per the use case, generating the keys is not possible. As a workaround, I wrote a Single message transformer that takes the message from Kafka, sets a particular Key, and sets Value equals null.
Here are my Kafka Connect Confgurations
"connector.class": "com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector",
"transforms.invalidaterediskeys.type": "com.github.cjmatta.kafka.connect.smt.InvalidateRedisKeys",
"redis.database": "0",
"redis.client.mode": "Standalone",
"topics": "test_redis_deletion2",
"tasks.max": "1",
"redis.hosts": "REDIS-HOST",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"transforms": "invalidaterediskeys"
}
Here is the code for the transformations :
public class InvalidateRedisKeys<R extends ConnectRecord<R>> implements Transformation<R> {
private static final Logger LOG = LoggerFactory.getLogger(InvalidateRedisKeys.class);
private static final ObjectMapper mapper = new ObjectMapper()
.configure(DeserializationConfig.Feature.FAIL_ON_UNKNOWN_PROPERTIES, false);
#Override
public ConfigDef config() {
return new ConfigDef();
}
#Override
public void configure(Map<String, ?> settings) {
}
#Override
public void close() {
}
#Override
public R apply(R r) {
try {
return r.newRecord(
r.topic(),
r.kafkaPartition(),
Schema.STRING_SCHEMA,
getKey(r.value()),
null,
null,
r.timestamp()
);
} catch (IOException e) {
LOG.error("a.jsonhandling.{}", e.getMessage());
return null;
} catch (Exception e) {
LOG.error("a.exception.{}", e.getMessage());
return null;
}
}
private String getKey(Object value) throws IOException {
A a = mapper.readValue(value.toString(), A.class);
long userId = a.getUser_id();
int roundId = a.getRound_id();
return KeyGeneratorUtil.getKey(userId, roundId);
}
}
where A is
public class A {
private long user_id;
private int round_id;
}
And KeyGeneratorUtil contains a static function that generates a relevant string and sends the results.
I took help from
https://github.com/cjmatta/kafka-connect-insert-uuid
https://github.com/apache/kafka/tree/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms
When I try to initialize Kafka Connect, it says invalid Configurations. Is there something that I am missing?

Unit testing, custom Call class for retrofit2 request: Reponse has private access

When I create custom Call class I can't return Response, because Response class is final. Is there any workaround for this?
public class TestCall implements Call<PlacesResults> {
String fileType;
String getPlacesJson = "getplaces.json";
String getPlacesUpdatedJson = "getplaces_updated.json";
public TestCall(String fileType) {
this.fileType = fileType;
}
#Override
public Response execute() throws IOException {
String responseString;
InputStream is;
if (fileType.equals(getPlacesJson)) {
is = InstrumentationRegistry.getContext().getAssets().open(getPlacesJson);
} else {
is = InstrumentationRegistry.getContext().getAssets().open(getPlacesUpdatedJson);
}
PlacesResults placesResults= new Gson().fromJson(new InputStreamReader(is), PlacesResults.class);
//CAN"T DO IT
return new Response<PlacesResults>(null, placesResults, null);
}
#Override
public void enqueue(Callback callback) {
}
//default methods here
//....
}
In my unit test class I want to use it like this:
Mockito.when(mockApi.getNearbyPlaces(eq("testkey"), Matchers.anyString(), Matchers.anyInt())).thenReturn(new TestCall("getplaces.json"));
GetPlacesAction action = new GetPlacesAction(getContext().getContentResolver(), mockEventBus, mockApi, "testkey");
action.downloadPlaces();
My downloadPlaces() method look like:
public void downloadPlaces() {
Call<PlacesResults> call = api.getNearbyPlaces(webApiKey, LocationLocator.getInstance().getLastLocation(), 500);
PlacesResults jsonResponse = null;
try {
Response<PlacesResults> response = call.execute();
Timber.d("response " + response);
jsonResponse = response.body();
if (jsonResponse == null) {
throw new IllegalStateException("Response is null");
}
} catch (UnknownHostException e) {
events.sendError(EventBus.ERROR_NO_CONNECTION);
} catch (Exception e) {
events.sendError(EventBus.ERROR_NO_PLACES);
return;
}
//TODO: some database operations
}

After looking at retrofit2 Response class more thoroughly I've found out that there is a static method that do what I need. So, I simply changed this line:
return new Response<PlacesResults>(null, placesResults, null);
to:
return Response.success(placesResults);
Everything works now.

Guice doesn't initialize property

I'm newly with Guice.
I want to use Guice for initializing object without writing new directly.
Here is my main():
public class VelocityParserTest {
public static void main(String[] args) throws IOException {
try {
PoenaRequestService poenaService = new PoenaRequestService();
System.out.println(poenaService.sendRequest("kbkCode"));
} catch (PoenaServiceException e) {
e.printStackTrace();
}
}
}
PoenaRequestService:
public class PoenaRequestService {
private static final String TEMPLATE_PATH = "resources/xml_messages/bp12/message01.xml";
public static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger.getLogger(PoenaRequestService.class);
#Inject
#Named("poena_service")
private HttpService poenaService;
public String sendRequest(/*TaxPayer taxPayer,*/ String kbk) throws PoenaServiceException {
LOG.info(String.format("Generating poena message request for string: %s", kbk));
Map<String, String> replaceValues = new HashMap<>();
replaceValues.put("guid", "guid");
replaceValues.put("iinbin", "iinbin");
replaceValues.put("rnn", "rnn");
replaceValues.put("taxOrgCode", "taxOrgCode");
replaceValues.put("kbk", "kbk");
replaceValues.put("dateMessage", "dateMessage");
replaceValues.put("applyDate", "applyDate");
ServiceResponseMessage result;
try {
String template = IOUtils.readFileIntoString(TEMPLATE_PATH);
Document rq = XmlUtil.parseDocument(StringUtils.replaceValues(template, replaceValues));
result = poenaService.execute(HttpMethod.POST, null, rq);
} catch (IOException e) {
throw new PoenaServiceException("Unable to read template file: " + TEMPLATE_PATH, e);
} catch (SAXException e) {
throw new PoenaServiceException("Unable to parse result document, please check template file: " + TEMPLATE_PATH, e);
} catch (HttpServiceException e) {
throw new PoenaServiceException(e);
}
if (result.isSuccess()) {
return (String) result.getResult();
}
throw new PoenaServiceException("HTTP service error code '" + result.getStatusCode() + "', message: " + result.getStatusMessage());
}
}
When I tried to debug this I see next picture:
As e result I got NullPointerException.
I couldn't figure out this behavior. Why does this exactly happen?
Any suggestions?

It's not working because you're not actually using Guice. You need to create an injector and bind your dependencies to something. Something akin to this:
public class VelocityParserTest {
public static void main(String[] args) throws IOException {
Injector injector = Guice.createInjector(new AbstractModule() {
#Override
protected void configure() {
bind(PoenaRequestService.class).asEagerSingleton();
bind(HttpService.class)
.annotatedWith(Names.named("poena_service"))
.toInstance(...);
}
});
try {
PoenaRequestService poenaService = injector.getInstance(PoenaRequestService.class);
System.out.println(poenaService.sendRequest("kbkCode"));
} catch (PoenaServiceException e) {
e.printStackTrace();
}
}
}

How to make Jackson throw exception as is when deserialization mapping fail

Jackson has a weird behavior in handling Exceptions that occur during deserialization mapping: it throws a JsonMappingException whose .getCause() returns the innermost of the exception chain.
//in main
ObjectMapper jsonMapper = new ObjectMapper();
String json = "{\"id\": 1}";
try {
Q q = jsonMapper.readValue(json, Q.class);
} catch (JsonMappingException e) {
System.out.println(e.getCause()); //java.lang.RuntimeException: ex 2
}
//class Q
public class Q {
#JsonCreator
public Q(#JsonProperty("id") int id) {
throw new RuntimeException("ex 0",
new RuntimeException("ex 1",
new RuntimeException("ex 2")));
}
}
In the code above, I use jsonMapper.readValue(..) to map the String json to an instance of class Q whose the constructor, marked #JsonCreator, throws a chain of RuntimeException: "ex 0", "ex 1", "ex 2". When the mapping fail, I expected the line System.out.println(e.getCause()); to print out ex 0, but it prints ex 2.
Why Jackson decides to do this and is there a way to configure it so that it doesn't discard my ex 0? I have tried
jsonMapper.configure(DeserializationFeature.WRAP_EXCEPTIONS, false);
but it doesn't do anything.

Inside of Jackson's StdValueInstantiator this method gets hit when an exception is thrown during deserialization:
protected JsonMappingException wrapException(Throwable t)
{
while (t.getCause() != null) {
t = t.getCause();
}
if (t instanceof JsonMappingException) {
return (JsonMappingException) t;
}
return new JsonMappingException("Instantiation of "+getValueTypeDesc()+" value failed: "+t.getMessage(), t);
}
As you can see, this will iterate through each "level" of your nested runtime exceptions and set the last one it hits as the cause for the JsonMappingException it returns.
Here is the code I needed to get this working:
Register a new module to the ObjectMapper.
#Test
public void testJackson() {
ObjectMapper jsonMapper = new ObjectMapper();
jsonMapper.registerModule(new MyModule(jsonMapper.getDeserializationConfig()));
String json = "{\"id\": \"1\"}";
try {
Q q = jsonMapper.readValue(json, Q.class);
System.out.println(q.getId());
} catch (JsonMappingException e) {
System.out.println(e.getCause()); //java.lang.RuntimeException: ex 2
} catch (JsonParseException e) {
} catch (IOException e) {
}
}
Create a custom module class.
public class MyModule extends SimpleModule {
public MyModule(DeserializationConfig deserializationConfig) {
super("MyModule", ModuleVersion.instance.version());
addValueInstantiator(Q.class, new MyValueInstantiator(deserializationConfig, Q.class));
addDeserializer(Q.class, new CustomDeserializer());
}
}
Create a custom ValueInstantiator class to override wrapException(...). Add the instantiator to the module.
public class MyValueInstantiator extends StdValueInstantiator {
public MyValueInstantiator(DeserializationConfig config, Class<?> valueType) {
super(config, valueType);
}
#Override
protected JsonMappingException wrapException(Throwable t) {
if (t instanceof JsonMappingException) {
return (JsonMappingException) t;
}
return new JsonMappingException("Instantiation of "+getValueTypeDesc()+" value failed: "+t.getMessage(), t);
}
}
Create a custom deserializer to get the module to work properly. Add this class to the module initialization as well.
public class CustomDeserializer extends StdScalarDeserializer<Q> {
public CustomDeserializer() {
super(Q.class);
}
#Override
public Q deserialize(JsonParser jp, DeserializationContext context) throws IOException {
JsonNode node = jp.getCodec().readTree(jp);
return new Q(node.get("id").asText());
}
#Override
public Object deserializeWithType(JsonParser jp, DeserializationContext ctxt, TypeDeserializer typeDeserializer) throws IOException {
return deserialize(jp, ctxt);
}
}

For anyone looking for a different solution, this worked for me on Spring Boot 2.2.8.RELEASE. NB: This is example is when you have a rest controller with request body that is has an enum and clients could send a wrong field string gender and you want to provide proper error message:
#PostMapping
public ResponseEntity<ProfileResponse> updateProfile(#RequestBody #Valid ProfileRequest profileRequest) {
ProfileResponse profile = //do something important here that returns profile object response
return ResponseEntity
.status(HttpStatus.OK)
.body(profile);
}
ProfileRequest looks like
#Data //Lombok getters and setters
public class ProfileRequest{
private GenderEnum gender;
//some more attributes
}
Add this property to the aplication.properties file to make sure that our custom exception GlobalRuntimeException (see later for definition) is not wrapped in JsonMappingException exception.
spring.jackson.deserialization.WRAP_EXCEPTIONS=false
Then create a class which spring boot will auto create a bean for (This will be used for Deserializing the field gender of type enum). If we don't find an the enum, then we know to throw an error.
#JsonComponent
public class GenderEnumDeserializer extends JsonDeserializer<GenderEnum> {
#Override
public GenderEnum deserialize(JsonParser p, DeserializationContext ctxt) throws IOException {
String val = p.getValueAsString();
GenderEnum genderEnum = GenderEnum.fromName(val);
if(genderEnum == null){
throw new GlobalRuntimeException("Invalid gender provided. Valid values are MALE | FEMALE | OTHER");
}
return genderEnum;
}
}
The "forName" method in GenderEnum looks like below.
public static GenderEnum fromName(String name) {
GenderEnum foundGenderEnum = null;
for (GenderEnum genderEnum : values()) {
if (genderEnum.name().equalsIgnoreCase(name)) {
foundGenderEnum = genderEnum;
}
}
return foundGenderEnum;
}
We would then setup catching the GlobalRuntimeException in our ControllerAdvice:
#ResponseBody
#ExceptionHandler(GlobalRuntimeException.class)
ResponseEntity<?> handleInvalidGlobalRuntimeException(HttpServletRequest request, GlobalRuntimeException ex) {
LOGGER.error("Error " + ex);
return ResponseEntity
.status(HttpStatus.BAD_REQUEST)
.body(new ErrorMessage(ex.getCustomMessage));
}
That's it.

How do you override Play framework controller in a unit testing context?

I am creating a basic POST JSON api endoint. I would like to unit test it, and want to make sure I am doing it appropriately in the Play framework. So far I am using Guice for dependency injection and JUnit for my unit testing library.
Here is my controller code:
public class NotificationController extends Controller {
private RabbitQueueService _rabbitQueueService;
#Inject
public NotificationController(RabbitQueueService service) {
_rabbitQueueService = service;
}
#BodyParser.Of(BodyParser.Json.class)
public Result post() {
ObjectMapper mapper = new ObjectMapper();
Notification notification;
try {
JsonNode notificationJsonNode = Controller.request().body().asJson();
notification = mapper.readValue(notificationJsonNode.toString(),
Notification.class);
_rabbitQueueService.push(notification);
return Results.created(notificationJsonNode, "UTF-8");
} catch (JsonParseException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return Results.badRequest();
}
}
My RabbitQueueService code:
public class RabbitQueueService {
private Channel _channel;
private Connection _connection;
public RabbitQueueService() {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost(Config.RABBITMQ_HOST);
try {
_connection = factory.newConnection();
_channel = _connection.createChannel();
_channel.queueDeclare(Config.RABBITMQ_QUEUE, false, false, false, null);
_channel.exchangeDeclare(Config.RABBITMQ_EXCHANGE, "fanout");
_channel.queueBind(Config.RABBITMQ_QUEUE, Config.RABBITMQ_EXCHANGE, "");
} catch (IOException e) {
e.printStackTrace();
}
}
public void push(Notification notification) {
try {
_channel.basicPublish(Config.RABBITMQ_EXCHANGE, "", null, notification.getBytes());
_channel.close();
_connection.close();
} catch (IOException e) {
e.printStackTrace();
}
}
public void pop() {
}
}
My MockQueueService code:
public class MockQueueService extends RabbitQueueService {
#Override
public void push(Notification notification) {
/* Do nothing because you know... thats what I do */
}
#Override
public void pop() {
/* Do nothing because you know... thats what I do */
}
}
and finally my current unit test code:
public class ApplicationTest {
#Test
public void addMessageToQueue() {
running(fakeApplication(), new Runnable() {
public void run() {
FakeRequest request = new FakeRequest("/POST", "/api/v1/notifications");
Notification notification = new Notification(UUID.randomUUID(),
new NotificationType(UUID.randomUUID(),
"Critical"),
"Test notification message");
try {
ObjectMapper mapper = new ObjectMapper();
String json = mapper.writeValueAsString(notification);
JsonNode node;
node = mapper.readTree(json);
request.withJsonBody(node);
} catch (IOException e) {
e.printStackTrace();
}
route(request);
}
});
}
}
This all works fine when making a curl request to test my endpoint through play run. My main question is: how do I use the MockQueueService in my unit test? I don't see anyway to do it with fakeApplication() helper. I could instantiate it directly like
NotificationController nc = new NotificationController(new MockQueueService());
nc.post();
but the problem is I need to override the body of the play request with an appropriate request body and I think I need a FakeRequest for that.
Any help, samples, or advice would be helpful.
UPDATE
I have posted a gist example with the necessary example files. The things specifically that I did to get it working:
Setup a new GlobalUnitTest file that I passed into the fakeApplication helper
Changed NotificationController to be a singleton. This allowed me to pull in the NotificationController instance so I could check the QueueService count as part of the assertion.

FakeApplication takes a bunch of arguments that you could use to inject your new service. You could use a combination of any of these:
additionalPlugins
additionalConfiguration
withGlobal
They each let you specify some additional configuration you could use only during testing. Another thing you could do is have a separate Global object just for testing, that is used to create your controllers. The Global object is used to return your controller instance when you use # in your route definition. Then, you can create a separate application.test.conf that refers to GlobalTest that is loaded when you run play test.