We Keep Coding

Apache kafka connect - custom websocket connector

The goal is :develop a custom Kafka connector that read ,messages from the websocket in a loop method. I try to give you an example on what I've realized:
I create an interface IWebsocketClientEndpoint
public interface IWebsocketClientEndpoint {
IWebsocketClientEndpoint Connect() ;
void Disconnect() throws IOException;
IWebsocketClientEndpoint addMessageHandler(IMessageHandler msgHandler);
void SendMessage(String message) throws Exception;
void SendMessage(ByteBuffer message) throws Exception;
void SendMessage(Object message) throws Exception;
boolean isOpen();
void Dispose()throws IOException;
}
and a class that implement above interface:
#ClientEndpoint
public class WebsocketClientEndpoint implements IWebsocketClientEndpoint {
private WebSocketContainer _container;
private Session _userSession = null;
private IMessageHandler _messageHandler;
private URI _endpointURI;
private WebsocketClientEndpoint(URI endpointURI) {
try {
_endpointURI = endpointURI;
_container = ContainerProvider.getWebSocketContainer();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private WebsocketClientEndpoint(URI endpointURI, int bufferSize) {
try {
_endpointURI = endpointURI;
_container = ContainerProvider.getWebSocketContainer();
_container.setDefaultMaxBinaryMessageBufferSize(bufferSize);
_container.setDefaultMaxTextMessageBufferSize(bufferSize);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
public static IWebsocketClientEndpoint Create(URI endpointURI){
return new WebsocketClientEndpoint(endpointURI);
}
public static IWebsocketClientEndpoint Create(URI endpointURI,int bufferSize){
return new WebsocketClientEndpoint(endpointURI,bufferSize);
}
public IWebsocketClientEndpoint Connect() {
try {
_container.connectToServer(this, _endpointURI);
return this;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
#OnOpen
public void onOpen(Session userSession) {
this._userSession = userSession;
if (this._messageHandler != null) {
this._messageHandler.handleOpen("Web socket "+ _endpointURI +" opened");}
}
#OnClose
public void onClose(Session userSession, CloseReason reason) {
this._userSession = null;
if (this._messageHandler != null) {
this._messageHandler.handleClose("Web socket "+ _endpointURI +" closed. Reason: " + reason.getReasonPhrase());}
}
public void Disconnect() throws IOException {
CloseReason reason = new CloseReason(CloseReason.CloseCodes.NORMAL_CLOSURE,"Web socket closed by user");
this._userSession.close(reason);
this._userSession = null;
//close notification to caller
if (this._messageHandler != null) {
this._messageHandler.handleClose("Web socket "+ _endpointURI +" closed. Reason: " + reason.getReasonPhrase());}
}
#Override
public IWebsocketClientEndpoint addMessageHandler(IMessageHandler msgHandler) {
this._messageHandler = msgHandler;
return this;
}
#OnMessage
public void onMessage(String message) {
if (this._messageHandler != null) {
this._messageHandler.handleMessage(message);
}
}
#OnMessage
public void onMessage(ByteBuffer bytes) {
if (this._messageHandler != null) {
this._messageHandler.handleMessage(bytes);
}
}
public void SendMessage(String message) throws Exception {
try{
this._userSession.getAsyncRemote().sendText(message);
}catch (Exception ex){
throw ex;
}
}
public void SendMessage(ByteBuffer message) throws Exception {
try{
this._userSession.getAsyncRemote().sendBinary(message);
}catch (Exception ex){
throw ex;
}
}
public void SendMessage(Object message) throws Exception {
this._userSession.getAsyncRemote().sendObject(message);
}catch (Exception ex){
throw ex;
}
}
#Override
public boolean isOpen() {
if (this._userSession != null){
return this._userSession.isOpen();
}
return false;
}
}
The class WebsocketClientEndpoint is dedicated to the creation of websocket and manage of connection, disconnection, send and receive message.
The goal is: how can I adapt the my websocket structure in the kafka connect structure? I could queue the message received ("public void handleMessage(String s)) from the socket in a ConcurrentLinkedQueue, and then, in the kafka connect loop method, unqueue them. But is it the best solution?
Below, the implementation of my Kafka custom connector
My kafka Connector
public class MySourceTask extends SourceTask {
IWebsocketClientEndpoint _clientEndPoint;
#Override
public void start(Map<String, String> props) {
_clientEndPoint = WebsocketClientEndpoint
.Create(new URI(socket))
.Connect();
_clientEndPoint.addMessageHandler(new IMessageHandler() {
#Override
public void handleMessage(String s) {
}
#Override
public void handleMessage(ByteBuffer byteBuffer) {
}
#Override
public void handleClose(String s) {
}
#Override
public void handleOpen(String s) {
}
});
}
#Override
public List<SourceRecord> poll() throws InterruptedException {
return null;
}
#Override
public void stop() {
_clientEndPoint.Dispose();
}
}
Thanks in advance to anyone

I'd suggest adding the interface to the class
extends SourceTask implements IMessageHandler
Then
_clientEndPoint.addMessageHandler(this);
And when you implement handleMessage, add those strings to some queue. Inside the poll method, you would pop data off that queue to create SourceRecord objects to return.
Inside of stop, call this.handleClose and clean up other resources.

Using OkHttp3 WebSocket with Retrofit to continuously track an android devices location

I have the following java code that I'd like to use in an android app to query an api for continuous lat/lng changes of a device that is running a client app, I want to track the device. I believe the WebSocketCall method I'm attempting to use is deprecated. From what I can tell, there's a problem with how I'm trying to use the webSocket call to create the retrofit client and enqueue the data from the WebSocketListner into retrofit. I've researched several WebSocketListener examples and being a total n00b, I haven't been able to figure out the code. My idea is to keep the connection open to the api via WebSocket and process the data response using retrofit. Any assistance would be greatly appreciated.
private WebSocketCall webSocket;
private void createWebSocket() {
final MainApplication application = (MainApplication) getActivity().getApplication();
application.getServiceAsync(new MainApplication.GetServiceCallback() {
#Override
public void onServiceReady(final OkHttpClient client, final Retrofit retrofit, WebService service) {
User user = application.getUser();
map.moveCamera(CameraUpdateFactory.newLatLngZoom(
new LatLng(user.getLatitude(), user.getLongitude()), user.getZoom()));
service.getDevices().enqueue(new WebServiceCallback<List<Device>>(getContext()) {
#Override
public void onSuccess(retrofit2.Response<List<Device>> response) {
for (Device device : response.body()) {
if (device != null) {
devices.put(device.getId(), device);
}
}
Request request = new Request.Builder().url(retrofit.baseUrl().url().toString() + "api/socket").build();
webSocket = WebSocketCall.create(client, request);
webSocket.enqueue(new WebSocketListener() {
#Override
public void onOpen(WebSocket webSocket, Response response) {
}
#Override
public void onFailure(IOException e, Response response) {
reconnectWebSocket();
}
#Override
public void onMessage(ResponseBody message) throws IOException {
final String data = message.string();
handler.post(new Runnable() {
#Override
public void run() {
try {
handleMessage(data);
} catch (IOException e) {
Log.w(MainFragment.class.getSimpleName(), e);
}
}
});
}
#Override
public void onClose(int code, String reason) {
reconnectWebSocket();
}
});
}
});
}
#Override
public boolean onFailure() {
return false;
}
});
}

So because I'm a total n00b it took some time and a lot of questions to figure this out. Maybe it'll help someone else in the future.
private WebSocket webSocket;
private void createWebSocket() {
final MainApplication application = (MainApplication) getActivity().getApplication();
application.getServiceAsync(new MainApplication.GetServiceCallback() {
#Override
public void onServiceReady(final OkHttpClient client, final Retrofit retrofit, WebService service) {
User user = application.getUser();
map.moveCamera(CameraUpdateFactory.newLatLngZoom(
new LatLng(user.getLatitude(), user.getLongitude()), user.getZoom()));
service.getDevices().enqueue(new WebServiceCallback<List<Device>>(getContext()) {
#Override
public void onSuccess(retrofit2.Response<List<Device>> response) {
for (Device device : response.body()) {
if (device != null) {
devices.put(device.getId(), device);
}
}
Request request = new Request.Builder().url(retrofit.baseUrl().url().toString() + "api/socket").build();
Log.e("WebSockets", "Headers: " + request.headers().toString());
WebSocketListener webSocketListener = new WebSocketListener() {
private static final int NORMAL_CLOSURE_STATUS = 1000;
#Override
public void onOpen(WebSocket webSocket, Response response) {
webSocket.send("{Auth-Token:secret-api-token-here}");
Log.e("WebSockets", "Connection accepted!");
}
#Override
public void onFailure(#NotNull WebSocket webSocket, #NotNull Throwable t, #Nullable Response response) {
reconnectWebSocket();
}
#Override
public void onMessage(#NotNull WebSocket webSocket, #NotNull String text) {
final String data = text;
Log.e("WebSockets", "Receiving : " + text);
handler.post(new Runnable() {
#Override
public void run() {
try {
handleMessage(data);
} catch (IOException e) {
Log.w(MainFragment.class.getSimpleName(), e);
}
}
});
}
#Override
public void onMessage(WebSocket webSocket, ByteString bytes) {
Log.e("WebSockets", "Receiving bytes : " + bytes.hex());
}
#Override
public void onClosing(WebSocket webSocket, int code, String reason) {
webSocket.close(NORMAL_CLOSURE_STATUS, null);
Log.e("WebSockets", "Closing : " + code + " / " + reason);
}
#Override
public void onClosed(#NotNull WebSocket webSocket, int code, #NotNull String reason) {
reconnectWebSocket();
}
};
webSocket = client.newWebSocket(request, webSocketListener);
}
});
}
#Override
public boolean onFailure() {
return false;
}
});
}

i'm confused with the "read" method of the ChannelOutboundHandle in netty framework

I'v got no idea about the "read" method of the ChannelOutboundHandle.
why this method occurred in the ChannelOutboundHandle, what's its for?
cause the outboundHandlers are used to handle the output, they are usually
used to write data to wire.
and also I got a problem when I'm using the "read" method.I wrote a EchoServer by netty. here is my code:
public class EchoServer {
public static void main(String[] args) {
// this handler is sharable
final EchoServerInboundHandler echoServerInboundHandler = new EchoServerInboundHandler();
EventLoopGroup eventLoopGroup = new NioEventLoopGroup();
ServerBootstrap serverBootstrap = new ServerBootstrap();
// setup group
serverBootstrap.group(eventLoopGroup)
// setup channelFactory
.channel(NioServerSocketChannel.class)
// listening port
.localAddress(new InetSocketAddress(8080))
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
protected void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(echoServerInboundHandler);
pipeline.addLast(new EchoServerOutboundHandler());
}
});
try {
ChannelFuture f = serverBootstrap.bind().sync();
f.addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) throws Exception {
if (future.isSuccess()) {
System.out.println(future.channel().localAddress()+" started!");
}
}
});
f.channel().closeFuture().sync();
} catch (InterruptedException e) {
e.printStackTrace();
}finally {
// release all threads
eventLoopGroup.shutdownGracefully().syncUninterruptibly();
}
}
}
public class EchoServerOutboundHandler extends ChannelOutboundHandlerAdapter {
#Override
public void read(ChannelHandlerContext ctx) throws Exception {
System.out.println("read");
//super.read(ctx);
}
#Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
System.out.println("msg:"+msg);
super.write(ctx, msg, promise);
}
#Override
public void flush(ChannelHandlerContext ctx) throws Exception {
super.flush(ctx);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
cause.printStackTrace();
ctx.close();
}
}
#ChannelHandler.Sharable
public class EchoServerInboundHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) {
ctx.writeAndFlush(msg);
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
cause.printStackTrace();
ctx.close();
}
#Override
public void channelActive(ChannelHandlerContext ctx) {
System.out.println("client:"+ctx.channel().remoteAddress()+" connected!");
}
#Override
public void channelInactive(ChannelHandlerContext ctx) {
System.out.println("client:"+ctx.channel().remoteAddress()+" disconnected!");
}
#Override
public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
System.out.println("EchoServerInboundHandler registered");
super.channelRegistered(ctx);
}
#Override
public void channelUnregistered(ChannelHandlerContext ctx) throws Exception {
System.out.println("EchoServerInboundHandler unregistered");
super.channelUnregistered(ctx);
}
}
as U can see, I comment the super.read(ctx); in the read method of the EchoServerOutboundHandler which cause a problem that it can't write data to the client. and also I found that this method will be called when a client established a connection.

At the very least, you need to provide the same code that the parent object would do in the super() method, which is ctx.read()
Which is what the JavaDoc says...
Calls ChannelHandlerContext.read() ...
Otherwise, all you're doing is printing something on your own, and not using Netty to handle anything.

Callback Duplicate Code

I'm using okhttp3 with asynchronous callbacks to get JSONArrays/JSONObjects from the server and then parsing them and creating the particular object which is passed to the callback function.
Most of the callback code is the same for every method, but there are some lines of code that differ.
Is there a pattern that I can use to reduce the lines of code so that I don't have to write the same code over and over again for the different objects?
I marked the lines of code that differ for every method.
The problem I have is calling the particular JSON parsing function without using switch/case and varying the callback object.
//-----------------------differs-------------------------
public void getUser(final HTTPResponseCallback<User> callback)
{
//-----------------------differs-------------------------
final String url = domain + USERS;
//-------------------------------------------------------
okHttpClient.newCall(buildRequest(url)).enqueue(new Callback()
{
Handler handler = new Handler(Looper.getMainLooper());
#Override
public void onFailure(Call call, IOException e)
{
handler.post(new Runnable()
{
#Override
public void run()
{
callback.onFailure();
}
});
}
#Override
public void onResponse(Call call, final Response response) throws IOException
{
if (response.isSuccessful())
{
try
{
String responseBody = response.body().string();
//-----------------------differs-------------------------
JSONObject jsonResponse = new JSONObject(responseBody);
final User user = JsonParser.parseUser(jsonResponse
//------------------------------------------------------
handler.post(new Runnable()
{
#Override
public void run()
{
//---------------------------------------last parameter differs----------------------------------------------
callback.onSuccess(new HTTPTransaction(response.code(), response.message(), response.header("ETag")), user);
//-----------------------------------------------------------------------------------------------------------
}
});
}
catch (JSONException e)
{
...
}
}
else
...
}
}
}

1) Make in parameterized with <T> as type can differs:
public class CallBackWrapper<T> {
...
public void getUser(final HTTPResponseCallback<T> callback) { ...
2) Introduce callback object for unique parts which will return instance of type T:
interface Worker {
T run(String responseBody);
}
public <T> void getUser(final HTTPResponseCallback<T> callback, Worker worker) { ...
3) Invoke needed worker:
String responseBody = response.body().string();
//-----------------------differs-------------------------
final T obj = worker.run(responseBody);
//------------------------------------------------------
handler.post(new Runnable()
{
#Override
public void run()
{
//---------------------------------------last parameter differs----------------------------------------------
callback.onSuccess(new HTTPTransaction(response.code(), response.message(), response.header("ETag")), obj);
//-----------------------------------------------------------------------------------------------------------
}
});

Spring-data-solr data import handler call

I'm using spring-data-solr and have data import handler(DIH) in solr. How can I call DIH by repository or SolrTemplate, or some else?

I'd recommend a Custom Respository using SolrCallback to execute desired request.
#Override
public SolrResponse dataImport(final String command) {
return solrTemplate.execute(new SolrCallback<SolrResponse>() {
#Override
public SolrResponse doInSolr(SolrServer solrServer) throws SolrServerException, IOException {
return new SolrRequest(METHOD.GET, "/dataimport?command=" + command) {
//..skipped some methods to shorten
#Override
public SolrResponse process(SolrServer server) throws SolrServerException, IOException {
SolrResponseBase response = new SolrResponseBase();
response.setResponse(server.request(this));
return response;
}
}.process(solrServer);
}
});
}

If someone else is struggling to get the DIH working in the current version (Solr 8.11) - this is what worked for me (be sure to adapt the core name):
solrTemplate.execute(new SolrCallback<SolrResponse>() {
#Override
public SolrResponse doInSolr(SolrClient solrClient) throws SolrServerException, IOException {
SolrRequest<SolrResponse> solrRequest = new SolrRequest<>(SolrRequest.METHOD.GET, "/<core_name>/dataimport?command=full-import&commit=true&clean=true") {
#Override
public SolrParams getParams() {
return null;
}
#Override
protected SolrResponse createResponse(SolrClient solrClient) {
SolrResponseBase response = new SolrResponseBase();
try {
response.setResponse(solrClient.request(this));
} catch (SolrServerException | IOException e) {
throw new RuntimeException(e);
}
return response;
}
};
solrRequest.setResponseParser(new DelegationTokenResponse.JsonMapResponseParser());
return solrRequest.process(solrClient);
}
});