I have the following application that runs a data pipeline and listens to Google pubsub:
Config:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.0.0</version>
</dependency>
Code:
Pipeline p = //Initialise pipeline
p.apply("read messages", readMessage())
.apply("log message", logMessage())
private static PubsubIO.Read<String> readMessage() {
return PubsubIO.readStrings()
.fromSubscription("my_subscription");
}
//Log message just logs the message
I faced an issue where messages were getting dropped intermittently (i.e. they were not making it to dataflow pipeline from pubsub). In order to debug, I wrote another application and made it listen to the same pubsub. Below is the config:
Config:
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
<version>0.22.0-beta</version>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-pubsub</artifactId>
<version>v1-rev358-1.22.0</version>
</dependency>
Code:
#Component
public class PubsubMessageReceiver implements MessageReceiver{
private static Logger logger = LoggerFactory.getLogger(PubsubMessageReceiver.class);
#Override
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
logger.info(message.getData().toStringUtf8());
consumer.ack();
}
}
I made it run for a couple of hours (by creating a new subscription to listen to the same topic as the original application), checked the log files and found some messages that did not make it to the dataflow pipeline.
Now, this is intermittent and I am not able to reproduce it (even with load testing). Does this have anything to do with google's beam libraries (looks like a race condition or similar)? If yes, does anyone know if this can be fixed by changing the configuration/updating the version?
Related
I tried this one https://github.com/zalando/logbook but it only works in spring based application. Does someone knows how to log request and response in Vertx framework?
This is a bit of a broad question, but let me try to answer it in a couple of segments.
First, I assume you are looking for a logger library. Vertx provides something like that, but it is deprecated and they encourage you to use third-party libs like Log4J or SLF4J. To use it, you need to add it as a dependency to your pom.xml like this (assuming you use maven):
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.32</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.32</version>
</dependency>
After that, you can instantiate a logger like this:
final static Logger logger = LoggerFactory.getLogger("loggerName");
and use it like this:
logger.info("Logging this message!")
As to where to log HTTP request, you have to go where you handle your HTTP routes and define and register a handler. For response, that is something you send so you easily log (with logger) in the place in the code where you create it. This is how you would handle HTTP request logging:
final Handler<RoutingContext> loggingHandler = routingContext -> {
// here you access properties of routingContext.request() and log what you want
}
router.route().handler(loggingHandler);
As for response, somewhere in your code you create response like this: HttpServerResponse response = context.response().setStatusCode(status); and then send it with response.end(content). Before calling .end() you can log what you need by accessing properties of response.
Add the following handler to your router and log to your needs
private fun loggingHandler(routingContext: RoutingContext) {
routingContext.addHeadersEndHandler {
// log context.request() and context.response() as required
}
routingContext.next()
}
I need to connect to event hub with enabled kafka with Spring Boot, and I have connection string and name space where should I connect.
I'm using such dependencies
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>spring-cloud-azure-eventhubs-stream-binder</artifactId>
<version>1.2.7</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
I found a tutorial where I need to login into azure from my local machine with az login and create auth file, BUT I was provided with connection string which should I use, so is there any way to specify
ONLY connection string with namespace like this :
spring.cloud.azure.eventhub.connection-string
spring.cloud.azure.eventhub.namespace
Because now it is complaining that is is missing resource-group.
How should I connect to EventHub?
tl;dr
My question contains incorrect dependencies, I've added two binders, which incorrect. When you start app spring cloud stream don't know what is primary. So you need to choose only one.
So as I want to work with Event Hub, but had not previous experience with it, but had experience with Kafka and Event Hub has mode to work by Kafka protocol I started to look in that way. All tutorial from Microsoft are not working (sad for me). They are outdated.
So, I started to think if it is working by Kafka protocol, maybe I can thread Event Hub as simple Kafka with some configuration changes. After googling I found a lot of tutorial how to do it.
All you need is to create regular Kafka consumer/producer. I've done it with Spring Cloud Stream
#Slf4j
#EnableBinding(Sink.class)
public class KafkaSink {
#StreamListener(Sink.INPUT)
public void consumerMessage(TestMessage testMessage) {
log.info("{}", testMessage);
}
}
#Component
#EnableBinding(Source.class)
public class KafkaSource {
private final MessageChannel output;
#Autowired
public KafkaSource(MessageChannel output) {
this.output = output;
}
public void send(TestMessage testMessage) {
output.send(MessageBuilder.withPayload(testMessage).build());
}
}
And then just add proper jaas configuration into application.* file. You need to get connection string for your Event Hub
My yaml file:
spring:
cloud:
stream:
bindings:
input:
destination: my-topic
output:
destination: my-topic
kafka:
binder:
auto-create-topics: true
brokers: ${EVENT_HUB_KAFKA_BROKER}
configuration:
sasl:
jaas:
config: ${EVENT_HUB_CONNECTION_STRING}
mechanism: PLAIN
security:
protocol: SASL_SSL
One important thing EVENT_HUB_KAFKA_BROKER should be Event Hub address, something like blablabla.servicebus.windows.net:9093 (don't forget port). For EVENT_HUB_CONNECTION_STRING you hould specify module which will be parsing connection string as password and it should be something like org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{your_connection_string}"\
I'm trying to turn an image stored as a Blob in my DB into a MultipartFile to serve back to the client when it's requested. I retrieve the Blob as a byte[] and I'm trying to convert it into a MultipartFile to serve back to the client
I'm trying to do it this way : https://stackoverflow.com/a/25820543/7082628
But IntelliJ is telling me it can't find the MockMultipartFile part when I do the import of: import org.springframework.mock.web.MockMultipartFile
I can import this in a test class no problem, but not outside of the test class. Can I do it here?
Also, I tried to do this by implementing a class with my own version of MultipartFile as stated in another popular answer, but it's telling me it can't find a serialzer.
Any suggestions?
There are couple of issues with your approach.
Multipart file is an HTTP Request format and not intended for response.
For a response all you have to do is write the file to the response.getOutputStream(). Which becomes way easier using Spring
Now coming to your original question that you can't import MockMultipartFile. That's most likely because you're using Maven and the dependency (most likely spring-boot-starter-test) has scope set to test.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope><!-- CHANGE THIS TO "runtime" -->
</dependency>
But again as I said before, there's no need to do that. You don't have to consider the Multipart when you're providing a response, it's a protocol to upload files.
Here's how you can provide a download link to a Blob in Spring
#GetMapping(path = "download")
public ResponseEntity<Resource> download(String param) throws IOException {
InputStreamResource resource = new InputStreamResource(/* InputStream of blob */);
long fileLength = /*Length of content, bytes.length or something */
return ResponseEntity.ok()
.contentLength(fileLength)
.contentType(MediaType.APPLICATION_OCTET_STREAM_VALUE)
.body(resource);
}
Finally the package i found which helped with
import org.springframework.mock.web.MockMultipartFile;
<properties>
<java.version>1.8</java.version>
<spring.version>5.1.2.RELEASE</spring.version>
</properties>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>${spring.version}</version>
</dependency>
I have implemented the following Web-Socket-Server using spring. We don't want to use STOMP and JSocks.
#Configuration
#EnableWebSocket
public class WebSocketConfig implements WebSocketConfigurer {
public void registerWebSocketHandlers(WebSocketHandlerRegistry registry) {
registry.addHandler(new WebSocketServerHandler(), "/websocket/user/{userId}");
}
}
#Service
public class WebSocketServerHandler extends TextWebSocketHandler {
private List<WebSocketSession> sessions = new ArrayList<WebSocketSession>();
#Override
public void handleTextMessage(WebSocketSession session, TextMessage message)
throws Exception {
session.sendMessage(message);
}
#Override
public void afterConnectionEstablished(WebSocketSession session) throws Exception {
sessions.add(session);
}
#Override
public void afterConnectionClosed(WebSocketSession session, CloseStatus status) throws Exception {
sessions = null;
}
}
Now I can successfully connect to that Web-Socket-Server from front-end client (browser). Now I want to connect to that Web-Socket-Server from some of my java class at Run-Time and then want to send messages to that Web-Socket-Server. Anyone have any idea how I can I do that?
I have added a test Rest Controller as below:
#RequestMapping(value = "/web-socket/message/{message}")
public void sendMessage(#PathVariable("message") final String message) throws Exception {
final String WS_URI = "ws://localhost:8080/web-socket/user/23";
final CountDownLatch latch = new CountDownLatch(1);
WebSocketClientHandler handler = new WebSocketClientHandler(latch);
WebSocketClient client = new StandardWebSocketClient();
WebSocketSession session = client.doHandshake(handler, WS_URI).get();
session.sendMessage(new TextMessage(message));
latch.await(5, TimeUnit.SECONDS);
session.close();
}
I am successfully able to send message to the websocket server, only for the 1st time by calling this Test RestController using google rest client. If I need to send message again, then I have to restart the tomcat server. Is there anything wrong what I am doing here?
spring-websocket has WebSocket client support. The basic contract is the WebSocketClient. Spring doesn't implement the entire WebSocket support, but offers an abstraction layer over other implementations. The common Spring client abstraction is the StandardWebSocketClient. A simple example look something like
public static void main(String... args) throws Exception {
final CountDownLatch latch = new CountDownLatch(1);
EchoHandler handler = new EchoHandler(latch);
WebSocketClient client = new StandardWebSocketClient();
WebSocketSession session = client.doHandshake(handler, ECHO_URL).get();
session.sendMessage(new TextMessage("Hello World"));
latch.await(5000, TimeUnit.SECONDS);
session.close();
}
Where EchoHandler is a client side WebSocketHandler (same as server side).
public class EchoHandler extends TextWebSocketHandler {
private final CountDownLatch latch;
public EchoHandler(CountDownLatch latch) {
this.latch = latch;
}
#Override
public void handleTextMessage(WebSocketSession session, TextMessage message) {
System.out.println("------- received client message ------");
System.out.println(message.getPayload());
System.out.println("--------- end client message ---------");
latch.countDown();
}
}
You can also wrap the WebSocketClient in a WebSocketConnectionManager if you are running the client in a String environment. This will give you some lifecycle helpers if you need it. See example in Spring Boot sample.
As far the the dependencies, like I said, Spring doesn't implement the entire WebSocket client support, so you need an implementation. If you are running the client in a server that supports WebSocket, then you will not need to add anything. The support implementation should already be on the classpath from the server. The main supported WebSocket implementations are Jetty, Tomcat, Undertow (mainly Wildfly or standalone Undertow), Tyrus (i.e. Glassfish, WebLogic).
If you are running the client in a standalone app, then you will need to add a WebSocket implementation. Unfortunately, from what I tested, none of the implementation provide a complete workable "client-only" jar. They either require or already pull in the complete (server included) implementation. So using just the client, will still require pulling in a bunch of server jars. Here's what I came up with from testing
Common for all
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-websocket</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>javax.websocket</groupId>
<artifactId>javax.websocket-api</artifactId>
<version>${websocket.version}</version>
</dependency>
Tomcat
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>tomcat-websocket</artifactId>
<version>${tomcat.version}</version>
</dependency>
<dependency>
<groupId>org.apache.tomcat.embed</groupId>
<artifactId>tomcat-embed-core</artifactId>
<version>${tomcat.version}</version>
</dependency>
Undertow
<dependency>
<groupId>io.undertow</groupId>
<artifactId>undertow-websockets-jsr</artifactId>
<version>${undertow.version}</version>
</dependency>
Tyrus
<!-- tyrus-client is pulled in by this. -->
<dependency>
<groupId>org.glassfish.tyrus</groupId>
<artifactId>tyrus-server</artifactId>
<version>${tyrus.version}</version>
</dependency>
Jetty
<dependency>
<groupId>org.eclipse.jetty.websocket</groupId>
<artifactId>websocket-client</artifactId>
<version>${jetty.version}</version>
</dependency>
With Jetty, it does't use the standard Java WebSocket API, so it is not used in the StandardWebSocketClient. You will need to do instead
JettyWebSocketClient client = new JettyWebSocketClient();
client.start();
Everything else above is the same though.
I just looked at the Spring source to see all the different dependencies they used. You can check it out and play around with the dependencies. Maybe there are different more efficient (lighter) combination of dependencies that will still work. The above is just what I tested with and was able to get to work.
Using .net tooling, I have been able create an ODATA service endpoint that exposes a legacy MSSQL database. It was my intent to then use odata4j to communicate to it. However, lacking pojos, I turned to restlet. I made all entities visible in the service, but when I run the restlet generator it fails to generate pojos saying:
java.util.concurrent.TimeoutException
at org.restlet.ext.xml.SaxRepresentation.parse(SaxRepresentation.java:230)
at org.restlet.ext.odata.internal.edm.Metadata.<init>(Metadata.java:79)
at org.restlet.ext.odata.Service.getMetadata(Service.java:488)
at org.restlet.ext.odata.Generator.main(Generator.java:137)
...
Can't get the metadata for `http://localhost:53088/ODataService.svc/`
...
java.io.IOException: Couldn't parse the source representation: java.io.IOException: The thread blocked at the cyclic barrier has timed out.
at org.restlet.ext.xml.SaxRepresentation.parse(SaxRepresentation.java:238)
at org.restlet.ext.odata.internal.edm.Metadata.<init>(Metadata.java:79)
at org.restlet.ext.odata.Service.getMetadata(Service.java:488)
at org.restlet.ext.odata.Generator.main(Generator.java:137)
at xxx.model.generator.ModelGenerator.main(ModelGenerator.java:12)
I decided to trim down the ODATA service to a single simple entity and try the code generation and it worked! But as I enabled more entities, I received an XML parsing error:
Can't get the metadata for `http://localhost:53088/ODataService.svc/`
java.io.IOException: Couldn't parse the source representation:\
org.xml.sax.SAXParseException: XML document structures must start and end within \
the same entity.
at org.restlet.ext.xml.SaxRepresentation.parse(SaxRepresentation.java:238)
at org.restlet.ext.odata.internal.edm.Metadata.<init>(Metadata.java:79)
at org.restlet.ext.odata.Service.getMetadata(Service.java:488)
at org.restlet.ext.odata.Generator.main(Generator.java:137)
at xxx.model.generator.ModelGenerator.main(ModelGenerator.java:12)
An error occurred:
Cannot retrieve the metadata.
Anyways, it seems to dislike the xml after some number of entities, rather than disliking a specific entity. Also the XML from http://localhost:53088/ODataService.svc/$metadata is valid with no errors.
Here is the Generator code:
import org.restlet.ext.odata.Generator;
public class ModelGenerator
{
public static final String [] URL_WORKSPACE = { "http://localhost:53088/ODataService.svc/", "src/main/java/"};
public static void main(String[] args)
{
Generator.main(URL_WORKSPACE);
}
}
Here are my maven POM details:
<properties>
<org.odata4j.version>0.7.0</org.odata4j.version>
<org.restlet.version>2.1.4</org.restlet.version>
</properties>
<dependencies>
<dependency>
<groupId>org.eclipse.persistence</groupId>
<artifactId>eclipselink</artifactId>
<version>2.5.1</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.odata4j</groupId>
<artifactId>odata4j-core</artifactId>
<version>0.7.0</version>
</dependency>
<dependency>
<groupId>org.restlet.jse</groupId>
<artifactId>org.restlet</artifactId>
<version>${org.restlet.version}</version>
</dependency>
<dependency>
<groupId>org.restlet.jse</groupId>
<artifactId>org.restlet.ext.odata</artifactId>
<version>${org.restlet.version}</version>
</dependency>
</dependencies>
Any suggestions would be appreciated. Thank you!