I need to connect to event hub with enabled kafka with Spring Boot, and I have connection string and name space where should I connect.
I'm using such dependencies
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>spring-cloud-azure-eventhubs-stream-binder</artifactId>
<version>1.2.7</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
I found a tutorial where I need to login into azure from my local machine with az login and create auth file, BUT I was provided with connection string which should I use, so is there any way to specify
ONLY connection string with namespace like this :
spring.cloud.azure.eventhub.connection-string
spring.cloud.azure.eventhub.namespace
Because now it is complaining that is is missing resource-group.
How should I connect to EventHub?
tl;dr
My question contains incorrect dependencies, I've added two binders, which incorrect. When you start app spring cloud stream don't know what is primary. So you need to choose only one.
So as I want to work with Event Hub, but had not previous experience with it, but had experience with Kafka and Event Hub has mode to work by Kafka protocol I started to look in that way. All tutorial from Microsoft are not working (sad for me). They are outdated.
So, I started to think if it is working by Kafka protocol, maybe I can thread Event Hub as simple Kafka with some configuration changes. After googling I found a lot of tutorial how to do it.
All you need is to create regular Kafka consumer/producer. I've done it with Spring Cloud Stream
#Slf4j
#EnableBinding(Sink.class)
public class KafkaSink {
#StreamListener(Sink.INPUT)
public void consumerMessage(TestMessage testMessage) {
log.info("{}", testMessage);
}
}
#Component
#EnableBinding(Source.class)
public class KafkaSource {
private final MessageChannel output;
#Autowired
public KafkaSource(MessageChannel output) {
this.output = output;
}
public void send(TestMessage testMessage) {
output.send(MessageBuilder.withPayload(testMessage).build());
}
}
And then just add proper jaas configuration into application.* file. You need to get connection string for your Event Hub
My yaml file:
spring:
cloud:
stream:
bindings:
input:
destination: my-topic
output:
destination: my-topic
kafka:
binder:
auto-create-topics: true
brokers: ${EVENT_HUB_KAFKA_BROKER}
configuration:
sasl:
jaas:
config: ${EVENT_HUB_CONNECTION_STRING}
mechanism: PLAIN
security:
protocol: SASL_SSL
One important thing EVENT_HUB_KAFKA_BROKER should be Event Hub address, something like blablabla.servicebus.windows.net:9093 (don't forget port). For EVENT_HUB_CONNECTION_STRING you hould specify module which will be parsing connection string as password and it should be something like org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="{your_connection_string}"\
Related
I'm practicing DGS framework's Subscription following the document, and I can trigger the subscription by using this Websocket support(Of course it can work)
implementation 'com.netflix.graphql.dgs:graphql-dgs-subscriptions-websockets-autoconfigure:latest.release'
But now I want to try configure the websocket path by myself and I'd like to use the RSocket to complete. I have no idea how to write the configuration (Wrote it in application.yml but maybe it should be written in a configuration Java class?).
I'd tried like this but the subscription didn't work at all.
application.yml
spring:
rsocket:
server:
transport: websocket
port: 7000
mapping-path: /subscriptions
And my DgsComponent
#DgsSubscription(field = "hello")
public Publisher<String> hello() {
return Flux.interval(Duration.ofSeconds(1))
.map(s -> "world" + s)
.take(10);
}
I'm new in RSocket and Netflix DGS, so if anyone could help me, thank you very much!!
Requirement is to set the topics attribute below at runtime without restarting the server.How can we achieve it here.
Currently we are reading the value from the properties file but here it require the server restart to reflect the changes done .
example:
sample.properties(inside the deployment directory)
topic.list=topic1,topic2
and would like to consume from topic3 in future without server restart.
NOTE : find that the topics is a final variable.
tried reading the key(topic.list) from the file system path (outside the deployment directory) but no luck.
Any suggestion.
<int-kafka:message-driven-channel-adapter
id="inAdapter"
channel="fromKafka"
connection-factory="connectionFactory"
key-decoder="kafkaKeyDecoder"
payload-decoder="kafkaDecoder"
topics="${topic.list}"
offset-manager="offsetManager"/>
You can use the Java DSL to dynamically add adapters for additional topics on demand...
#Autowired
private IntegrationFlowContext flowContext;
public void addAnotherListenerForTopics(String... topics) {
IntegrationFlow flow =
IntegrationFlows.from(Kafka.messageDrivenChannelAdapter(consumerFactory(), topics))
.channel("fromKafka")
.get();
this.flowContext.registration(flow).register();
}
and
bean.addAnotherListenerForTopics("added.new");
pom:
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-java-dsl</artifactId>
<version>1.2.1.RELEASE</version>
</dependency>
Note that if you are using broker partition assignment, the new container needs a different group id to avoid revoking the existing assignments.
In my DropWizard (v0.7.0) app, I have a DummyHealthCheck like so:
public class DummyHealthCheck extends HealthCheck {
#Override
protected Result check() throws Exception {
return Result.healthy();
}
}
Then in my main Application impl:
public class MyApplication extends Application<MyConfiguration> {
#Override
public void run(MyConfiguration configuration, Environment environment)
throws Exception {
environment.jersey().register(new DummyHealthCheck());
}
}
When I start up the server, it starts successfuly (no exceptions/errors), however I get the following message:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! THIS APPLICATION HAS NO HEALTHCHECKS. THIS MEANS YOU WILL NEVER KNOW !
! IF IT DIES IN PRODUCTION, WHICH MEANS YOU WILL NEVER KNOW IF YOU'RE !
! LETTING YOUR USERS DOWN. YOU SHOULD ADD A HEALTHCHECK FOR EACH OF YOUR !
! APPLICATION'S DEPENDENCIES WHICH FULLY (BUT LIGHTLY) TESTS IT. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
But when I go to http://localhost:8081/healthcheck I see:
{"deadlocks":{"healthy":true}}
What is going on here? How do I register my health check?
Also, I have configured DropWizard to use SSL (self-signed) on port 8443; I have verified this works with my normal endpoints. I am surprised, however, to see that my admin app is still exposed on 8081 over HTTP. How do I configure it for HTTPS as well?
Question 1:
You don't register it with Jersey, as Health Checks are DropWizard specific. They should be registered as follows
environment.healthChecks().register("dummy", new DummyHealthCheck());
as explained here. If it was registered as above, you would see
{"deadlocks":{"healthy":true}, "dummy":{"healthy":true}}
Question 2:
I assume you already have done something similar to
server:
applicationConnectors:
- type: https
port: 8443
keyStorePath: example.keystore
keyStorePassword: example
validateCerts: false
in your yaml, as seen here. That is just for the application. You will also need to configure the admin
server:
applicationConnectors:
- ...
adminConnectors:
- type: https
port: 8444 // should a different port from the application
keyStorePath: example.keystore
keyStorePassword: example
validateCerts: false
I have seen a couple of threads about this issue, but none of them seem to really answer the question directly.
Background, I have spring security installed, working, and running smoothly in other parts of the application. My username is "developer".
Running on Java 7, Glassfish 4, Spring 4, and using Angular + StompJS
Let's get some code here:
package com.myapp.config;
import org.springframework.context.annotation.Configuration;
import org.springframework.messaging.simp.config.MessageBrokerRegistry;
import org.springframework.web.socket.config.annotation.AbstractWebSocketMessageBrokerConfigurer;
import org.springframework.web.socket.config.annotation.EnableWebSocketMessageBroker;
import org.springframework.web.socket.config.annotation.StompEndpointRegistry;
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketBrokerConfig extends AbstractWebSocketMessageBrokerConfigurer {
public final static String userDestinationPrefix = "/user/";
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/stomp").withSockJS().setSessionCookieNeeded(true);
}
#Override
public void configureMessageBroker(MessageBrokerRegistry registry) {
registry.setApplicationDestinationPrefixes("/app");
//registry.enableStompBrokerRelay("/topic,/user");
registry.enableSimpleBroker("/topic", "/user");
registry.setUserDestinationPrefix(userDestinationPrefix);
}
}
Ok, now here is a controller, to send out stuff every 3 seconds:
import org.springframework.messaging.simp.SimpMessagingTemplate;
…
#Autowired
private SimpMessagingTemplate messagingTemplate;
…
#Scheduled(fixedDelay = 3000)
public void sendStuff ()
{
Map<String, Object> map = new HashMap<>();
map.put(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON);
System.out.print("Sending data! " + System.currentTimeMillis());
//messagingTemplate.convertAndSend("/topic/notify", "Public: " + System.currentTimeMillis());
messagingTemplate.convertAndSendToUser("developer", "/notify", "User: " + System.currentTimeMillis());
messagingTemplate.convertAndSendToUser("notYou", "/notify", "Mr Developer Should Not See This: " + System.currentTimeMillis());
}
And finally the JavaScript using SockJS
var client = new SockJS('/stomp');
var stomp = Stomp.over(client);
stomp.connect({}, function(s) {
//This should work
stomp.subscribe('/user/' + s.headers['user-name'] + '/notify', console.debug);
//This SHOULD NOT
stomp.subscribe('/user/notYou/notify', console.debug);
});
client.onclose = $scope.reconnect;
And finally, for kicks, the pom.xml
<dependency>
<groupId>javax.websocket</groupId>
<artifactId>javax.websocket-api</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-messaging</artifactId>
<version>4.0.6.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-websocket</artifactId>
<version>4.0.6.RELEASE</version>
</dependency>
Here is what does work:
I can produce wonderfully communication back and forth between the client and the server
It's fast
messagingTemplate.convertAndSend and messagingTemplate.convertAndSendToUser
This is the problem (noted above): Anyone can subscribe to other users feeds.
Now, there are a few other versions of this floating around, I will list them below, and explain why the answers are all wrong:
What are the security issues around an open websocket connection?
Spring websocket with stomp security - every user can subscribe to any other users queue?
Websocket: How To Push A Message To A Target User
Here's the problem:
Look at messagingTemplate.convertAndSendToUser - All that does is add the "user prefix" and then the username provided and then use messagingTemplate.convertAndSend which does not apply security.
Then people say that "you need to use spring security just like everywhere else" - the problem here is A) that I am SENDING data to the client asynchronously, so B) I will be using this code completely outside of the user's session, possibly from a different user (say to send a notification to another logged in user).
Let me know if this is too closely related to a different post, but I this is a big problem for me and I wanted to do this justice.
I can get more details though if anyone needs more details.
New Spring Security 4x now fully support Web Socket, you can refer the link Preview Spring Security WebSocket Support
Or SpringSecuritySupportWebSocket.html in case you need a complete example,
I think you must make these changes:
1) You must not enableSimpleBroker for "/user" because it's a special queue handled automatically by the broker
2) if the server uses for example the annotation "#SendToUser("/queue/private")" the client must subscribe to the queue "/user/queue/private" : you must not prepend the username in the queue because it's a transparent operation handled by the broker
I'm sure this works correctly because I'm using it in my setup.
I've not tried with the convertAndSendToUser() method but since its semantic should be the same of the annotation, it should work too.
You can override configureInbound method in a JavaConfig class extending AbstractSecurityWebSocketMessageBrokerConfigurer.
#Override
protected void configureInbound(MessageSecurityMetadataSourceRegistry messages) {
messages
.nullDestMatcher().authenticated() 1
.simpSubscribeDestMatchers("/user/queue/errors").permitAll() 2
.simpDestMatchers("/app/**").hasRole("USER") 3
.simpSubscribeDestMatchers("/user/**", "/topic/friends/*").hasRole("USER") 4
.simpTypeMatchers(MESSAGE, SUBSCRIBE).denyAll() 5
.anyMessage().denyAll(); 6
}
}
There, you can configure credentials to subscribe a channel, send messages or several other things, as mentioned in Spring WebSocket documntation https://docs.spring.io/spring-security/site/docs/current/reference/html/websocket.html#websocket-authorization
I have the below route. In unit test, since I doesn't have the FTP server available, I'd like to use camel's test support and send an invalid message to "ftp://hostname/input" and verify that it failed and routed to "ftp://hostname/error".
I gone through the documentation which mainly talks about using the "mock:" endpoint but I am not sure how to use it in this scenario.
public class MyRoute extends RouteBuilder
{
#Override
public void configure()
{
onException(EdiOrderParsingException.class).handled(true).to("ftp://hostname/error");
from("ftp://hostname/input")
.bean(new OrderEdiTocXml())
.convertBodyTo(String.class)
.convertBodyTo(Document.class)
.choice()
.when(xpath("/cXML/Response/Status/#text='OK'"))
.to("ftp://hostname/valid").otherwise()
.to("ftp://hostname/invalid");
}
}
As Ben says you can either setup a FTP server and use the real components. The FTP server can be embedded, or you can setup a FTP server in-house. The latter is more like an integration testing, where you may have a dedicated test environment.
Camel is very flexible in its test kit, and if you want to build an unit test that do not use the real FTP component, then you can replace that before the test. For example in your example you can replace the input endpoint of a route to a direct endpoint to make it easier to send a message to the route. Then you can use an interceptor to intercept the sending to the ftp endpoints, and detour the message.
The advice with part of the test kit offers these capabilities: http://camel.apache.org/advicewith.html. And is also discussed in chapter 6 of the Camel in action book, such as section 6.3, that talks about simulating errors.
In your example you could do something a like
public void testSendError() throws Exception {
// first advice the route to replace the input, and catch sending to FTP servers
context.getRouteDefinitions().get(0).adviceWith(context, new AdviceWithRouteBuilder() {
#Override
public void configure() throws Exception {
replaceFromWith("direct:input");
// intercept valid messages
interceptSendToEndpoint("ftp://hostname/valid")
.skipSendToOriginalEndpoint()
.to("mock:valid");
// intercept invalid messages
interceptSendToEndpoint("ftp://hostname/invalid")
.skipSendToOriginalEndpoint()
.to("mock:invalid");
}
});
// we must manually start when we are done with all the advice with
context.start();
// setup expectations on the mocks
getMockEndpoint("mock:invalid").expectedMessageCount(1);
getMockEndpoint("mock:valid").expectedMessageCount(0);
// send the invalid message to the route
template.sendBody("direct:input", "Some invalid content here");
// assert that the test was okay
assertMockEndpointsSatisfied();
}
From Camel 2.10 onwards we will make the intercept and mock a bit easier when using advice with. As well we are introducing a stub component. http://camel.apache.org/stub
Have a look at MockFtPServer!
<dependency>
<groupId>org.mockftpserver</groupId>
<artifactId>MockFtpServer</artifactId>
<version>2.2</version>
<scope>test</scope>
</dependency>
With this one you can simulate all sorts of behaviors like permission problems, etc:
Example:
fakeFtpServer = new FakeFtpServer();
fakeFtpServer.setServerControlPort(FTPPORT);
FileSystem fileSystem = new UnixFakeFileSystem();
fileSystem.add(new DirectoryEntry(FTPDIRECTORY));
fakeFtpServer.setFileSystem(fileSystem);
fakeFtpServer.addUserAccount(new UserAccount(USERNAME, PASSWORD, FTPDIRECTORY));
...
assertTrue("Expected file to be transferred", fakeFtpServer.getFileSystem().exists(FTPDIRECTORY + "/" + FILENAME));
take a look at this unit test and those in the same directory...they'll show you how to standup a local FTP server for testing and how to use CamelTestSupport to validate scenarios against it, etc...
example unit test...
https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FromFileToFtpTest.java
which extends this test support class...
https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpsServerTestSupport.java
In our project we do not create mock FTP Server to test the route but we use properties that can be replaced by a file Camel Component for the local development and unit testing.
Your code would look like this:
public class MyRoute extends RouteBuilder
{
#Override
public void configure()
{
onException(EdiOrderParsingException.class)
.handled(true)
.to("{{myroute.error}}");
from("{{myroute.input.endpoint}}")
.bean(new OrderEdiTocXml())
.convertBodyTo(String.class)
.convertBodyTo(Document.class)
.choice()
.when(xpath("/cXML/Response/Status/#text='OK'"))
.to("{{myroute.valid.endpoint}}}")
.otherwise()
.to("{{myroute.invalid.endpoint}}");
}
}
And locally and for system test we use a file endpoint declared in the property file:
myroute.input.endpoint=file:/home/user/myproject/input
myroute.valid.endpoint=file:/home/user/myproject/valid
myroute.invalid.endpoint=file:/home/user/myproject/invalid
myroute.error=file:/home/user/myproject/error
or in a JUnit CamelTestSupport you can use the useOverridePropertiesWithPropertiesComponent method to set the properties you want to overrides.
As an alternative you can also use a "direct" route instead but you can miss some File options that can be tested by the unit test.
And we only test the FTP connection with the real system by setting the properties like this:
myroute.input.endpoint=ftp://hostname/input
myroute.valid.endpoint=ftp://hostname/valid
myroute.invalid.endpoint=ftp://hostname/invalid
myroute.error=ftp://hostname/error
With this you can also have different configuration for e.g production server that will differentiate from the Integration Test Environment.
Example of Properties for Production environment:
myroute.input.endpoint=ftp://hostname-prod/input
myroute.valid.endpoint=ftp://hostname-prod/valid
myroute.invalid.endpoint=ftp://hostname-prod/invalid
myroute.error=ftp://hostname-prod/error
In my opinion it is totally acceptable to use file endpoint to simplify the JUnit code and it will test the route only and not the connection.
Testing the connection is more like an Integration Test and should be executed on the real server connected with the real external system (in your case FTP servers, but can be other endpoints/systems as well).
By using properties you can also configure different URL's per environment (For example: we have 3 testing environments and one production environment, all with different endpoints).