i am using spring cloud stream rabbit binder with spring cloud function and defining listeners like:
public Function<Flux<SomeObject>, Flux<OtherObject>> foo() {
//some code
}
I also reroute failed messages to DLQ. Problem is when fatal error like org.springframework.messaging.converter.MessageConversionException happens. It does not get processed by ConditionalRejectingErrorHandler like mentioned in https://docs.spring.io/spring-amqp/reference/html/#exception-handling, and keeps cycling forever.
Is there a way to make this work with ConditionalRejectingErrorHandler?
Right now i fix the problem by using #ServiceActivator(inputChannel = "errorChannel") and handling the errors myself.
Dependencies:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.4.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot.experimental</groupId>
<artifactId>spring-boot-starter-data-r2dbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-consul-config</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-hateoas</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
</dependency>
</dependencies>
We've long debated error handling and other features that we use for imperative functions and how they apply (or can they even be applied) to reactive functions and tried few different things, but unfortunately it all comes down to impedance mismatch.
The approaches you are describing are based on the operating on a single Message. That is the unit of work within the imperative style message handlers such as Function<String, String>. You use reactive style and by doing so changed the unit of work from a single message in the stream to the entire stream.
In short:
- Function<?, ?> - unit of work is Message
- Function<Flux<?>, Flux<?>> - unit of work is the entire stream
You can also observe it easily as reactive function is only invoked once for the duration of the life of the application while imperative is invoked once per each arriving message.
The reason why I am saying that is that the framework-based approaches we use for imperative message handlers (functions) can not be applied to reactive without causing side-effects. And generally reactive developers understand this especially given the richness of the reactive API specifically with regard to error handling
In any event, we'll update documentation accordingly.
Related
I have been trying to set up OAuth2 with Twitch through Spring for a few days now. I have resolved quite a few problems in that process but this one has stumped me. When I attempt to access one of the endpoints that I am trying to require users to be authenticated with Twitch for I am getting redirected to localhost:8080/login and being shown a page that simply reads "Login with OAuth 2.0" and has nothing else on it. My expectation was that Spring would automatically redirect me to Twitch's authentication portal and then Twitch would send me back to the Spring application after going through the OAuth process. Instead I am simply being shown that page and nothing is happening.
As far as what has been done so far to remedy this problem... pretty much nothing. I have not been able to find anyone else running into this problem so I am thinking that there are a few possible issues... Based on this tutorial https://spring.io/guides/tutorials/spring-boot-oauth2/ it seems like Spring has out of the box functionality for a lot of different OAuth providers. I am guessing that Twitch is not one of them. That makes me concerned that something on Twitch's backend might be causing this problem for Spring (if that is the case then I will need to make a custom authenticator). The other possibility I have thought of is that Spring might need to be told where to redirect users to for them to get authenticated. If that is the case then I would appreciate some help with that as I have not been able to find any indication that that is something that needs to be done online (and I have no idea how to do it).
Some important files from my project:
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.7</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.redacted</groupId>
<artifactId>redacted</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>redacted</name>
<description>redacted</description>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.apache.tomcat.embed</groupId>
<artifactId>tomcat-embed-jasper</artifactId>
<version>9.0.44</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<version>2.4.4</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-rest</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</project>
application.properties:
spring.mvc.view.prefix=/WEB-INF/jsp/
spring.mvc.view.suffix=.jsp
spring.security.oauth2.client.registration.twitch.client-id=redacted
spring.security.oauth2.client.registration.twitch.client-secret=redacted
spring.security.oauth2.client.registration.twitch.client-authentication-method=post
spring.security.oauth2.client.registration.twitch.redirect-uri=http://localhost:8080/login/oauth2/code/twitch
spring.security.oauth2.client.registration.twitch.provider=twitch
spring.security.oauth2.client.registration.twitch.scope=user:read:email
spring.security.oauth2.client.registration.twitch.authorization-grant-type=AUTHORIZATION_CODE
spring.security.oauth2.client.provider.twitch.authorization-uri=https://id.twitch.tv/oauth2/authorize
spring.security.oauth2.client.provider.twitch.token-uri=https://id.twitch.tv/oauth2/token
spring.security.oauth2.client.provider.twitch.user-info-uri=https://id.twitch.tv/oauth2/userinfo
spring.security.oauth2.client.provider.twitch.user-name-attribute=redacted
and also my SpringSecurityConfiguration file in the event it matters:
package com.redacted;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
#Configuration
#EnableWebSecurity
public class SpringSecurityConfiguration extends WebSecurityConfigurerAdapter {
#Override
public void configure(HttpSecurity httpSecurity) throws Exception {
//This allows users to access the "/" and "/Info" endpoints without authenticating with Twitch. To go anywhere else they will have to authenticate.
httpSecurity.antMatcher("/**").authorizeRequests().antMatchers("/", "/Info").permitAll().anyRequest().authenticated().and().oauth2Login();
}
}
My project structure:
Thank you for your time and consideration, I appreciate the help.
-Epoch
Edit:
Something I have recently found that I thought might be prudent to add to this discussion - when I am attempting to access a secured page in my current setup I see this screen:
It looks as if the intended way Spring normally displays this screen is like this:
It's almost as if Spring simply is not seeing my Oauth provider. Not sure if this information is helpful but I thought I would include it. Springs intended behavior for Oauth is that when there is only one provider configured the /login page is skipped altogether. I would actually prefer this behavior to be exhibited (but given it sees no providers I presume it shows the page).
I was able to recreate the issue. You have specified:
spring.security.oauth2.client.registration.twitch.authorization-grant-type=AUTHORIZATION_CODE
While it should be:
spring.security.oauth2.client.registration.twitch.authorization-grant-type=authorization_code
Spring Boot is unable to load your configuration properties properly, so it's as if you have 0 client registrations configured.
Try adding the .oauth2Client() in your configure method instead of .oauth2Login().
I am new at Kafka's universe and I am really stucked here. So, any help would be very much appreciated.
I have created a table out of a kafka stream, using the below KSQL statement:
CREATE TABLE calc AS
SELECT id, datetime, count(*)
FROM streamA
GROUP BY id, datetime
HAVING count(*) = total;
where "streamA" is a stream created by "topicA"
I am currently using:
Java 8,
Spring Boot v2.2.9
My pom.xml looks like:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<!-- Packaging -->
<modelVersion>4.0.0</modelVersion>
<packaging>jar</packaging>
<properties>
<spring-cloud.version>Hoxton.SR8</spring-cloud.version>
</properties>
<!-- Versioning -->
<groupId>some.name</groupId>
<artifactId>kafka.project</artifactId>
<version>2020.2.0-SNAPSHOT</version>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.9.RELEASE</version>
<relativePath />
</parent>
<!-- Meta-data -->
<name>[${project.artifactId}]</name>
<description>Kafka Project</description>
<!-- Dependencies -->
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.junit.vintage</groupId>
<artifactId>junit-vintage-engine</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<!-- Build settings -->
<build>
<!-- Plugins -->
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
So, have two questions:
Is there any way to access that table via Kafka Streams API?
Could I do something similar (e.g. creating that table) through my application instead of KSQL?
Thank you in advance
UPDATE
Thank you for your suggestion Shrey Jakhmola (What's the way of running KSQL from spring boot app), but I have a big dataset which needs to be accessed in a regular basis. I don't think this solution would be ideal.
#Joshua Oliphant, yes this table is generated by a stream which is created from a topic.
Is there any way to access that table via Kafka Streams API?
Table calc will be backed by a changelog topic called CALC. You are free to consume this topic in your application if you need. Either using the standard consumer or Kafka Streams.
However, if all you're wanting to do is query the current state of the table, then you can do so using ksqlDB's pull queries. These allow you to pull rows back from the table being built by ksqlDB. The functionality is basic, as its not part of the core streaming SQL that ksqlDB provides, but meets some use-cases.
If you need something beyond this, then there are other options open to you:
You can pump the result into a more traditional sql system of your choice, e.g. postgres, and query that. (You can use ksql's CREATE SINK CONNECTOR to export the data to postgres).
You can consume the data in your own app using the standard Kafka client. (Though this only works well if each instance of your app can hold all the data in the table).
You can use Kafka Streams within your app to consume the table. This has the benefit that multiple instances of your app can cluster together, so that each only consumes a portion of the table's data. You may then want to make use of Kafka Streams Interactive Queries to access the current state of the table.ation will load
Could I do something similar (e.g. creating that table) through my application instead of KSQL?
If you want to cut ksqlDB out of the equation, then yes, ksqlDB is internally using KAfka streams, so anything you can do with ksqlDB, you can also do directly with Kafka Streams.
SQL like:
CREATE TABLE calc AS
SELECT id, datetime, count(*)
FROM streamA
GROUP BY id, datetime
HAVING count(*) = total;
Would map to something like (rough code):
StreamsBuilder builder = new StreamsBuilder();
builder
.stream("streamA", Consumed.with(<appropriate serde>))
.groupBy(<a mapper that returns id and datetime as new key>)
.count()
.filter(<filter>);
.toStream()
.to("CALC");
new KafkaStreams(builder.build(), props, clients).start();
I was trying to deploy my own custom sink of spring cloud data flow onto cloud foundry.
My Dependency are below :
<dependencies>
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>spring-cloud-starter-stream-sink-log</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-rabbit</artifactId>
<version>1.2.0.RC1</version>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-core-dependencies</artifactId>
<version>1.2.0.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>log-app-dependencies</artifactId>
<version>1.2.0.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
And the custom code is very basic as below :
#EnableBinding(Sink.class)
public class SinkConfiguration {
private static final Log logger = LogFactory.getLog(SinkConfiguration.class);
#ServiceActivator(inputChannel=Sink.INPUT)
public void loggerSink(String payload) {
logger.info("Hello, Rahul. The time is: " + payload);
}
}
All I see when i deploy this application is that the Error channel subscriber is created , but no Input subscriber was created. Due to this no messages are being received on to this app. Source for this app is a custom source with rest controller. The default out of box streamer app -- LogSink works successfully. But i need to create a customsink to build things on top. Do anyone see an issue what I am missing here?
If your goal is to create an entirely new custom sink, you would develop that as a standalone application by following the Spring Initializr procedure. It is simple this way and you don't need to use the existing log-sink starter in this case.
If you are trying to patch any of the OOTB application; in this case, the log-sink, then follow the patching procedure. Pay attention to importing the configuration class. Unless you do that, the associated app-starter's behavior won't kick in.
Also, it seems you're using an old release for rabbit-binder. It is better to rely on the Spring Initializr generated artifact as opposed to handcrafting dependency versions.
I am specifically talking about the following.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-rest</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-rest-webmvc</artifactId>
<version>2.5.4.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-rest-core</artifactId>
<version>2.5.4.RELEASE</version>
</dependency>
The documentation here isn't very clear on the differences between spring-boot-starter-data-rest and spring-data-rest-webmvc. Is the difference essentially just the spring boot auto-configuration being included in the spring-boot-starter-data-rest artifact?
I assume that they both would include spring-data-rest-core at some point? How do I know which version of Spring Data REST WebMVC I would be using with spring-boot-start-data-rest?
I found this similar question but doesn't directly answer my question.
While I was fixing some issues in my client within an application using paho and spring framework, I discovered the spring MQTT integration. I'm not sure if this fix my problems, and I not even sure what is this.
Could someone explain what is the difference of using this together?
Do spring framework bring errors into paho that the integration solve?
or is save enough use this two technologies separate?
The dependencies used by me are:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>1.2.5.RELEASE</version>
</dependency>
<dependency>
<!-- Import dependency management from Spring Boot -->
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>1.2.5.RELEASE</version>
<type>pom</type>
<scope>import</scope>
</dependency>
and :
<dependency>
<artifactId>org.eclipse.paho.client.mqttv3</artifactId>
<groupId>org.eclipse.paho</groupId>
<version>1.0.2</version>
</dependency>
The integration of the dependency below it will be quite painful. I want to know if there is any advantage on doing this.
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-mqtt</artifactId>
<version>4.0.4.RELEASE</version>
</dependency>
Thank you all!
Spring Integration is sub framework in Spring eco system which allows to configure Enterprise Integration Patterns easily for Enterprise Application Integration.
It is good if you want to do processing of data with Source, Filter, Accumulator, Processor, Sink kind of a architecture with highly cohesive and loosely coupled components.
It is ideal to use where you have multiple systems speaking different protocols together which needs to integrated to with other systems without much fuss.