Local Apache Spark on heroku - java

I have a problem running the apache spark in local mode on Heroku with Spring. My issue is:
#Configuration
public class SparkConfig {
private String master = "local";
#Bean
public JavaSparkContext javaSparkContext() {
SparkConf conf = new SparkConf().setAppName("local")
.setMaster(master);
return new JavaSparkContext(conf);
}
}
Error:
ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'mainController': Unsatisfied dependency expressed through field 'javaSparkContext'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'javaSparkContext' defined in class path resource [com/sensorservice/config/SparkConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.spark.api.java.JavaSparkContext]: Factory method 'javaSparkContext' threw exception; nested exception is java.lang.IllegalArgumentException: System memory 326631424 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
I tried to modify the parameters Spark related to memory but this had no effect. It is possible to launch Spark free Heroku?

Related

Disable #SqsListener for local environment

I have the following value configured in my SQS Listener in springboot. The queue is configured only for DEV and STAGE environments. So I want to disable it in my local after testing. How do i achieve that?
#SqsListener(value = { "${cloud.aws.endpoint}" } , deletionPolicy =SqsMessageDeletionPolicy.ON_SUCCESS)
public void processMessage(String message) throws Exception {
}
I tried to add the following to my application-local.xml(to keep it local), but it didn't work. Anyone went through the same scenario?
autoconfigure:
exclude:
- org.springframework.cloud.aws.autoconfigure.messaging.MessagingAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextStackAutoConfiguration
- org.springframework.cloud.aws.autoconfigure.context.ContextRegionProviderAutoConfiguration
I am getting the following error everytime.
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'stackResourceRegistryFactoryBean' defined in class path resource [org/springframework/cloud/aws/autoconfigure/context/ContextStackAutoConfiguration.class]: Unsatisfied dependency expressed through method 'stackResourceRegistryFactoryBean' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'autoDetectingStackNameProvider' defined in class path resource [org/springframework/cloud/aws/autoconfigure/context/ContextStackAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.aws.core.env.stack.config.StackNameProvider]: Factory method 'autoDetectingStackNameProvider' threw exception; nested exception is java.lang.IllegalArgumentException: No valid instance id defined
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:799) ~[spring-beans-5.2.12.RELEASE.jar:5.2.12.RELEASE]
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) [aws-java-sdk-core-1.11.867.jar:na]
Caused by: java.net.ConnectException: Host is down (connect failed)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'autoDetectingStackNameProvider' defined in class path resource [org/springframework/cloud/aws/autoconfigure/context/ContextStackAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.aws.core.env.stack.config.StackNameProvider]: Factory method 'autoDetectingStackNameProvider' threw exception; nested exception is java.lang.IllegalArgumentException: No valid instance id defined
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:657) ~[spring-beans-5.2.12.RELEASE.jar:5.2.12.RELEASE]
at org.sprin
I've faced similar issues while using #SqsListener. What I usually do is comment out the annotation when running the app locally - it's ugly but it works.

unable to deploy to heroku since jhipster update

I upgrade my project from jhipster 5.8.2 to 6.5.1.
All works fine in develop mode, on my Windows computer
Now, I try to deploy on my Test heroku platform, but I have an exception :
Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException:
Unable to start web server; nested exception is java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'servletEndpointRegistrar'
defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/ServletEndpointManagementContextConfiguration$WebMvcServletEndpointManagementContextConfiguration.class]:
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]:
Factory method 'servletEndpointRegistrar' threw exception;
nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'auditEventsEndpoint'
defined in class path resource [org/springframework/boot/actuate/autoconfigure/audit/AuditEventsEndpointAutoConfiguration.class]:
Unsatisfied dependency expressed through method 'auditEventsEndpoint' parameter 0;
nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException:
Error creating bean with name 'customAuditEventRepository'
defined in URL [jar:file:/app/build/libs/yvidya-0.0.1-SNAPSHOT.war!/WEB-INF/classes!/fr/yoni/yvidya/repository/CustomAuditEventRepository.class]:
Unsatisfied dependency expressed through constructor parameter 0;
nested exception is org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'persistenceAuditEventRepository':
Cannot create inner bean '(inner bean)#75cf0de5' of type [org.springframework.orm.jpa.SharedEntityManagerCreator] while setting bean property 'entityManager';
nested exception is org.springframework.beans.factory.BeanCreationException:
Error creating bean with name '(inner bean)#75cf0de5': Cannot resolve reference to bean 'entityManagerFactory' while setting constructor argument;
nested exception is org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]:
Invocation of init method failed;
This is my CustomAuditEventRepository
#Repository
public class CustomAuditEventRepository implements AuditEventRepository {
private static final String AUTHORIZATION_FAILURE = "AUTHORIZATION_FAILURE";
/**
* Should be the same as in Liquibase migration.
*/
protected static final int EVENT_DATA_COLUMN_MAX_LENGTH = 255;
private final PersistenceAuditEventRepository persistenceAuditEventRepository;
private final AuditEventConverter auditEventConverter;
private final Logger log = LoggerFactory.getLogger(getClass());
public CustomAuditEventRepository(PersistenceAuditEventRepository persistenceAuditEventRepository,
AuditEventConverter auditEventConverter) {
this.persistenceAuditEventRepository = persistenceAuditEventRepository;
this.auditEventConverter = auditEventConverter;
}
For deploy, I use gitlab-ci, scripts launch are :
./gradlew compileJava -x check -PnodeInstall --no-daemon
./gradlew -Pint -Pwar clean bootWar
rm -rf build/libs/yvidya-*original.war
heroku plugins:install java
heroku deploy:jar build/libs/*.war --app yvidya-int
And my config Database :
#Configuration
#EnableJpaRepositories("fr.yoni.yvidya.repository")
#EnableJpaAuditing(auditorAwareRef = "springSecurityAuditorAware")
#EnableTransactionManagement
public class DatabaseConfiguration {
private final Logger log = LoggerFactory.getLogger(DatabaseConfiguration.class);
}
My config JDBC
spring:
profiles:
include:
- swagger
devtools:
restart:
enabled: false
livereload:
enabled: false
datasource:
type: com.zaxxer.hikari.HikariDataSource
url: jdbc:postgres://XXXX.compute.amazonaws.com:5432/YYYYYY
username: YYIYYI
password: TTTTTTTTTTTTTTTTTTTTTTTTTTTTT
hikari:
poolName: Hikari
auto-commit: false
jpa:
database-platform: io.github.jhipster.domain.util.FixedPostgreSQL95Dialect
database: POSTGRESQL
have you an idea why I can not deploy on heroku since upgrade ?

#RepositoryRestResource did not work Spring boot 2.2.1.RELEASE. while running occured error RegionRepository must only contain a single path segment

#RepositoryRestResource(path = "/region", collectionResourceRel = "list", excerptProjection = CustomRegion.class)
public interface RegionRepository extends JpaRepository<Region, Integer> {
}
data rest class
#SpringBootApplication
public class ProfUzApplication {
public static void main(String[] args) {
SpringApplication.run(ProfUzApplication.class, args);
}
}
main running class
Occured error
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'repositorySearchController' defined in URL [jar:file:/C:/Users/saidk/.m2/repository/org/springframework/data/spring-data-rest-webmvc/3.2.1.RELEASE/spring-data-rest-webmvc-3.2.1.RELEASE.jar!/org/springframework/data/rest/webmvc/RepositorySearchController.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityLinks' defined in class path resource [org/springframework/data/rest/webmvc/config/RepositoryRestMvcConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.rest.webmvc.support.RepositoryEntityLinks]: Factory method 'entityLinks' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'resourceMappings' defined in class path resource [org/springframework/data/rest/webmvc/config/RepositoryRestMvcConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.rest.core.mapping.RepositoryResourceMappings]: Factory method 'resourceMappings' threw exception; nested exception is java.lang.IllegalStateException: Path /region configured for uz.pdp.prof.repository.RegionRepository must only contain a single path segment!
You have to mention path='endpoint name' without using '/' in #RepositoryRestResource. You have to use #RepositoryRestResource(path = "region"), it's direct spring MVC to create RESTful endpoints at /region
The problem is happening because you defined the path using "/". Spring does not allow us to define a composed path like "/api/v1/my_entity". Just remove the "/" from your path and it will work.
I created a new project based on 2.2.2 a couple days ago and got the same issue.
After a few tries, the way to fix it is to use the same value in path and collectionResourceRel. In your case, try to use "region" in both path and collectionResourceRel.

Spring Redisson Sentinel Error - At least two sentinels are required

I am using Sentinel mode for Redis. I am using Redisson Client version 3.9.1
<dependencies>
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson-spring-data-20</artifactId>
<version>${redisson-version}</version>
</dependency>
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson</artifactId>
<version>${redisson-version}</version>
</dependency>
</dependencies>
The configuration of Sentinel using Redisson:
#EnableCaching
public class RedisCache {
#Autowired
private ObjectMapper objectMapper;
#Bean
public RedissonClient redissonClient() {
Config config = new Config();
config.setCodec(new JsonJacksonCodec(objectMapper));
config.useSentinelServers()
.setMasterName("redis-master")
.addSentinelAddress("redis://localhost:26378")
.addSentinelAddress("redis://localhost:26379");
return Redisson.create(config);
}
#Bean
public RedissonConnectionFactory redissonConnectionFactory(RedissonClient redisson) {
return new RedissonConnectionFactory(redisson);
}
}
I am running two sentinel servers and one Redis master with no slave.
This is just a setup on my local environment.
Output of some commands:
redis-cli -p 26379
127.0.0.1:26379> SENTINEL get-master-addr-by-name redis-master
1) "127.0.0.1"
2) "6379"
redis-cli -p 26378
127.0.0.1:26378> SENTINEL get-master-addr-by-name redis-master
1) "127.0.0.1"
2) "6379"
Whenever I am running my Spring Boot Application, I get the following error:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redissonClient' defined in class path resource [xxxx/cache/impl/redis/config/RedisCache.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.redisson.api.RedissonClient]: Factory method 'redissonClient' threw exception; nested exception is org.redisson.client.RedisConnectionException: At least two sentinels should be defined in Redis configuration!
......
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0 (AbstractBeanFactory.java:317)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton (DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean (AbstractBeanFactory.java:315)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean (AbstractBeanFactory.java:204)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType (ServletContextInitializerBeans.java:226)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType (ServletContextInitializerBeans.java:214)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans (ServletContextInitializerBeans.java:91)
at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init> (ServletContextInitializerBeans.java:80)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans (ServletWebServerApplicationContext.java:250)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize (ServletWebServerApplicationContext.java:237)
at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup (TomcatStarter.java:54)
at org.apache.catalina.core.StandardContext.startInternal (StandardContext.java:5245)
at org.apache.catalina.util.LifecycleBase.start (LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase$StartChild.call (ContainerBase.java:1420)
at org.apache.catalina.core.ContainerBase$StartChild.call (ContainerBase.java:1410)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.566 s
[INFO] Finished at: 2018-12-07T13:44:14+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.1.1.RELEASE:run (default-cli) on project xxx: An exception occurred while running. null: InvocationTargetException: Unable to start web server; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat: Error creating bean with name 'servletEndpointRegistrar' defined in class path resource [org/springframework/boot/actuate/autoconfigure/endpoint/web/ServletEndpointManagementContextConfiguration$WebMvcServletEndpointManagementContextConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.endpoint.web.ServletEndpointRegistrar]: Factory method 'servletEndpointRegistrar' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'healthEndpoint' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthEndpoint]: Factory method 'healthEndpoint' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.actuate.autoconfigure.redis.RedisHealthIndicatorConfiguration': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'redissonConnectionFactory' defined in class path resource [xxxxxxx/cache/impl/redis/config/RedisCache.class]: Unsatisfied dependency expressed through method 'redissonConnectionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redissonClient' defined in class path resource [xxxxxx/cache/impl/redis/config/RedisCache.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.redisson.api.RedissonClient]: Factory method 'redissonClient' threw exception; nested exception is org.redisson.client.RedisConnectionException: At least two sentinels should be defined in Redis configuration!
How do I resolve this error?
PS: I have tried master-slave configuration also, with one sentinel attached to the host and other one attached to the slave, but the error remains the same. I have tried sentinels running on different local machines with different private IPs, but the error remains the same.
Tried with 3 sentinel servers, no benefits.
Redisson doesn't seem to be working with localhost addresses. When I replaced the sentinel nodes addresses with the ones hosted on AWS, it worked fine.
Redisson, in my opinion, is a very immature client. Consider using lettuce or jedis.
See the bug filed at https://github.com/redisson/redisson/issues/1736
which doesn't allow it to shut down.
Most of the problems I faced were when I used the sentinel mode. Cluster mode I have not tried. Works fine in single mode.
Also, they don't cater to bugs or issues filed on github for months which is obvious as they have a pro and paid version, so you are on your own.
I won't recommend using Redisson as of now.
Redisson perfectly works with localhost addresses. In your case SENTINEL SENTINELS command returns empty result. Which means that Redis setup is not configured properly.
Make sure you run both redis-server as well as redis-sentinal <config>.
Once done, check which port sentinal is running at. It could also be 5000, it was in my case. Then pass "redis://localhost:5000" to addSentinelAddress.

Spring Boot and Teradata UnsatisfiedDependencyException

I am trying to create a simple Spring application with Teradata database.
It was working with mySql database and driver but after changing to Teradata driver/databased i receive following exception :
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in class path resource [org/springframework/boot/autoconfigure/jdbc/DataSourceConfiguration$Tomcat.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.tomcat.jdbc.pool.DataSource]: Factory method 'dataSource' threw exception; nested exception is java.lang.IllegalArgumentException: URL must start with 'jdbc'
Application Properties :
spring.datasource.url =jdbc:teradata://servername/db
spring.datasource.username = dbc
spring.datasource.password = dbc
spring.datasource.driverClassName=com.ncr.teradata.TeraDriver
I think that is problem of Spring you might be using #Autowire on constructor but spring can't inject class with that name. To solve the problem, just define a DataSource in your context - at which point Spring will be able to inject the bean and correctly bootstrap the context.

Categories