We Keep Coding

The specified queue does not exist - AWS.SimpleQueueService.NonExistentQueue with Spring-Boot

I'm unable to attach a #SqsListener in a spring boot application. It throws AWS.SimpleQueueService.NonExistentQueue exception.
I've gone through the question: Specified queue does not exist
and as far as I know, all the configurations are correct.
#Component
public class SQSListenerImpl{
#SqsListener(value = Constants.SQS_REQUEST_QUEUE, deletionPolicy = SqsMessageDeletionPolicy.NEVER)
public void listen(String taskJson, Acknowledgment acknowledgment, #Headers Map<String, String> headers) throws ExecutionException, InterruptedException {
//stuff
}
#PostConstruct
private void init(){
final AmazonSQS sqs = AmazonSQSClientBuilder.defaultClient();
LOGGER.info("Listing all queues in your account.\n");
for (final String queueUrl : sqs.listQueues().getQueueUrls()) {
LOGGER.info(" QueueUrl: " + queueUrl);
}
}
}
application.properties
cloud.aws.stack.auto=false
cloud.aws.region.static=ap-southeast-1
logging.level.root=INFO
Logs from above code:
[requestId: MainThread] [INFO] [SQSListenerImpl] [main] QueueUrl: https://sqs.ap-southeast-1.amazonaws.com/xxxxx/hello-world
[requestId: MainThread] [INFO] [SQSListenerImpl] [main] QueueUrl: https://sqs.ap-southeast-1.amazonaws.com/xxxxx/some-name2
[requestId: MainThread] [INFO] [SQSListenerImpl] [main] QueueUrl: https://sqs.ap-southeast-1.amazonaws.com/xxxxx/some-name3
[requestId: MainThread] [WARN] [SimpleMessageListenerContainer] [main] Ignoring queue with name 'hello-world': The queue does not exist.; nested exception is com.amazonaws.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: 3c0108aa-7611-528f-ac69-5eb01fasb9f3)
[requestId: MainThread] [INFO] [Http11NioProtocol] [main] Starting ProtocolHandler ["http-nio-8080"]
[requestId: MainThread] [INFO] [TomcatWebServer] [main] Tomcat started on port(s): 8080 (http) with context path ''
[requestId: MainThread] [INFO] [Startup] [main] Started Startup in 11.391 seconds (JVM running for 12.426)
Aws credentials used are under ~/.aws/ directory.
Now my question is, if sqs.listQueues() can see the queue then why can't #SqsListener? Am I missing something or doing something wrong?

I tried with SpringBoot Aws clound like you and got same error.
Then i used the full http url as queue name and got access denied error
#SqsListener(value = "https://sqs.ap-southeast-1.amazonaws.com/xxxxx/hello-world")
So in the end, i end up using AWS SDK directly to get message from SQS

Here's what I'm doing with Spring Cloud.
Using SPEL I'm attaching a value from my application.properties to the Annotation #SqsListener like this
#SqsListener(value = "#{queueConfiguration.getQueue()}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
One thing to note, make sure you use the full HTTPS path for the queue.
For all local development, I'm using "localstack" and using a local implementation of SQS but the same code applies as it gets deploy in ECS. The other piece to note is that the role or instance needs to be able to Receive Messages via IAM to make this happen.

Using full URL worked for me.
#SqsListener("https://sqs.ap-south-1.amazonaws.com/xxxxxxxxx/queue-name")

Using below code work for me
#SqsListener(value="https://sqs.us-west-1.amazonaws.com/xxxxx/queue-name")

Tomcat 9 Valve Causing Server Start Failure

I have written a custom tomcat valve to parse HTTP headers and use them to authenticate. The valve works by extending AuthenticatorBase. I compiled it and placed it in $CATALINA_HOME/lib. Here is code:
public class TomcatLogin extends AuthenticatorBase {
private String[] roleNames = null;
private String ivcreds = null;
private String ivuser = null;
private String ivgroups = null;
private GenericPrincipal principal = null;
// Constructor defers to super class (Authenticator base)
public TomcatLogin(){
super();
}
protected boolean doAuthenticate(Request request, HttpServletResponse response)
{
List<String> groupsList = null;
System.out.println("Obtaining Headers from Request");
try {
ivuser = request.getHeader("iv-user");
ivcreds = request.getHeader("iv-creds");
ivgroups = request.getHeader("iv-groups");
} catch(Exception e) {
e.printStackTrace();
}
// Require all header credentials for proper authentication
if(ivuser == null || ivcreds == null || ivgroups == null)
return false;
// Split ivgroups by comma seporated value
// Then remove head and tail quotation marks
roleNames = ivgroups.split(",");
for(int i=0; i<roleNames.length; i++){
roleNames[i] = roleNames[i].substring(1, roleNames[i].length()-1 );
groupsList.add(roleNames[i]);
}
principal = new GenericPrincipal(ivuser, ivcreds, groupsList);
request.setUserPrincipal(principal);
return true;
}
public String getAuthMethod() {
return "HTTPAuthenticator";
}
}
I then tell Tomcat to use the valve in the server.xml. The documentation for extending AuthenticatorBase says When this class is utilized, the Context to which it is attached (or a parent Container in a hierarchy) must have an associated Realm that can be used for authenticating users and enumerating the roles to which they have been assigned. I thought I had configured it correctly, but it throws and error and Tomcat fails to start. Here is the server.xml config:
<?xml version="1.0" encoding="UTF-8"?>
...
<Server>
<Service name="Catalina">
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<!-- Here is my valve -->
<Valve className="package.of.my.custom.valve.TomcatLogin" />
<Host ... >
...
</Host>
</Engine>
</Service>
</Server>
And here is the Error message I am getting:
10-Jan-2019 10:11:03.576 SEVERE [main] org.apache.tomcat.util.digester.Digester.endElement End event threw exception
java.lang.reflect.InvocationTargetException
... A bunch of useless stacktrace
Caused by: java.lang.IllegalArgumentException: Configuration error: Must be attached to a Context at org.apache.catalina.authenticator.AuthenticatorBase.setContainer(AuthenticatorBase.java:278)
at org.apache.catalina.core.StandardPipeline.addValve(StandardPipeline.java:335)
at org.apache.catalina.core.ContainerBase.addValve(ContainerBase.java:1133)
... 27 more
10-Jan-2019 10:11:03.579 WARNING [main] org.apache.catalina.startup.Catalina.load Catalina.start using conf/server.xml: Error at (107, 82) : Configuration error: Must be attached to a Context
10-Jan-2019 10:11:03.579 SEVERE [main] org.apache.catalina.startup.Catalina.start Cannot start server. Server instance is not configured.
I think my valve is written correctly, so my guess is that the issue is in the configuration. Not sure why it is not getting a context to attach to. Any ideas?
Edit:
I tried putting the valve in my app's META-INF/context.xml (I had to make one since there wasn't one to begin with). Here it is:
<?xml version="1.0"?>
<Context>
<Valve className="package.of.my.custom.valve.TomcatLogin" />
</Context>
The server then start, but it fails to deploy any of the applications. I am getting similar error to an IBM valve which I originally tried to use over this custom implementation where it cannot find the AuthenticatorBase class from catalina.jar. Here are the SEVERE errors I am getting:
10-Jan-2019 15:34:06.673 SEVERE [main] org.apache.catalina.core.ContainerBase.addChildInternal ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/sample-DB]]
...
Stacktrace info
...
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:441)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.tomcat.util.digester.Digester.startDocument(Digester.java:1102)
Caused by: java.lang.NullPointerException
at java.nio.charset.Charset.put(Charset.java:538)
...
Stacktrace
...
10-Jan-2019 15:34:06.688 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/fmac/deploy/sample.war] has finished in [11] ms
10-Jan-2019 15:34:06.689 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/fmac/deploy/sample-auth.war]
10-Jan-2019 15:34:06.692 SEVERE [main] org.apache.tomcat.util.digester.Digester.startElement Begin event threw error
java.lang.NoClassDefFoundError: org/apache/catalina/authenticator/AuthenticatorBase
...
Stacktrace
The Error Below is the most confusing one. How can it not find this class?
...
Caused by: java.lang.ClassNotFoundException: org.apache.catalina.authenticator.AuthenticatorBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
...
Stacktrace
...
10-Jan-2019 15:34:13.823 SEVERE [https-jsse-nio2-8443-exec-3] org.apache.coyote.http11.Http11Processor.service Error processing request
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.tomcat.util.buf.B2CConverter
at org.apache.catalina.connector.CoyoteAdapter.convertURI(CoyoteAdapter.java:1072)

Put your <Valve> inside of your <Context> element. That should usually be within a META-INF/context.xml file in your web application and not in conf/server.xml.

So I was finally able to find out what the issues were. We have a custom tomcat implementation, part of which involves appending the classpath in the java command. For some reason, some of the database drivers that were being used caused the Tomcat to fail to find any of the libraries in CATALINA_HOME/lib. I'm running inside a Docker container and this was some old vestigial stuff from a VM version. We ended up just having to toss those
drivers out.
Not really sure why they would completely override the base lib/ directory, but at least these errors when away and I was actually able to use the pre-built authenticator I had instead of fine tuning this custom one.

How to configure apache camel to see the reason of the shutdown?

Preconditions:
Camel 2.17
I had defined some routes, that routes contain entries like:
.to ("log:org.apache.camel?level=DEBUG")
My logback config contains:
<logger name="org.apache.camel" level="TRACE" />
Context definition begins with:
<camel:camelContext id="someContext" ... trace="true">
When I am starting Camel, then I see Camel is proceeding and finally without ANY error report just shutting down. This looks like:
2016-10-04 13:40:56,146 [localhost-startStop-1] TRACE org.apache.camel.model.ProcessorDefinitionHelper - There are 6 properties on: From[direct:process]
2016-10-04 13:40:58,042 [localhost-startStop-1] DEBUG org.apache.camel.spring.SpringCamelContext - onApplicationEvent: org.springframework.context.event.ContextClosedEvent[source=Root WebApplicationContext: startup date [Tue Oct 04 13:37:25 CEST 2016]; root of context hierarchy]
2016-10-04 13:40:58,066 [localhost-startStop-1] INFO org.apache.camel.spring.SpringCamelContext - Apache Camel 2.17.3 (CamelContext: someContext) is shutting down
I have as well:
onException( java.lang.Exception.class )
.handled( false )
.to( "log:GeneralError?level=ERROR" );
But this is more related to the exchange processing and not to startup.
Is there any generic way to check what is going on out there?
For example:
Is there any class missing and class loader fails?
Or is any exception thrown?
Or some connection fails?
Complete route definition:
final RouteDefinition kafkaRouteDefinition = from( "kafka:{{kafka.broker.endpoints}}" +
"?topic={{kafka.topic.name}}" +
"&groupId=my_group" +
"&autoOffsetReset=earliest" +
"&consumersCount={{kafka.consumer.count}}" );
LOG.info( "Kafka route definition: " + kafkaRouteDefinition.toString() );
kafkaRouteDefinition
.routeId( Constants.ROUTE_ID_PROCESS_KAFKA_MESSAGES )
.to( "log:org.apache.camel?level=DEBUG" )
.process( new RawMessageProcessor() ).id( RawMessageProcessor.class.getSimpleName() )
.to( "log:org.apache.camel?level=DEBUG" )
.unmarshal( inputMessageFormat ).id( "ConvertRawMessageToLogline" )
.to( "log:org.apache.camel?level=DEBUG" )
.process( new LoglineMessageProcessor() ).id( LoglineMessageProcessor.class.getSimpleName() )
.to( "log:org.apache.camel?level=DEBUG" )
.to( Constants.CAMEL_PROCESS_ENDPOINT )
.to( "log:org.apache.camel?level=DEBUG" )
.multicast().stopOnException()
.to( "log:org.apache.camel?level=DEBUG" )
.to( Constants.CAMEL_STORE_ENDPOINT
, Constants.CAMEL_INDEX_ENDPOINT
)
.to( "log:org.apache.camel?level=DEBUG" )
.end();

I had Similar issue [But I was using Spring]
This happens when main method which loads camel Context exits before camel context is fully loaded
Add below code in your test case & it should run
org.apache.camel.spring.Main main = new Main();
main.setApplicationContextUri("camel-context.xml");
main.start();
Thread.sleep(1000);
More over you can also make autostart stop & start camel context later whenever needed
<camelContext id="myCamel" xmlns="http://camel.apache.org/schema/spring" autoStartup="false">
<route>
<from uri="direct:start"/>
<to uri="mock:result"/>
</route>
</camelContext>
then in some java file
ApplicationContext ac = ...
SpringCamelContext camel = (SpringCamelContext) ac.getBean("myCamel");
// now start Camel manually
camel.start();

Get org.springframework to stop polluting my logs

I am using Payara 4.1 and Netbeans 8.1.
When I run my application, these four lines are among the first to be logged:
#!## LogManagerService.postConstruct : rootFolder=/opt/payara41/glassfish
#!## LogManagerService.postConstruct : templateDir=/opt/payara41/glassfish/lib/templates
#!## LogManagerService.postConstruct : src=/opt/payara41/glassfish/lib/templates/logging.properties
#!## LogManagerService.postConstruct : dest=/opt/payara41/glassfish/domains/domain1/config/logging.properties
I added the line org.springframework=WARNING at the end of the last logging.properties file given above, and restarted my server. That didn't seem to have an effect. When I open the asadmin shell /opt/payara41/bin/asadmin and run list-log-attributes, here is what I get:
asadmin> list-log-attributes
com.sun.enterprise.server.logging.GFFileHandler.excludeFields <>
com.sun.enterprise.server.logging.GFFileHandler.file <${com.sun.aas.instanceRoot}/logs/server.log>
com.sun.enterprise.server.logging.GFFileHandler.flushFrequency <1>
com.sun.enterprise.server.logging.GFFileHandler.formatter <com.sun.enterprise.server.logging.ODLLogFormatter>
com.sun.enterprise.server.logging.GFFileHandler.logtoConsole <false>
com.sun.enterprise.server.logging.GFFileHandler.maxHistoryFiles <0>
com.sun.enterprise.server.logging.GFFileHandler.multiLineMode <true>
com.sun.enterprise.server.logging.GFFileHandler.retainErrorsStasticsForHours <0>
com.sun.enterprise.server.logging.GFFileHandler.rotationLimitInBytes <2000000>
com.sun.enterprise.server.logging.GFFileHandler.rotationOnDateChange <false>
com.sun.enterprise.server.logging.GFFileHandler.rotationTimelimitInMinutes <0>
com.sun.enterprise.server.logging.SyslogHandler.useSystemLogging <false>
handlerServices <com.sun.enterprise.server.logging.GFFileHandler,com.sun.enterprise.server.logging.SyslogHandler>
handlers <java.util.logging.ConsoleHandler>
java.util.logging.ConsoleHandler.formatter <com.sun.enterprise.server.logging.UniformLogFormatter>
java.util.logging.FileHandler.count <1>
java.util.logging.FileHandler.formatter <java.util.logging.XMLFormatter>
java.util.logging.FileHandler.limit <50000>
java.util.logging.FileHandler.pattern <%h/java%u.log>
log4j.logger.org.hibernate.validator.util.Version <warn>
org.springframework <WARNING>
Command list-log-attributes executed successfully.
I have tried the suggestions given in this SO question but that didn't work. I'm stumped. :( I really do not want to see all those INFO logging lines printed by the spring framework.
=== EDIT ===
Here is what is in my log4j.properties (copied from SO):
# Define the root logger with appender file
log4j.rootLogger = DEBUG, stdout
# Define the file appender
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
# Set the name of the logs destination
log4j.appender.stdout.target=System.out
# Set the immediate flush to true (default)
log4j.appender.stdout.ImmediateFlush=true
# Set the threshold to debug mode
log4j.appender.stdout.Threshold=debug
# Set the append to false, overwrite
log4j.appender.stdout.Append=false
# Define the layout for appender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.conversionPattern=%d{yyyy-MM-dd}:%m%n
log4j.logger.org.springframework=WARNING
Here are all the places I have copied that file to:
$PROJECT_DIR/src/log4j.properties
$PROJECT_DIR/src/main/resources/META-INF/log4j.properties
$PROJECT_DIR/src/main/webapp/WEB-INF/log4j.properties
$PROJECT_DIR/src/main/webapp/WEB-INF/classes/log4j.properties
$PROJECT_DIR/target/$PROJECT-1.0/WEB-INF/log4j.properties
$PROJECT_DIR/target/$PROJECT-1.0/WEB-INF/classes/log4j.properties
$PROJECT_DIR/target/$PROJECT-1.0/WEB-INF/classes/META-INF/log4j.properties
$PROJECT_DIR/target/classes/META-INF/log4j.properties
/opt/payara41/glassfish/domains/domain1/config/log4j.properties
In addition, I have added log4j.logger.org.springframework=WARNING and org.springframework=WARNING to Configurations > server-config > Logger Settings > Module Log Levels in the Payara UI. Nothing is working. Payara's server log still has INFO lines for the spring framework.

I tried everything that has been suggested so far, to no avail. So I ended up doing something pretty hacky. I wrote my own log handler and installed it to domain1/lib/ext:
package org.springsuppressor;
import java.util.logging.ConsoleHandler;
import java.util.logging.Level;
public class MyHandler extends ConsoleHandler {
public MyHandler() {
super();
setLevel(Level.WARNING);
}
}
The handler above sets the log level for all loggers to WARNING. I could suppress messages for just the spring framework, but I don't yet need to do that.

Cloudant j2se prototype throwing HTTPRequest exception

I'm trying to follow the replication guide on github for cloudant to prototype a basic one way connection between a local couchdb and and a locally running j2se app utilizing the cloudant sync library.
The local datastore file is being created, however almost immediately after running my application, I'm getting a runtime error - which at a glance I'm a little unsure of how to resolve: Asside from the cloudant code and dependancies;
This is the entire code for my app - I have apache couchdb installed on my machine and the database called 'baseball' exists already:
import com.cloudant.sync.datastore.DatastoreManager;
import com.cloudant.sync.datastore.Datastore;
import com.cloudant.sync.replication.*;
import com.cloudant.sync.notifications.*;
import com.google.common.eventbus.Subscribe;
import java.util.concurrent.CountDownLatch;
import java.io.*;
import java.net.*;
//https://github.com/cloudant/sync-android/blob/master/doc/replication.md
public class CloudApp{
public static void main(String [] args) throws Exception{
File path = new File("datastores");
System.out.println(path.getAbsolutePath());
DatastoreManager manager = new DatastoreManager(path.getAbsolutePath());
URI uri = new URI("http://127.0.0.1:5984/baseball");
Datastore ds = manager.openDatastore("my_datastore");
// Create a replictor that replicates changes from the remote
// database to the local datastore.
Replicator replicator = ReplicatorFactory.oneway(uri, ds);
// Use a CountDownLatch to provide a lightweight way to wait for completion
CountDownLatch latch = new CountDownLatch(1);
Listener listener = new Listener(latch);
replicator.getEventBus().register(listener);
replicator.start();
latch.await();
replicator.getEventBus().unregister(listener);
if (replicator.getState() != Replicator.State.COMPLETE) {
System.out.println("Error replicating FROM remote");
System.out.println(listener.error);
}
}
}
class Listener {
private final CountDownLatch latch;
public ErrorInfo error = null;
Listener(CountDownLatch latch) {
this.latch = latch;
}
#Subscribe
public void complete(ReplicationCompleted event) {
latch.countDown();
}
#Subscribe
public void error(ReplicationErrored event) {
this.error = event.errorInfo;
latch.countDown();
}
}
And this is the runtime error as well as log outputs:
[I]DatastoreManager: path: /Users/reecegriffin/Documents/workspace/Cloudant 2/datastores
[I]DatastoreManager: dbDirectory: /Users/reecegriffin/Documents/workspace/Cloudant 2/datastores/my_datastore
Mar 10, 2014 9:29:02 AM com.almworks.sqlite4java.Internal log
INFO: [sqlite] DB[1]: instantiated [/Users/reecegriffin/Documents/workspace/Cloudant 2/datastores/my_datastore/db.sync]
Mar 10, 2014 9:29:02 AM com.almworks.sqlite4java.Internal log
INFO: [sqlite] Internal: loaded sqlite4java-osx from /Users/reecegriffin/Documents/workspace/Cloudant 2/src/libsqlite4java-osx.jnilib
Mar 10, 2014 9:29:02 AM com.almworks.sqlite4java.Internal log
INFO: [sqlite] Internal: loaded sqlite 3.7.10, wrapper 0.2
Mar 10, 2014 9:29:02 AM com.almworks.sqlite4java.Internal log
INFO: [sqlite] DB[1]: opened
Exception in thread "main" java.lang.IllegalStateException: java.lang.RuntimeException: Stub!
at com.cloudant.mazha.HttpRequests.createHttpClient(HttpRequests.java:185)
at com.cloudant.mazha.HttpRequests.<init>(HttpRequests.java:69)
at com.cloudant.mazha.CouchClient.<init>(CouchClient.java:47)
at com.cloudant.sync.replication.CouchClientWrapper.<init>(CouchClientWrapper.java:45)
at com.cloudant.sync.replication.BasicPullStrategy.<init>(BasicPullStrategy.java:90)
at com.cloudant.sync.replication.PullReplication.createReplicationStrategy(PullReplication.java:40)
at com.cloudant.sync.replication.BasicReplicator.getReplicationStrategy(BasicReplicator.java:41)
at com.cloudant.sync.replication.BasicReplicator.start(BasicReplicator.java:61)
at CloudApp.main(CloudApp.java:27)
Caused by: java.lang.RuntimeException: Stub!
at org.apache.http.params.AbstractHttpParams.<init>(AbstractHttpParams.java:5)
at org.apache.http.params.BasicHttpParams.<init>(BasicHttpParams.java:6)
at com.cloudant.mazha.HttpRequests.getHttpConnectionParams(HttpRequests.java:199)
at com.clou
dant.mazha.HttpRequests.createHttpClient(HttpRequests.java:173)
... 8 more
Many thanks for any help.
Regards,
Reece.

I noticed in some other threads regarding Stub related apache commons HTTPclient related stuff, that the android library build path order can have an effect. I was trying to run from within eclipse & sure enough - after I opened up the build path editor and moved android.jar to be last on my build path, the run time error went away.

There's some more information on the "stub!" issue in Using android.jar in Java project - RuntimeException Stub?.
We'll take a look into whether we can mitigate the effects of the build path order in code, but I'm not sure whether we'll be able to fix the issue.
This problem hasn't come up for us in tests. Perhaps this is because the Android dependency is specified last in our build.gradle file, meaning it ends up in a position in the path ordering that works.