We Keep Coding

Apache Camel Route is getting started automatically

Route loadfile is getting started automatically when I start main class.
On exception, when process should finish. It starts loadfile again and again.
It should get start from timer and then should call loadfile route, but loadfile is starting independent as well as from timer.
CamelContext context = new DefaultCamelContext(sr);
try {
context.addRoutes(new RouteBuilder() {
#Override
public void configure() throws Exception {
onException(Exception.class)
.log(LoggingLevel.INFO, "Extype:${exception.message}")
.stop();
from("timer://alertstrigtimer?period=60s&repeatCount=1")
.startupOrder(1)
.log(LoggingLevel.INFO, "*******************************Job-Alert-System: Started: alertstrigtimer******************************")
.to("direct:loadFile").stop();
from("direct:loadFile").routeId("loadfile")
.log(LoggingLevel.INFO, "*******************************Job-Alert-System: Started: direct:loadFile******************************")
.from(getTriggerFileURI(getWorkFilePath(), getWorkFileName())).choice()
.
.
});
context.start();
Thread.sleep(40000);
Following is log:
[main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.21.1 (CamelContext: camel-1) is starting
[main] INFO org.apache.camel.management.ManagedManagementStrategy - JMX is enabled
[main] INFO org.apache.camel.impl.converter.DefaultTypeConverter - Type converters loaded (core: 194, classpath: 14)
[main] INFO org.apache.camel.impl.DefaultCamelContext - StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
[main] INFO org.apache.camel.impl.DefaultCamelContext - Route: route1 started and consuming from: timer://alertstrigtimer?period=60s&repeatCount=1
[main] INFO org.apache.camel.impl.DefaultCamelContext - Skipping starting of route loadfile as its configured with autoStartup=false
[main] INFO org.apache.camel.impl.DefaultCamelContext - Route: loadDataAndAlerts started and consuming from: direct://loadDataAndAlerts
[main] INFO org.apache.camel.impl.DefaultCamelContext - Total 4 routes, of which 2 are started
[main] INFO org.apache.camel.impl.DefaultCamelContext - Apache Camel 2.21.1 (CamelContext: camel-1) started in 0.761 seconds
[Camel (camel-1) thread #1 - timer://alertstrigtimer] INFO route1 - *******************************Job-Alert-System: Started: alertstrigtimer******************************
[Camel (camel-1) thread #2 - timer://alertstrigtimer] INFO loadfile - *******************************Job-Alert-System: Started: direct:loadFile******************************
[Camel (camel-1) thread #1 - file://null] INFO loadfile - *******************************Job-Alert-System: Started: direct:loadFile******************************

The problem could be cause by this line .from(getTriggerFileURI(getWorkFilePath(), getWorkFileName())) in loadfile route. Route with multiple from endpoint is known as Multiple Input and this pattern is removed in Camel 3.x.
From RedHat,
from("URI1").from("URI2").from("URI3").to("DestinationUri");
..., exchanges from each of the input endpoints,
URI1, URI2, and URI3, are processed independently of each other and in
separate threads. In fact, you can think of the preceding route as
being equivalent to the following three separate routes:
from("URI1").to("DestinationUri");
from("URI2").to("DestinationUri");
from("URI3").to("DestinationUri");
Rather than using multiple from endpoint (extra independent input), try content enricher pattern (pollEnrich for file component).

Camel process receivnig hang up singal and shutsdown gracefully

i have a Camel process which pulls file from Azure container and process it in Azure environment.
I expect the process to run continuously, but it shuts down after random interval.
Logs:
CamelHangupInterceptor: INFO (MainSupport.java:87) - Received hang up - stopping the main instance.
CamelHangupInterceptor: DEBUG (Main.java:187) - Stopping Spring ApplicationContext: org.springframework.context.support.ClassPathXmlApplicationContext#47ef968d
2019-01-29 21:39:50,782: main: INFO (MainSupport.java:502) - MainSupport exiting code: 0
...
closes all the routes after inflight exchange is compeleted since using DefaultShutdownStrategy.
Spring context route:
- Start with a scheduler for initial delay,
- then <dealy> component to randomly generate time (logic used for scalability to avoid race condition)
- invoking the custom class implementing Process class, which has Azure container url with credentials and fetch the file from container
- then using wireTap component to downloading the file
- finally invoking another class implementing Process class.
The Camel (v 2.20) process starts and executes as expected, but after a random interval process shuts down.
I see hangup signal received, but not sure how it happens. Logs shows graceful shutdown. Is there a way to send hangup signal to Camel process from external process?
in one of the route i am using to stop the exchange to stop the route forcibly.
exchange.setProperty(Exchange.ROUTE_STOP, Boolean.TRUE);
More logs:
2019-01-29 21:40:17,838: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultExecutorServiceManager.java:363) - Forcing shutdown of ExecutorService: org.apache.camel.util.concurrent.RejectableThreadPoolExecutor#647b9364[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0][WireTap]
...
2019-01-29 21:40:17,840: Camel Thread #0 - CamelHangupInterceptor: DEBUG (RouteService.java:289) - Shutting down services on route: route1
2019-01-29 21:40:17,841: Camel Thread #0 - CamelHangupInterceptor: DEBUG (BeanComponent.java:72) - Clearing BeanInfo cache[size=1, hits=1, misses=1, evicted=0]
2019-01-29 21:40:17,849: Camel Thread #0 - CamelHangupInterceptor: DEBUG (SimpleLanguage.java:136) - Clearing simple language predicate cache[size=0, hits=0, misses=0, evicted=0]
2019-01-29 21:40:17,849: Camel Thread #0 - CamelHangupInterceptor: DEBUG (SimpleLanguage.java:142) - Clearing simple language expression cache[size=4, hits=1, misses=4, evicted=0]
2019-01-29 21:40:17,849: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultManagementAgent.java:358) - Unregistered MBean with ObjectName: org.apache.camel:context=camelAzureBlobContext,type=routecontroller,name="camelAzureBlobContext"
2019-01-29 21:40:17,849: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultManagementAgent.java:358) - Unregistered MBean with ObjectName: org.apache.camel:context=camelAzureBlobContext,type=health,name="camelAzureBlobContext"
2019-01-29 21:40:17,850: Camel Thread #0 - CamelHangupInterceptor: DEBUG (TimerListenerManager.java:128) - Removed TimerListener: org.apache.camel.management.mbean.ManagedCamelContext#473692b
2019-01-29 21:40:17,850: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultManagementAgent.java:358) - Unregistered MBean with ObjectName: org.apache.camel:context=camelAzureBlobContext,type=context,name="camelAzureBlobContext"
2019-01-29 21:40:17,850: Camel Thread #0 - CamelHangupInterceptor: INFO (MainLifecycleStrategy.java:44) - CamelContext: camelAzureBlobContext has been shutdown, triggering shutdown of the JVM.
2019-01-29 21:40:17,850: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultExecutorServiceManager.java:363) - Forcing shutdown of ExecutorService: org.apache.camel.util.concurrent.RejectableThreadPoolExecutor#72b68833[Running, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1][ShutdownTask]
2019-01-29 21:40:17,851: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultInflightRepository.java:183) - Shutting down with no inflight exchanges.
2019-01-29 21:40:17,851: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultServicePool.java:110) - Stopping service pool: org.apache.camel.impl.SharedPollingConsumerServicePool#13ed066e
2019-01-29 21:40:17,851: Camel Thread #0 - CamelHangupInterceptor: DEBUG (DefaultServicePool.java:110) - Stopping service pool: org.apache.camel.impl.SharedProducerServicePool#151ab2b9
...
2019-01-29 21:40:17,856: Camel Thread #0 - CamelHangupInterceptor: DEBUG (MBeanInfoAssembler.java:79) - Clearing cache[size=30, hits=12, misses=30, evicted=0]
2019-01-29 21:40:17,864: Camel Thread #0 - CamelHangupInterceptor: DEBUG (IntrospectionSupport.java:134) - Clearing cache[size=93, hits=192, misses=93, evicted=0]
2019-01-29 21:40:17,869: Camel Thread #0 - CamelHangupInterceptor: INFO (DefaultCamelContext.java:3575) - Apache Camel 2.20.0 (CamelContext: camelAzureBlobContext) uptime 32 minutes
-- ROUTE INFORMATION--
<camel:camelContext id="camelAzureBlobContext" xmlns="http://camel.apache.org/schema/spring" typeConverterStatisticsEnabled="true" autoStartup="true">
<endpoint id="listBlobendpoint"
uri="azure-blob://storageaccount/containerName?credentials=containercredientiasl&operation=listBlobs" /> <!-- changed the actual values -->
<dataFormats>
<json id="inputMsg" library="Jackson" unmarshalTypeName="pacakage.requiredinputpojojacksonclass" /> <!-- renamed the class name-->
</dataFormats>
<onException>
<exception>com.fasterxml.jackson.core.JsonParseException</exception>
<exception>com.fasterxml.jackson.databind.JsonMappingException</exception>
<handled>
<constant>true</constant>
</handled>
<process ref="parseExceptionResponse" />
</onException>
<route>
<from uri="scheduler://tempScheduler?initialDelay=5000&delay=50000" /> <!-- changed the actual values -->
<setHeader headerName="BlobListingDetails">
<simple resultType="com.microsoft.azure.storage.blob.BlobListingDetails">METADATA</simple>
</setHeader>
<delay>
<method ref="blobcamelprocess" method="randomDelayToPoll"></method> <!-- Method which has some random number generation-->
</delay>
<to uri="ref:listBlobendpoint" /> <!-- bean which actually sets the metadata value-->
<process ref="blobcamelprocess" />
<!-- creating recipient list to update metadata of the container blob file -->
<recipientList>
<header>update_metadata</header>
</recipientList>
<log message="Message | $simple{in.header[filename]}" loggingLevel="INFO"></log>
<wireTap uri="file://location?fileName=$simple{in.header[filename]}"/>
<unmarshal ref="inputMsg" />
<process ref="messageconversionprocess" /> <!-- bean which actually converts uses the parsed json to construct java object-->
<process ref="deleteblobProcess" /> <!-- bean that will be used to delete the file from the blob store -->
<recipientList>
<header>delete_blob</header> <!-- endpoint details is set from the above bean and passed here -->
</recipientList>
</route>
</camel:camelContext>

As per the logs, your whole Camel context is shutting down. Its quite unlikely that the context shutdown is initiated by your ROUTE_STOP call.
Looking at MainSupport.java source, the INFO log that you see in your log files is executed as a part of hang up notification hook, notified by a JVM Shutdown in progress. When this happens, does your JVM die as well? This documentation says hooks are notified when a JVM Abnormal termination is initiated by SIGINT, SIGTERM or SIGHUP
Perhaps we can look into further details if you could provide answers to the following
Is Camel running in a web container or something else? I see MainSupport.java :)
What JVM version do you use?
What OS platform is it running on?
When Camel context shuts down, is the JVM still running or JVM exits too?
Do you see additional exceptions in the log files?
Can we correlate this CamelContext shutdown to a system resource exhaustion event? (like running short of memory or something like that). Quite unlikely to be the OOMKiller since its is a graceful shutdown.
On a side note, don't be misled by the name CamelHangupInterceptor it doesn't mean Camel received a SIGHUP.

Tomcat 9 Valve Causing Server Start Failure

I have written a custom tomcat valve to parse HTTP headers and use them to authenticate. The valve works by extending AuthenticatorBase. I compiled it and placed it in $CATALINA_HOME/lib. Here is code:
public class TomcatLogin extends AuthenticatorBase {
private String[] roleNames = null;
private String ivcreds = null;
private String ivuser = null;
private String ivgroups = null;
private GenericPrincipal principal = null;
// Constructor defers to super class (Authenticator base)
public TomcatLogin(){
super();
}
protected boolean doAuthenticate(Request request, HttpServletResponse response)
{
List<String> groupsList = null;
System.out.println("Obtaining Headers from Request");
try {
ivuser = request.getHeader("iv-user");
ivcreds = request.getHeader("iv-creds");
ivgroups = request.getHeader("iv-groups");
} catch(Exception e) {
e.printStackTrace();
}
// Require all header credentials for proper authentication
if(ivuser == null || ivcreds == null || ivgroups == null)
return false;
// Split ivgroups by comma seporated value
// Then remove head and tail quotation marks
roleNames = ivgroups.split(",");
for(int i=0; i<roleNames.length; i++){
roleNames[i] = roleNames[i].substring(1, roleNames[i].length()-1 );
groupsList.add(roleNames[i]);
}
principal = new GenericPrincipal(ivuser, ivcreds, groupsList);
request.setUserPrincipal(principal);
return true;
}
public String getAuthMethod() {
return "HTTPAuthenticator";
}
}
I then tell Tomcat to use the valve in the server.xml. The documentation for extending AuthenticatorBase says When this class is utilized, the Context to which it is attached (or a parent Container in a hierarchy) must have an associated Realm that can be used for authenticating users and enumerating the roles to which they have been assigned. I thought I had configured it correctly, but it throws and error and Tomcat fails to start. Here is the server.xml config:
<?xml version="1.0" encoding="UTF-8"?>
...
<Server>
<Service name="Catalina">
<Engine name="Catalina" defaultHost="localhost">
<Realm className="org.apache.catalina.realm.LockOutRealm">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>
<!-- Here is my valve -->
<Valve className="package.of.my.custom.valve.TomcatLogin" />
<Host ... >
...
</Host>
</Engine>
</Service>
</Server>
And here is the Error message I am getting:
10-Jan-2019 10:11:03.576 SEVERE [main] org.apache.tomcat.util.digester.Digester.endElement End event threw exception
java.lang.reflect.InvocationTargetException
... A bunch of useless stacktrace
Caused by: java.lang.IllegalArgumentException: Configuration error: Must be attached to a Context at org.apache.catalina.authenticator.AuthenticatorBase.setContainer(AuthenticatorBase.java:278)
at org.apache.catalina.core.StandardPipeline.addValve(StandardPipeline.java:335)
at org.apache.catalina.core.ContainerBase.addValve(ContainerBase.java:1133)
... 27 more
10-Jan-2019 10:11:03.579 WARNING [main] org.apache.catalina.startup.Catalina.load Catalina.start using conf/server.xml: Error at (107, 82) : Configuration error: Must be attached to a Context
10-Jan-2019 10:11:03.579 SEVERE [main] org.apache.catalina.startup.Catalina.start Cannot start server. Server instance is not configured.
I think my valve is written correctly, so my guess is that the issue is in the configuration. Not sure why it is not getting a context to attach to. Any ideas?
Edit:
I tried putting the valve in my app's META-INF/context.xml (I had to make one since there wasn't one to begin with). Here it is:
<?xml version="1.0"?>
<Context>
<Valve className="package.of.my.custom.valve.TomcatLogin" />
</Context>
The server then start, but it fails to deploy any of the applications. I am getting similar error to an IBM valve which I originally tried to use over this custom implementation where it cannot find the AuthenticatorBase class from catalina.jar. Here are the SEVERE errors I am getting:
10-Jan-2019 15:34:06.673 SEVERE [main] org.apache.catalina.core.ContainerBase.addChildInternal ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/sample-DB]]
...
Stacktrace info
...
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:441)
Caused by: java.lang.ExceptionInInitializerError
at org.apache.tomcat.util.digester.Digester.startDocument(Digester.java:1102)
Caused by: java.lang.NullPointerException
at java.nio.charset.Charset.put(Charset.java:538)
...
Stacktrace
...
10-Jan-2019 15:34:06.688 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/fmac/deploy/sample.war] has finished in [11] ms
10-Jan-2019 15:34:06.689 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/fmac/deploy/sample-auth.war]
10-Jan-2019 15:34:06.692 SEVERE [main] org.apache.tomcat.util.digester.Digester.startElement Begin event threw error
java.lang.NoClassDefFoundError: org/apache/catalina/authenticator/AuthenticatorBase
...
Stacktrace
The Error Below is the most confusing one. How can it not find this class?
...
Caused by: java.lang.ClassNotFoundException: org.apache.catalina.authenticator.AuthenticatorBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
...
Stacktrace
...
10-Jan-2019 15:34:13.823 SEVERE [https-jsse-nio2-8443-exec-3] org.apache.coyote.http11.Http11Processor.service Error processing request
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.tomcat.util.buf.B2CConverter
at org.apache.catalina.connector.CoyoteAdapter.convertURI(CoyoteAdapter.java:1072)

Put your <Valve> inside of your <Context> element. That should usually be within a META-INF/context.xml file in your web application and not in conf/server.xml.

So I was finally able to find out what the issues were. We have a custom tomcat implementation, part of which involves appending the classpath in the java command. For some reason, some of the database drivers that were being used caused the Tomcat to fail to find any of the libraries in CATALINA_HOME/lib. I'm running inside a Docker container and this was some old vestigial stuff from a VM version. We ended up just having to toss those
drivers out.
Not really sure why they would completely override the base lib/ directory, but at least these errors when away and I was actually able to use the pre-built authenticator I had instead of fine tuning this custom one.

Camel Route as a String load and Run

I have created camel route and store in database, i need to load and run.
Till now i am created code which is created route dynamically but result is not as per an expectation.
I have downloaded example of CXF and run using mvn camel:run it was run and as result it was calling web service, but now i need a create java project and call, I have camel Route in String variable. so load route which is store as String, and call web service.
Code For Same:
public static void main(String[] args) throws Exception {
DefaultCamelContext context = new DefaultCamelContext();
// append the routes to the context
String myString = "<route id=\"my_Sample_Camel_Route_with_CXF\" xmlns=\"http://camel.apache.org/schema/spring\">"
+ " <from uri=\"file:src/data?noop=true\"/>"
+ " <log loggingLevel=\"INFO\" message=\">>> ${body}\"/>"
+ " <to uri=\"cxf://http://www.webservicex.net/stockquote.asmx?wsdlURL=src/wsdl/stockquote.wsdl&serviceName={http://www.webserviceX.NET/}StockQuote&portName={http://www.webserviceX.NET/}StockQuoteSoap&dataFormat=MESSAGE\"/>"
+ " <log loggingLevel=\"INFO\" message=\">>> ${body}\"/>"
+ " </route>";
;
InputStream is = new ByteArrayInputStream(myString.getBytes());
RoutesDefinition routes = context.loadRoutesDefinition(is);
context.addRouteDefinitions(routes.getRoutes());
context.setTracing(true);
// at the end start the camel context
context.start();
Thread.sleep(3000);
context.stop();
System.out.println("Done");
/***
* Jars are
*
aopalliance-1.0.jar
asm-3.3.jar
camel-core-2.6.0-fuse-00-00.jar
camel-cxf-2.6.0-fuse-00-00.jar
camel-http-2.6.0-fuse-00-00.jar
camel-spring-2.6.0-fuse-00-00.jar
commons-codec-1.2.jar
commons-httpclient-3.1.jar
commons-logging-1.1.1.jar
commons-logging-api-1.1.jar
commons-management-1.0.jar
cxf-api-2.3.2-fuse-00-00.jar
cxf-common-schemas-2.3.2-fuse-00-00.jar
cxf-common-utilities-2.3.2-fuse-00-00.jar
cxf-rt-bindings-soap-2.3.2-fuse-00-00.jar
cxf-rt-bindings-xml-2.3.2-fuse-00-00.jar
cxf-rt-core-2.3.2-fuse-00-00.jar
cxf-rt-databinding-jaxb-2.3.2-fuse-00-00.jar
cxf-rt-frontend-jaxrs-2.3.2-fuse-00-00.jar
cxf-rt-frontend-jaxws-2.3.2-fuse-00-00.jar
cxf-rt-frontend-simple-2.3.2-fuse-00-00.jar
cxf-rt-transports-common-2.3.2-fuse-00-00.jar
cxf-rt-transports-http-2.3.2-fuse-00-00.jar
cxf-rt-ws-addr-2.3.2-fuse-00-00.jar
cxf-tools-common-2.3.2-fuse-00-00.jar
geronimo-javamail_1.4_spec-1.7.1.jar
geronimo-servlet_2.4_spec-1.1.1.jar
jaxb-impl-2.1.13.jar
jettison-1.2.jar
jsr311-api-1.1.1.jar
log4j-1.2.16.jar
neethi-2.0.4.jar
spring-aop-3.0.5.RELEASE.jar
spring-asm-3.0.5.RELEASE.jar
spring-beans-3.0.5.RELEASE.jar
spring-context-3.0.5.RELEASE.jar
spring-core-3.0.5.RELEASE.jar
spring-expression-3.0.5.RELEASE.jar
spring-tx-3.0.5.RELEASE.jar
spring-web-3.0.5.RELEASE.jar
stax2-api-3.0.2.jar
woodstox-core-asl-4.0.8.jar
wsdl4j-1.6.2.jar
xml-resolver-1.2.jar
XmlSchema-1.4.7.jar
*/
}
After running same program i am getting out like
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
log4j:WARN No appenders could be found for logger (org.apache.camel.impl.DefaultCamelContext).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jun 11, 2015 12:36:19 PM org.apache.cxf.service.factory.ReflectionServiceFactoryBean buildServiceFromWSDL
INFO: Creating Service {http://www.webserviceX.NET/}StockQuote from WSDL: src/wsdl/stockquote.wsdl
Done

Representing a route list from xml in camel DSL

How can I represent this route in Camel's DSL:
<camel:camelContext id="camel-context">
<camel:route id="conductor-event" trace="true">
<camel:from uri="direct:conductor/event"/>
<camel:log message="handling conductor-event: id=${exchangeId}"/>
<!-- execute each filter in sorted order -->
<camel:bean ref="beaner.BProcessors"/>
<camel:log message="after: [bprocessors]: id=${exchangeId}"/>
<!-- map the event to a route -->
<camel:recipientList parallelProcessing="false">
<camel:method ref="beaner.Mappings" />
</camel:recipientList>
<camel:log message="after event mapping: id=${exchangeId}"/>
</camel:route>
</camel:camelContext>
I have this so far, but I get a "Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 0: %7BCamelToEndpoint=...":
RouteDefinition routeDef = from("direct:conductor/event")
.log( "handling conductor-event: id=${exchangeId}" )
.beanRef( "beaner.BProcessors" )
.log( "after: [bprocessors]: id=${exchangeId}" );
ExpressionClause<RecipientListDefinition<RouteDefinition>> recipientList = routeDef.recipientList();
recipientList.properties().setParallelProcessing( false );
recipientList.method( "beaner.EventMappings" );
routeDef.log( "after event mapping: id=${exchangeId}" );

here is the route in JavaDSL...note that the recipientList parallelProcessing is false by default...
from("direct:conductor/event")
.log("handling conductor-event: id=${exchangeId}")
.beanRef("beaner.BProcessors")
.log("after: [bprocessors]: id=${exchangeId}")
.recipientList(bean("beaner.Mappings"))
.log("after event mapping: id=${exchangeId}");

You should use a RouteBuilder class in Java DSL to access the DSL.
Then inside the configure method you can build the routes almost identical as in XML DSL.
See the getting started guide here: http://camel.apache.org/walk-through-an-example.html