We Keep Coding

Cluster instability with TCPPING protocol

I have 8 different processes distributed across 6 different servers with the following TCP/TCPPING protocol configuration:
<config xmlns="urn:org:jgroups" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP
bind_port="${jgroups.tcp.bind_port:16484}"
bind_addr="${jgroups.tcp.bind_addr:127.0.0.1}"
recv_buf_size="20M"
send_buf_size="20M"
max_bundle_size="64K"
sock_conn_timeout="300"
use_fork_join_pool="true"
thread_pool.min_threads="10"
thread_pool.max_threads="100"
thread_pool.keep_alive_time="30000" />
<TCPPING
async_discovery="true"
initial_hosts="${jgroups.tcpping.initial_hosts:127.0.0.1[16484]}"
port_range="5" #/>
<MERGE3 min_interval="10000" max_interval="30000" />
<FD_SOCK get_cache_timeout="10000"
cache_max_elements="300"
cache_max_age="60000"
suspect_msg_interval="10000"
num_tries="10"
sock_conn_timeout="10000"/>
<FD timeout="10000" max_tries="10" />
<VERIFY_SUSPECT timeout="10000" num_msgs="5"/>
<BARRIER />
<pbcast.NAKACK2
max_rebroadcast_timeout="5000"
use_mcast_xmit="false"
discard_delivered_msgs="true" />
<UNICAST3 />
<pbcast.STABLE
stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="4M" />
<AUTH
auth_class="com.qfs.distribution.security.impl.CustomAuthToken"
auth_value="distribution_password"
token_hash="SHA" />
<pbcast.GMS
print_local_addr="true"
join_timeout="10000"
leave_timeout="10000"
merge_timeout="10000"
num_prev_mbrs="200"
view_ack_collection_timeout="10000"/>
</config>
The cluster keeps splitting in subgroups, then merges again and again which results in high memory usages. I can also see in the logs a lot of "suspect" warning resulting from the periodic heartbeats sent by all other cluster members. Am I missing something ?
EDIT
After enabling gc logs, nothing suspect appeared to me. On the other hand, I've noticed this jgroups logs appearing a lot:
WARN: lonlx21440_FrtbQueryCube_QUERY_29302: I was suspected by woklxp00330_Sba-master_DATA_36219; ignoring the SUSPECT message and sending back a HEARTBEAT_ACK
DEBUG: lonlx21440_FrtbQueryCube_QUERY_29302: closing expired connection for redlxp00599_Sba-master_DATA_18899 (121206 ms old) in send_table
DEBUG: I (redlxp00599_Sba-master_DATA_18899) will be the merge leader
DEBUG: redlxp00599_Sba-master_DATA_18899: heartbeat missing from lonlx21503_Sba-master_DATA_2175 (number=1)
DEBUG: redlxp00599_Sba-master_DATA_18899: suspecting [lonlx21440_FrtbQueryCube_QUERY_29302]
DEBUG: lonlx21440_FrtbQueryCube_QUERY_29302: removed woklxp00330_Sba-master_DATA_36219 from xmit_table (not member anymore)enter code here
and this one
2020-08-31 16:35:34.715 [ForkJoinPool-3-worker-11] org.jgroups.protocols.pbcast.GMS:116
WARN: lonlx21440_FrtbQueryCube_QUERY_29302: failed to collect all ACKs (expected=6) for view [redlxp00599_Sba-master_DATA_18899|104] after 2000ms, missing 6 ACKs from (6) lonlx21503_Sba-master_DATA_2175, lonlx11179_DRC-master_DATA_15999, lonlx11184_Rrao-master_DATA_31760, lonlx11179_Rrao-master_DATA_25194, woklxp00330_Sba-master_DATA_36219, lonlx11184_DRC-master_DATA_49264
I still can;'t figure out where the instability comes from.
Thanks

Any instability is not due to TCPPING protocol - this belongs to the Discovery protocol family and its purpose is to find new members, not kick them out of the cluster.
You use both FD_SOCK and FD to find if members left, and then VERIFY_SUSPECT to confirm that the node is not reachable. The setting seems pretty normal.
First thing to check is your GC logs. If you experience STW pauses longer than, say, 15 seconds, chances are that the cluster disconnect because of unresponsiviness due to GC.
If your GC logs are fine, increase logging level for FD, FD_SOCK and VERIFY_SUSPECT to TRACE and see what's going on.

NPE while trying to render a View in Spring Webflow

When trying to use a webflow, I keep getting the following error:
{
"failure":"true",
"exception.message":"org.springframework.webflow.execution.ActionExecutionException: Exception thrown executing org.springframework.webflow.action.ViewFactoryActionAdapter#200a5beb in state 'accountCannotLogInView' of flow 'login' -- action execution attributes were 'map[[empty]]'",
"exception.stacktrace":"org.springframework.webflow.execution.ActionExecutionException: Exception thrown executing org.springframework.webflow.action.ViewFactoryActionAdapter#200a5beb in state 'accountCannotLogInView' of flow 'login' -- action execution attributes were 'map[[empty]]'
...
Caused by: java.lang.IllegalStateException: Exception occurred rendering view null
at org.springframework.webflow.mvc.view.AbstractMvcView.render(AbstractMvcView.java:191)
at org.springframework.webflow.action.ViewFactoryActionAdapter.doExecute(ViewFactoryActionAdapter.java:40)
at org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188)
at org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51)
... 66 more
Caused by: java.lang.NullPointerException
at org.springframework.webflow.mvc.servlet.ServletMvcView.doRender(ServletMvcView.java:50)
at org.springframework.webflow.mvc.view.AbstractMvcView.render(AbstractMvcView.java:187)
... 69 more
"
}
Does anybody have any idea what the cause of this might be?
From my login-webflow.xml:
<action-state id="checkAccountType">
<evaluate expression="lookupAccountTypeAction.lookup(flowScope.principal)" />
<transition on="EMAIL" to="accountCannotLogInView" />
<transition on="Login" to="checkAccountStatus" />
<transition on="CONTACT" to="accountCannotLogInView" />
</action-state>
<end-state id="accountCannotLogInView" view="myAccountCannotLogInView"/>
From default_views.properties:
myAccountCannotLogInView.(class)=org.springframework.web.servlet.view.JstlView
myAccountCannotLogInView.url=/WEB-INF/view/jsp/default/ui/myAccountCannotLogInView.jsp
From my log file:
2013-09-09 08:29:25,645 DEBUG [org.springframework.webflow.engine.EndState] - Entering state 'accountCannotLogInView' of flow 'login'
2013-09-09 08:29:25,645 DEBUG [org.springframework.webflow.execution.ActionExecutor] - Executing org.springframework.webflow.action.ViewFactoryActionAdapter#359d136a
2013-09-09 08:29:25,645 DEBUG [org.springframework.webflow.mvc.view.AbstractMvcView] - Rendering MVC [null] with model map [{flowRequestContext=[RequestControlContextImpl#55bf756b externalContext = org.springframework.webflow.mvc.servlet.MvcExternalContext#2bb4d74, currentEvent = EMAIL, requestScope = map['ticketGrantingTicketId' -> 'TGT-1-GxALVR7PEtbagbnRlStOTbHoRHlb61YVm1m2hvWx3pgWCEXgPb-cas01.example.org'], attributes = map[[empty]], messageContext = [DefaultMessageContext#cb5efc8 sourceMessages = map[[null] -> list[[empty]]]], flowExecution = [FlowExecutionImpl#3f5c0d33 flow = 'login', flowSessions = list[[FlowSessionImpl#419d87bc flow = 'login', state = 'accountCannotLogInView', scope = map['principal' -> sso_test_user, 'userInfoBean' -> com.mycompany.authentication.UserInfoBean#54394557[username=<null>,firstName=<null>,lastName=<null>,emailAddress=<null>,streetAddress=<null>,city=<null>,state=<null>,zip=<null>,country=<null>], 'changePasswordBean' -> com.mycompany.authentication.ChangePasswordBean(username=null, newPassword=null, confirmNewPassword=null), 'service' -> https://my.server.com:443/casauth/facade/norenew?idp=https://my.server.com/idp/externalAuthnCallback, 'credentials' -> [username: sso_test_user], 'warnCookieValue' -> false, 'ticketGrantingTicketId' -> 'TGT-1-GxALVR7PEtbagbnRlStOTbHoRHlb61YVm1m2hvWx3pgWCEXgPb-cas01.example.org']]]]], flashScope=map[[empty]], principal=sso_test_user, currentUser=null, userInfoBean=com.mycompany.authentication.UserInfoBean#54394557[username=<null>,firstName=<null>,lastName=<null>,emailAddress=<null>,streetAddress=<null>,city=<null>,state=<null>,zip=<null>,country=<null>], changePasswordBean=com.mycompany.authentication.ChangePasswordBean(username=null, newPassword=null, confirmNewPassword=null), service=https://my.server.com:443/casauth/facade/norenew?idp=https://my.server.com/idp/externalAuthnCallback, credentials=[username: sso_test_user], flowExecutionKey=e1s1, warnCookieValue=false, flowExecutionUrl=/authentication/login?username=%5BLjava.lang.String%3B%405b65afa5&submit=%5BLjava.lang.String%3B%4070eea883&_eventId=%5BLjava.lang.String%3B%4044796a61&service=%5BLjava.lang.String%3B%407f372965&lt=%5BLjava.lang.String%3B%407e7ee722&password=%5BLjava.lang.String%3B%403d78aa0f&execution=%5BLjava.lang.String%3B%403ce4de50, ticketGrantingTicketId=TGT-1-GxALVR7PEtbagbnRlStOTbHoRHlb61YVm1m2hvWx3pgWCEXgPb-cas01.example.org}]
2013-09-09 08:29:25,645 DEBUG [org.springframework.webflow.engine.impl.FlowExecutionImpl] - Attempting to handle [org.springframework.webflow.execution.ActionExecutionException: Exception thrown executing org.springframework.webflow.action.ViewFactoryActionAdapter#359d136a in state 'accountCannotLogInView' of flow 'login' -- action execution attributes were 'map[[empty]]'] with root cause [java.lang.NullPointerException]
2013-09-09 08:29:25,645 DEBUG [org.springframework.webflow.engine.impl.FlowExecutionImpl] - Rethrowing unhandled flow execution exception
And yes, the file does exist in that location. I've been banging on this for a few days, so if anybody has any insight, I'd really appreciate it.

And the correct answer is "user error". I'm building CAS via maven. I created profiles to hold logging, and then incorrectly configured it so that the default_views.properties file wasn't in the profiles. As a result, when CAS did its overlay as part of the maven build, it included the baseline default_views.properties instead of my copy.

it seems like your property is not parsed to your flow.xml.
defining your view hardcoded should work.
see here: Specifying view identifiers

The reason for this issue is: The JSP view is not found.
In cas-servlet.xml, mention the properties file which has the basename of the custom JSP view inside the "viewResolver" bean as below:
<bean id="viewResolver"
class="org.springframework.web.servlet.view.ResourceBundleViewResolver"
p:order="0">
<property name="basenames">
<util:list>
<value>${cas.viewResolver.basename}</value>
<value>custom_view</value>
<value>protocol_views</value>
</util:list>
</property>
</bean>
And in the custom_view.properties, mention the following:
casRegisterView.(class)=org.springframework.web.servlet.view.JstlView
casRegisterView.url=/WEB-INF/view/jsp/default/ui/casRegisterView.jsp
Here the JSP view is "casRegisterView.jsp".
Place custom_view.properties in /src/main/resources.

Extracting exception thrown by unsuccessful message-processors in FirstSuccessful routing message processor

I am addressing a fail-safe scenario as below.
But in the recoveryflow I am not able to get the exception or exceptionPayload that occurred due to the failure of the subflow1 or subflow2.
Is there any way I can get the exception thrown by the unsuccessful message processors in the First-sucessful routing processor.
I have looked into the Mule FirstSuccessful class code. From the loop in it, I can see the exception is caught but not saved.
Is there a way I can extend and have my own first-successful which can provide me the exception as well?
<flow name="main_flow" >
....
....
<first-successful>
<flow-ref name="subflow_1" />
<flow-ref name="recoveryFlow1" />
</first-successful>
....
....
<first-successful>
<flow-ref name="subflow_2" />
<flow-ref name="recoveryFlow2" />
</first-successful>
....
....
</flow>
<sub-flow name="subflow_1">
....
....
<out-bound call to web-service />
....
....
</sub-flow>
<sub-flow name="subflow_2">
....
....
<out-bound call to web-service />
....
....
</sub-flow>
<sub-flow name="revoceryFlow1" >
<process communication based on the exception thrown by previous unsuccessful sub-flow 1>
<some more processing so that the remaining main flow is not impacted because of the failue of the subflow 1 >
</sub-flow>
<sub-flow name="revoceryFlow2" >
<process communication based on the exception thrown by previous unsuccessful sub-flow 2>
<some more processing so that the remaining main flow is not impacted because of the failue of the subflow 2 >
</sub-flow>
Please advise if I am going in wrong direction. Is there any better approach than mine?

I got the solution.
There is not straight solution from the Mule.
I have overwriten the FirstSuccessful behaviour and added properties with the exception details. These properties are then accessed in the next message processors inside the first-successful.
catch (Exception ex)
{
event.getMessage().setProperty("prevError", ex.getMessage() , PropertyScope.INVOCATION);
event.getMessage().setProperty("prevErrorClass", ex.getClass() , PropertyScope.INVOCATION);
failed = true;
}

How to configure 2.6 spring: Failed to create route route2 at:

I'm trying to upgrade from Camel 2.0 to 2.6
I have this in my applicationContext-camel.xml file...
<camel:route >
<camel:from uri="transactionSaleBuffer" />
<camel:policy ref="routeTransactionPolicy"/>
<camel:transacted ref="transactionManagerETL" />
<camel:to uri="detailFactProcessor" />
</camel:route>
by adding in the two lines in the middle (policy and transacted) I get the exception...
Caused by: org.apache.camel.FailedToCreateRouteException: Failed to create route route2 at: >>> From[transactionSaleBuffer] <<< in route: Route[[From[transactionSaleBuffer]] -> [Tr
ansacted[ref:trans... because of Route route2 has no output processors. You need to add outputs to the route such as to("log:foo").
I can see this is because the Camel class RouteDefinition.java makes a call to ProcessorDefinitionHelper.hasOutputs(outputs, true).
This passes in an array of one Object ([Transacted[ref:transactionManagerETL]])
This one object has one two children
[Transacted[ref:transactionManagerETL]]
CHILD-[Policy[ref:routeTransactionPolicy],
CHILD-To[detailFactProcessor]
The Policy child has no outputs, so the exception is thrown.
Yet I don't know how to add a child, my XML above matches the schema.
Maybe I'm missing something else?
My setup matches the example...Apache Camel: Book in One Page (See section: Camel 1.x - JMS Sample)
Can anyone please help me out.
Thanks!
Jeff Porter

Try as follows
<camel:route>
<camel:from uri="transactionSaleBuffer" />
<camel:transacted ref="transactionManagerETL" />
<camel:policy ref="routeTransactionPolicy">
<camel:to uri="detailFactProcessor" />
</camel:policy>
</camel:route>

Can you guess where this exception is coming from?

I've deployed a GWT app and have set up an exception reporter so that I get log entries on the server when a client throws an exception. Usually this is very helpful and I've killed a lot of bugs with this information. However, this stack trace has me stumped:
com.allen_sauer.gwt.log.server.ServerLogImplJDK14 log: Uncaught Exception:
com.google.gwt.event.shared.UmbrellaException: One or more exceptions caught, see full set in UmbrellaException#getCauses
at Unknown.EJb(StackTraceCreator.java:147)
at Unknown.SAb(StackTraceCreator.java:387)
at Unknown.eHb(Throwable.java:46)
at Unknown.KXb(com.google.gwt.dev.jjs.ast.JProgram:0)
at Unknown.AM(SimpleEventBus.java:214)
at Unknown.EM(SimpleEventBus.java:103)
at Unknown.bXb(HandlerManager.java:96)
at Unknown.ux(Widget.java:107)
at Unknown.USb(DomEvent.java:116)
at Unknown.xx(UIObject.java:529)
at Unknown.Mx(Widget.java:141)
at Unknown.ixc(DOM.java:1264)
at Unknown.anonymous(DOMImplStandard.java:187)
at Unknown.HIb(Impl.java:214)
at Unknown.anonymous(Impl.java:57)
at .(:0)
at ==================Caused by==================.(:0)
at Unknown.EJb(StackTraceCreator.java:147)
at Unknown.SAb(StackTraceCreator.java:387)
at Unknown.y0c(Throwable.java:46)
at Unknown.KZc(Number.java:192)
at Unknown.Y$c(Float.java:60)
at Unknown.wV(TextBoxBase.java:106)
at Unknown.yTb(ClickEvent.java:53)
at Unknown.AM(SimpleEventBus.java:204)
at Unknown.EM(SimpleEventBus.java:103)
at Unknown.bXb(HandlerManager.java:96)
at Unknown.ux(Widget.java:107)
at Unknown.USb(DomEvent.java:116)
at Unknown.xx(UIObject.java:529)
at Unknown.Mx(Widget.java:141)
at Unknown.ixc(DOM.java:1264)
at Unknown.anonymous(DOMImplStandard.java:187)
at Unknown.HIb(Impl.java:214)
at Unknown.anonymous(Impl.java:57)
These are all GWT files, except StackTraceCreator, which is part of gwt-log, the program reporting these exceptions to me. I've never seen this exception on a development machine, and I don't have even the beginnings of reproduction steps.
How can I start to track this down?

GWT apps are by default compiled with obfuscation turned on.
So you can either turn obfuscation off, or try to re-symbolize your stack traces, as explained in this issue report of gwt-log: http://code.google.com/p/gwt-log/issues/detail?id=38
This should probably be used best together with these options in your .gwt.xml file (I assume, you already did that):
<set-property name="compiler.emulatedStack" value="true" />
<set-configuration-property name="compiler.emulatedStack.recordLineNumbers"
value="true"/>
<set-configuration-property name="compiler.emulatedStack.recordFileNames"
value="true"/>
(see http://code.google.com/p/gwt-log/wiki/GettingStarted)