I've got an application leaking out java heap at a decent rate (400 users leaves 25% free after 2hours...after logoff all memory is restored) and we've identified the items causing the memory leak as Strings placed in session that appear to be generated by Portal itself. The values are the encoded Portal URIs (very long endcoded strings ... usually sized around 19kb), and the keys seem to be seven (7) randomly generated characters prefixed by RES# (for example, RES#NhhEY37).
We've stepped through the application using session tracing and snapping off heapdumps which has resulted in determining that there is one of these objects created and added to session on almost every page ... in fact, it seems like it is on each page that submits data (which is most pages). So, it's either 1:1 with pages in general, or 1:1 with forms.
Has anyone encountered a similar problem as this? We are opening a ticket with IBM, but wanted to ask this community as well. Thanks in advance!
Can it be the portlet cache? You could have servlet caching activated and declare a long portlet expiry time. Quoting from techjournal:
Portlets can advertise their ability to be cached in the fragment cache by setting their expiry time in their portlet.xml descriptor (see Portlet descriptor example)
<!-Expiration value is in seconds, -1 = no time limit, 0 = deactivated-->
<expiration-cache>3600</expiration-cache> <!- 1 Hour cache -->
To use the fragment caching functions, servlet caching needs to be activated in the Web Container section of WebSphere Application Server administrative console (see Portlet descriptor example). WebSphere Application Server also provides also a cache monitor enterprise application (CacheMonitor.ear), which is very useful for visualizing the contents of the fragment cache.
Update
Do you have portlets that set EXPIRATION_CACHE? Quote:
Modifying the local cache at runtime
For standard portlets, the portlet window can modify the expiration time at runtime by setting the EXPIRATION_CACHE property in the RenderResponse, as follows:
RenderResponse.setProperty(
PortletResponse.EXPIRATION_CACHE,
(new Integer(3000)).toString() );
Note that for me the value is a bit counter-intuitive, -1 means never expire, 0 means don't cache.
The actual issue turned out to be a working feature within Portal. Specifically, Portal's action protection which prevents the same action from being submitted twice, while keeping the navigational ability of the portal. There is a cache that retains the actions results for every successful action and uses them to compare and reject duplicates.
The issue for us was the fact that we required "longer than normal" user sessions (60+ minutes) and with 1,000+ concurrent users, we leaked out on this protection mechanism after just a couple hours.
IBM recommended that we just shut off the cache entirely using the following portlet.xml configuration entry:
wps.multiple.action.execution = true
This allows double submits, which may or may not harm business functionality. However, our internal Portal framework already contained a mechanism to prevent double submits, so this was not an issue for us.
At our request, IBM did come back with a patch for this issue which makes the cache customizeable, that is, let's you configure the number of action results that you store in cache for each user and thus you can leverage Portal's mechanism again, at a reduced session overhead. Those portal configuration settings were:
wps.multiple.action.cache.bound.enabled = true
wps.multiple.action.cache.key.maxsize = 40
wps.multiple.action.cache.value.maxsize = 10
You'll need to contact IBM about this patch as it is not currently in a released fixpack.
Is your Websphere Portal Server having latest fix pack installed?
http://www-01.ibm.com/support/docview.wss?uid=swg24024380&rs=0&cs=utf-8&context=SSHRKX&dc=D420&loc=en_US&lang=en&cc=US
Also you may be interested in following discussion
http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14427700&tstart=0
Update:
Just throwing some blind folded darts.
"RES#" to me sounds like resource.
From the forum stack trace,
"DefaultActionResultManager.storeDocument"
indicates it is storing the document.
Hence looks like your resources(generated portal pages) are being cached. Check if there is some paramater that can lmit cache size of resource.
Also in another test set cache expiration to 5 minutes instead of an hour.
Related
I'm trying to connect Snowflake to Spring boot based application. Everything works well however due to some crazy defaults in snowflake, I'm having to deal with the below error:
https://community.snowflake.com/s/article/SAP-BW-Java-lang-NoClassDefFoundError-for-Apache-arrow
I can fire the queries via Spring data JPA but not able to map the results at all. The solution asks for changing the default resultset format from ARROW to JSON:
Customer can use this property (jdbc_query_result_format=json) in datasouce property of Application server or session property in application like
Statement = connection.createStatement();
Statement.executeQuery("ALTER SESSION SET JDBC_QUERY_RESULT_FORMAT='JSON'");
I'm just wondering how could I achieve that with Spring ie. run the above SQL in the bootstrap period. I'm using Spring data JPA queries.
The parameter JDBC_QUERY_RESULT_FORMAT can be set per account/user/session, therefore if you always want to have JSON for that user you can set it directly via Snowflake UI only once:
ALTER USER <youruser> SET JDBC_QUERY_RESULT_FORMAT='JSON';
This would only impact your user and the account would still have ARROW as default. And you wouldn't need to set it at session level all the time.
It turns out that the original issue can be fixed easily by tweaking some memory settings in your applications.
Summarising the solution below:
Increase the heap memory and setting -XX:MaxDirectMemorySize > 64M by setting the JAVA_OPTS property for your application.
The rationale behind this is that Arrow format inherently make use of the available heap memory in order to optimise the large results. Not having enough heap space will lead to the above mentioned error.
I still need to perform more tests to find out the ideal values for heap memory, stack size, and this variable → -XX:MaxDirectMemorySize.
EDIT: Below settings worked:
"JAVA_OPTS": "-Xss230k -XX:ReservedCodeCacheSize=100m -Xmx1800m -XX:MaxDirectMemorySize=1500m"
6 instances of the app running in parallel.
2GB RAM per instance ie. 12GB RAM in total (probably an overkill but it snowflake driver does eat up a lot of ram in general)
we have a complex infrastructure with WebSEAL, Websphere Portal and a couple of Wesphere AS' where we [obviously] want to set up SSO. we successfully configured TAI++ etc, but unfortunately default LTPA2 token name ("LtpaToken2") is not acceptable for compatibility reasons and we want to change it. Websphere AS 8.5, that we're using, has a special option in "General security -> Single Sign-on" to do so, but that doesn't work however; no matter what we enter, the token issued is still named "LtpaToken2" (well, yes we restarted the AS apparently ;P).
is it something known around websphere community? afaiu this can be customized by implementing a token factory and a token interceptor/validator (i.e. as is suggested here: Generate LTPAToken 2 in custom Web Application), but i'd expect the edit-box that supposedly does so to actually work.
BTW, i take it that renaming LTPA2 tokens is not at all possible in earlier Webspheres?
It is definitely possible since v8. I've tested it on 8.5.5.1 and works fine (I can set ltpa cookie to any name). Verify your settings accessing directly to WAS for example - in admin console, not via TAM. Your problem might be related to WebSEAL and Tai.
I don't quite understand what you mean that ltpatoken2 is not acceptable for compatibility reasons? It was always like that, so changing it will rather introduce incompatibilities, than keeping the default.
I would like to totally disable session creation and management in my web app to eradicate the memory (and other resource) usage currently associated with Tomcat's standard session manager. This includes disabling sesison cookies and/or url rewriting as, if I'm succesful, there will be no sessions to track.
My web application has a single servlet that passes the xml it receives to an API/engine. This engine can run inside or outside a servlet container and it creates, tracks and manages sessions in its own way. I have zero need for the sessions in Tomcat and I'd like to reduce to the barest minimum the resources Tomcat uses for session management.
I ran some searches on the topic. The searches came up with some topics, including some on this website. It appears that the tightest way to address this issue is to create your own Manager implementation that, bascially, provides an 'empty' implementation that does the barest minimum. (There were some alternative suggestions but I found them to be relatively weak. These suggestions included "just don't call getSession()", and "set the 'cookies' attribute of a context to false". I think implementing a session manager that does what I want is better than these suggestions and it is the path I have elected to go down.)
Given this information, that rolling your own session manager is a good way to go, I then downloaded the Tomcat source code to take a look at code related to a Manager implementation. It all looks doable but it looks like a few hours work for me to come up with my attempt at a sesssion manager. Before committing to that path and the work involved, I thought I'd put it out there - Does anyone have a minimal session manager implementation for Tomcat they can share? One that does nothing would be best, but I'll take anything including tips and battle stories from anyone who has written their own session manager. I am working with Tomcat 6.
My Apache Wicket web application uses JDO for its data persistence in GAE/J.
On application start-up, the home page enqueues a task before it is shown (with zero delay to its default ETA). This task causes the construction of a new Wicket web page, in order to construct the JVM's singleton Persistence Manager Factory (PMF) instance for use by the application during its lifetime.
I have set the application to use concurrent requests by adding
<threadsafe>true</threadsafe>
to the application's appengine-web.xml file.
Despite this, after a single request to visit the application's home page, I get two application instances: one created by the home page visit request, and the other created by the execution of the enqueued task (about 6 to 7 seconds later).
I could try to solve this problem by delaying the execution of the enqueued task (by around 10 seconds, perhaps?), but why should I need to try this when I have enabled concurrent requests? Should the first GAE/J application instance not be able to handle two requests close together without causing a second instance to be brought forth? I presume that I am doing something wrong, but what is it?
I have searched Stack Overflow's set of tags ([google-app-engine] [java]), and the depreciating group "Google App Engine for Java" too, but have found nothing relevant to my question.
I would appreciate any pointers.
If you want the task to use an existing instance, you can set the X-AppEngine-FailFast header, which according to the GAE docs:
This header instructs the Scheduler to immediately fail the request if an existing instance is not available. The Task Queue will retry and back-off until an existing instance becomes available to service the request
It's worth checking out the Managing Your App's Resource Usage document for performance and tuning techniques.
We have a web-based application with tech stack -
1. Java Struts based
2. Hibernate
3. DB - Oracle
4. App server - JBoss server
We are facing an issue related to concurrent usage of the application by two or more users. When I am doing an operation and I submit the changes, the next page or success message that comes up is of a different operation that another user is performing at the same time.
Users are logged in as different users and so are using different sessions.
We have no clue of where the problem is, so I am not sure what other details I can provide.
Has anyone else faced such an issue or any pointers?
Are you using application context instead of session context? Moreover, as Eed3si9n said, beware of Singletons, that might be causing this.
"In addition check for the use of static fields. One app I was brought in to fix used a static string for error message. As soon as any user received an error they all did. Worked fine until there wasmore than one concurrent user." – Michael Rutherfurd (posted it as a comment)
I am not familiar with specific libraries you are using, but let me try.
How stateless are your application code? Do you have any sort of global state like singleton with member fields? If the service is stateful and are using singleton, you could have such mixups.
Check if the form is defined as application scope and the message you showing on the screen is coming from that form.