I'm trying to connect Snowflake to Spring boot based application. Everything works well however due to some crazy defaults in snowflake, I'm having to deal with the below error:
https://community.snowflake.com/s/article/SAP-BW-Java-lang-NoClassDefFoundError-for-Apache-arrow
I can fire the queries via Spring data JPA but not able to map the results at all. The solution asks for changing the default resultset format from ARROW to JSON:
Customer can use this property (jdbc_query_result_format=json) in datasouce property of Application server or session property in application like
Statement = connection.createStatement();
Statement.executeQuery("ALTER SESSION SET JDBC_QUERY_RESULT_FORMAT='JSON'");
I'm just wondering how could I achieve that with Spring ie. run the above SQL in the bootstrap period. I'm using Spring data JPA queries.
The parameter JDBC_QUERY_RESULT_FORMAT can be set per account/user/session, therefore if you always want to have JSON for that user you can set it directly via Snowflake UI only once:
ALTER USER <youruser> SET JDBC_QUERY_RESULT_FORMAT='JSON';
This would only impact your user and the account would still have ARROW as default. And you wouldn't need to set it at session level all the time.
It turns out that the original issue can be fixed easily by tweaking some memory settings in your applications.
Summarising the solution below:
Increase the heap memory and setting -XX:MaxDirectMemorySize > 64M by setting the JAVA_OPTS property for your application.
The rationale behind this is that Arrow format inherently make use of the available heap memory in order to optimise the large results. Not having enough heap space will lead to the above mentioned error.
I still need to perform more tests to find out the ideal values for heap memory, stack size, and this variable → -XX:MaxDirectMemorySize.
EDIT: Below settings worked:
"JAVA_OPTS": "-Xss230k -XX:ReservedCodeCacheSize=100m -Xmx1800m -XX:MaxDirectMemorySize=1500m"
6 instances of the app running in parallel.
2GB RAM per instance ie. 12GB RAM in total (probably an overkill but it snowflake driver does eat up a lot of ram in general)
Related
While I have managed to increase the max post size on my JBoss following
http://www.mastertheboss.com/web/jboss-web-server/configuring-wildfly-upload-file-size/
now I'd like my webapp to inform users. So I would like to render on the screen that the server accepts upload up to XXX MB.
How can a webapp retrieve the max post size setting from JBoss? Is there even a container agnostic way that could work across different servlet containers?
The max-post-size allows for an expression. That means you could use a system property which could be set on the attribute and retrieved by your application.
In CLI you'd do something like this:
/system-property=max.post.size:add(value=25485760)
/subsystem=undertow/server=default-server/http-listener=default/:write-attribute(name=max-post-size,value=${max.post.size:10485760})
While the default, :10485760, is not strictly required, I'd advise it.
Then in your application you simply just to simply retrieve the system property.
Is there any way to determine database connection pool size. I want to find out min pool size, max pool size.
The reason is as follows:
My application is running on Wildfly-9.0.1.Final.
I have configure datasource in -ds.xml file.
I have so many clients and for each one there is -ds.xml file.
In each file I have specified max-pool-size = 30.
But for some clients this size(30) happens to be small as more and more user tries to get connection from pool concurrently. Then in that case I need to increase max-pool-size to higher number. I want something like that will help me to fetch these parameters and then based on that I will perform some logic. Like if pool-size have reached to 25/30 then it will trigger email as an alert so that developer can increase its pool size. This way it will be helpful to avoid problems that client do faces when he could not get connection when all are being aquired.
Is there any way to access these connection pool parameters programatically.
Well there are multiple ways. If your datasource would be configured in the standalone.xml you could easily achieve your goal
via the CLI command (have a look here if you're not familiar with the CLI)
/subsystem=datasources/data-source=ExampleDS/statistics=pool:read-resource(include-runtime=true)
through JMX
by reading the following MBean
jboss.as:subsystem=datasources,data-source=ExampleDS,statistics=pool
Beware: In both cases ExampleDS has to be replaced with your actual datasource name.
Update
If you directly drop a -ds.xml into the deployments directory you can read the statistics like this:
/deployment=my-ds.xml/subsystem=datasources/data-source=java\:jboss\/my\/jndiName\/for\/DeployedDS/statistics=pool:read-resource(include-runtime=true)
or
jboss.as:deployment=my-ds.xml,subsystem=datasources,data-source="java:jboss/my/jndiName/for/DeployedDS",statistics=pool
Note that in any case you will have to enable these statistics first before you can acces any useful information with the methods shown above. These statistics might be a performance drawback. You can enable these statistics for example via CLI:
/deployment=my-ds.xml/subsystem=datasources/data-source=java\:jboss\/my\/jndiName\/for\/DeployedDS/statistics=pool:write-attribute(name=statistics-enabled, value=true)
When working with WildFly I'd generally recommend to configure datasources in the standalone.xml - it's way better supported.
Please try this with jboss-cli:
[standalone#localhost:9999 /]
/subsystem=datasources/data-source=ExampleDS/statistics=pool:read-resource(include-runtime=true)
I have been given a java web application for which I have the source code to. The application queries an Oracle database to return data back to the user in web page. I need to update a value for the returned data in the database, without knowing the table or column names. Is there a general way to determine what query the application is submitting to return the data for a particular page so I can find the right tables and columns?
I am new to java and web development so not sure where to start looking.
Thanks!
Well, there's always the old fashioned way of finding out. You can find the source code for the specific page you're looking at and identify the query that's being executed to retrieve the data. I'm assuming that's not what you're looking for, though.
Some other options include using JDBC (Enabling and Using JDBC Logging) logging feature or JProfiler (the JDBC probe shows you all SQL statements in the events view). Once you find the SQL statement, you can use standard text search features within your IDE to locate the specific code and make alterations.
Hope that helps!
If you can run a controlled test (e.g., you are the only person on that web application), you could turn on SQL tracing on the DB connection and then run your transaction several times. To do this
look at all the connections from that application using v$session -- you can control this by tweaking your connection pool setting (e.g., set min and max connection to 1). Assuming this is your test environment.
turn on 10046 trace (see https://oracle-base.com/articles/misc/sql-trace-10046-trcsess-and-tkprof -- there are many other examples).
The 10046 trace will show you what the application is doing -- SQL by SQL. You can even set the level to 12 to get the bind variable values (assuming you are using prepared statements).
I have a requirement to cache xml bean java objects by reading xml’s from database . I am using a HashMap in memory to maintain my java objects. I am using spring for DI and Weblogic 11g app server.
Can you please suggest me a mechanism to reload cache when there is an update in xml files.
You can make use of weblogic p13n cache for this purpose, instead of using your own HashMap to cache the java objects. You will have to configure p13n-cache-config.xml file, which contains, TTL, max value etc. for your cache.
Coming to the first point, the cache will be automatically reloaded after the TTL time is done with. For manually clearing cache, You can implement a Servlet kind of thing, which you can hit directly from your browser (can restrict it for a particular URL). In that servlet clear the cache which you want to reload.
weblogic p13n cache provides you method for cluster aware cache clear as well, in case you need it, in case you want to use your own HashMap for caching, provide a update method for that HashMap and clear the java objects that you want to be reloaded and then call the cache creation method.
I've got an application leaking out java heap at a decent rate (400 users leaves 25% free after 2hours...after logoff all memory is restored) and we've identified the items causing the memory leak as Strings placed in session that appear to be generated by Portal itself. The values are the encoded Portal URIs (very long endcoded strings ... usually sized around 19kb), and the keys seem to be seven (7) randomly generated characters prefixed by RES# (for example, RES#NhhEY37).
We've stepped through the application using session tracing and snapping off heapdumps which has resulted in determining that there is one of these objects created and added to session on almost every page ... in fact, it seems like it is on each page that submits data (which is most pages). So, it's either 1:1 with pages in general, or 1:1 with forms.
Has anyone encountered a similar problem as this? We are opening a ticket with IBM, but wanted to ask this community as well. Thanks in advance!
Can it be the portlet cache? You could have servlet caching activated and declare a long portlet expiry time. Quoting from techjournal:
Portlets can advertise their ability to be cached in the fragment cache by setting their expiry time in their portlet.xml descriptor (see Portlet descriptor example)
<!-Expiration value is in seconds, -1 = no time limit, 0 = deactivated-->
<expiration-cache>3600</expiration-cache> <!- 1 Hour cache -->
To use the fragment caching functions, servlet caching needs to be activated in the Web Container section of WebSphere Application Server administrative console (see Portlet descriptor example). WebSphere Application Server also provides also a cache monitor enterprise application (CacheMonitor.ear), which is very useful for visualizing the contents of the fragment cache.
Update
Do you have portlets that set EXPIRATION_CACHE? Quote:
Modifying the local cache at runtime
For standard portlets, the portlet window can modify the expiration time at runtime by setting the EXPIRATION_CACHE property in the RenderResponse, as follows:
RenderResponse.setProperty(
PortletResponse.EXPIRATION_CACHE,
(new Integer(3000)).toString() );
Note that for me the value is a bit counter-intuitive, -1 means never expire, 0 means don't cache.
The actual issue turned out to be a working feature within Portal. Specifically, Portal's action protection which prevents the same action from being submitted twice, while keeping the navigational ability of the portal. There is a cache that retains the actions results for every successful action and uses them to compare and reject duplicates.
The issue for us was the fact that we required "longer than normal" user sessions (60+ minutes) and with 1,000+ concurrent users, we leaked out on this protection mechanism after just a couple hours.
IBM recommended that we just shut off the cache entirely using the following portlet.xml configuration entry:
wps.multiple.action.execution = true
This allows double submits, which may or may not harm business functionality. However, our internal Portal framework already contained a mechanism to prevent double submits, so this was not an issue for us.
At our request, IBM did come back with a patch for this issue which makes the cache customizeable, that is, let's you configure the number of action results that you store in cache for each user and thus you can leverage Portal's mechanism again, at a reduced session overhead. Those portal configuration settings were:
wps.multiple.action.cache.bound.enabled = true
wps.multiple.action.cache.key.maxsize = 40
wps.multiple.action.cache.value.maxsize = 10
You'll need to contact IBM about this patch as it is not currently in a released fixpack.
Is your Websphere Portal Server having latest fix pack installed?
http://www-01.ibm.com/support/docview.wss?uid=swg24024380&rs=0&cs=utf-8&context=SSHRKX&dc=D420&loc=en_US&lang=en&cc=US
Also you may be interested in following discussion
http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14427700&tstart=0
Update:
Just throwing some blind folded darts.
"RES#" to me sounds like resource.
From the forum stack trace,
"DefaultActionResultManager.storeDocument"
indicates it is storing the document.
Hence looks like your resources(generated portal pages) are being cached. Check if there is some paramater that can lmit cache size of resource.
Also in another test set cache expiration to 5 minutes instead of an hour.