I'm deploying a war file (Thingworx Iot Software) on tomcat with elastic beanstalk. The software is trying to create some folders/files under Several folders (at server root) I've given tomcat permissions to write to.
drwxrwxr-x 3 tomcat tomcat 4096 Jan 12 22:41 ThingworxBackupStorage
drwxr-xr-x 2 tomcat tomcat 4096 Jan 13 20:47 ThingworxPlatform
drwxrwxr-x 9 tomcat tomcat 4096 Jan 12 22:41 ThingworxStorage
Heres the error I'm getting:
Caused by: java.io.IOException: THINGWORX WARNING: could not create directory: /ThingworxStorage/database
at com.thingworx.common.utils.PathUtilities.createPath(PathUtilities.java:39)
at com.thingworx.system.configuration.PlatformSettings.prepareLocation(PlatformSettings.java:302)
... 20 more
I'm running Java 8 and Tomcat 8 on an AWS EB setup.
Seams like it should just be a simple permissions thing, but from what I can tell I've already done all that properly. What am I missing?
Related
I am trying to change the client_max_body_size property of my Elastic Beanstalk NGINX reverse-proxy in order to allow uploads for larger JPEG files. Therefore, I added the folder ".ebextensions" to the root directory of my WAR file (the WAR file is also including a Spring Boot application) and added a file ".ebextensions/01_files.config" with the following content:
files:
"/etc/nginx/conf.d/proxy.conf" :
mode: "000755"
owner: root
group: root
content: |
client_max_body_size 20M;
I deploy the WAR file via Travis-CI to Elastic Beanstalk. However, it seems that the file is beeing ignored by Elastic Beanstalk since uploads with a filesize e.g. 2MB do not work and when connecting with SSH to the instance and looking for "/etc/nginx/conf.d/proxy.conf" the file does not exist.
I already successfully validated above content with an YAML validator. I know, there exists plenty of related questions but non of those seem to fix my problem. I also checked if ".ebextensions/01_files.config" is included in the WAR file in root directory. And when I check "/tmp/eb_extracted_jar", the file ".ebextensions/01_files.config" also exists with the correct content. I can't even find any errors in the "/var/log/cfn-init.log". I noticed that, just for some seconds, the file "proxy.conf" appeared in "/etc/nginx/conf.d/" during deployment but then it has been removed.
Can this problem occure because of the deployment to Elastic Beanstalk via Travis-CI? Or did I miss something else that is important?
EDIT:
I just recognized that the "proxy.conf" file is created every time for a few seconds when the application is deployed but after a few seconds it disappears (checked with ls -lsa in "/etc/nginx/conf.d/", see the timestamps with 13:34 for "elasticbeanstalk" directory and "healthd_http.conf" and 13:43 for "proxy.conf")
4 drwxr-xr-x 3 root root 4096 6. Dec 13:43 .
4 drwxr-xr-x 4 root root 4096 6. Dec 13:34 ..
4 drwxr-xr-x 2 root root 4096 6. Dec 13:34 elasticbeanstalk
4 -rw-r--r-- 1 root root 148 6. Dec 13:34 healthd_http.conf
4 -rwxr-xr-x 1 root root 26 6. Dec 13:43 proxy.conf
And after a few seconds ls -lsa "/etc/nginx/conf.d/":
4 drwxr-xr-x 3 root root 4096 6. Dec 13:44 .
4 drwxr-xr-x 4 root root 4096 6. Dec 13:44 ..
4 drwxr-xr-x 2 root root 4096 6. Dec 13:44 elasticbeanstalk
4 -rw-r--r-- 1 root root 148 6. Dec 13:44 healthd_http.conf
After hours of reading docs, I found out that I missed some important parts of the official AWS docs for the Elastic Beanstalk Java SE Platform (see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/java-se-platform.html).
Definitely my mistake: I used the wrong file extension, the file extension in the folder ".ebextensions" has to be ".conf" but not ".config".
At least for the Java SE Platform: one can directly add NGINX config files within the ".ebextensions" directory without using the "files: ..." syntax to generate a file with a specific content, i.e. to create the proxy file in "/etc/nginx/conf.d/proxy.conf" just add ".ebextension/nginx/conf.d/proxy.conf" with the content client_max_body_size 20M; directly. Subsequently, "proxy.conf" will then be deployed to "/etc/nginx/conf.d/proxy.conf" and automatically included by the default NGINX config.
Hope this answer saves someone else the time it took me to figure that out.
I am currently trying to deploy a GWT web app to a tomcat instance, and the program isn't running correctly. On close examination of the tomcat logs, it looks like Tomcat is running into a AccessDeniedException.
SEVERE: Exception while dispatching incoming RPC call
com.google.gwt.user.client.rpc.SerializationException: Type
'java.nio.file.AccessDeniedException' was not included in the set of
types which can be serialized by this SerializationPolicy or its Class
object could not be loaded. For security purposes, this type will not
be serialized.: instance = java.nio.file.AccessDeniedException:
/luceneIndex
I am perplexed however, as the folder should be easily accessible to tomcat.
drwxrwxrwx 4 tomcat tomcat 4096 Mar 14 20:32 SearchTest
-rwxrwxrwx 1 tomcat tomcat 602 Mar 12 20:14 SearchTest.css
-rwxrwxrwx 1 tomcat tomcat 3038 Mar 13 23:13 SearchTest.html
drwxrwxrwx 5 tomcat tomcat 4096 Mar 14 20:24 WEB-INF
-rwxrwxrwx 1 tomcat tomcat 1082 Mar 12 20:14 favicon.ico
drwxrwxrwx 2 tomcat tomcat 4096 Mar 14 20:24 luceneFiles
drwxrwxrwx 2 tomcat tomcat 4096 Mar 14 20:24 luceneIndex
What could be causing this issue?
Are you using 'java.nio.file.AccessDeniedException' in your client-side code? The GWT compiler translates all the client-side Java code into JavaScript. Since not all Java functionalities are available in a web browser, GWT only supports a subset of the Java runtime library.
You can take a look at these two page JRE Compatablity and JRE Emulation.
As far as I can tell, none of the classes within java.nio package are emulated. That might be the reason you got your error.
A CentOS 7 server has Java 7 and tomcat 8 installed. I need to set the permgen space, but typing printenv on the command line does not include JAVA_OPTS in the list of environmental variables. I typed ps to get the list of running processes, and then I typed jmap -heap <pid> with the pid from the running java process, which listed values for MaxPermSize and PermSize among other values.
How do I set new values for these properties in a way that will persist when the system is rebooted, etc.? I hesitate to just create a JAVA_OPTS variable if the server in question has another place where it stores these persistent values.
EDIT:
As per #ChrisRobak's suggestion, I went searching for tomcat conf files. There is no /etc/tomcat directory, but /etc has directories called /etc/java, /etc/.java, /etc/jvm, and /etc/jvm-common. Also, the tomcat conf is as follows:
[root#myserver tomcat]# cd /opt/tomcat/conf
[root#myserver conf]# ls -al
total 280
drwxrwx---. 3 root root 4096 Oct 30 20:02 .
drwxr-xr-x. 14 root root 4096 Oct 30 20:58 ..
drwxrwx---. 7 root root 4096 Oct 30 21:00 Catalina
-rwxrwx---. 1 root root 12624 Nov 2 2014 catalina.policy
-rwxrwx---. 1 root root 6560 Nov 2 2014 catalina.properties
-rwxrwx---. 1 root root 1852 Jan 2 2015 context.xml
-rwxrwx---. 1 root root 3451 Nov 2 2014 logging.properties
-rwxr-x--- 1 root root 5018 Oct 30 21:52 server.xml
-rwxrwx---. 1 root root 1783 Nov 2 2014 tomcat-users.xml
-rwxrwx---. 1 root root 1888 Nov 2 2014 tomcat-users.xsd
-rwxrwx---. 1 root root 168082 Nov 2 2014 web.xml
[root#myserver conf]#
Which file should I look in?
Alternatively, is there a command line way of setting the Java options which will not only be persistent, but also not cause side effects due to conflicts with config files? I would just create a JAVA_OPTS variable if I thought the rest of the server would go to the JAVA_OPTS as the default.
ANSWER?
When none of the config files and scripts seemed to have JAVA_OPTS set, I finally just decided to type the following in the CENTOS 7 terminal:
export JAVA_OPTS="-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=512m"
The result is that the MaxPermSize has been reset. If anyone else has a specific answer to CentOS 7 that is better than this, please alert me.
I think you can persist those in
/etc/tomcat8/tomcat8.conf
or a similar variant, the server i had access to was /etc/tomcat5/tomcat5.conf but i don't know about your version in particular
I'm having an issue very strange in my enviroment wich is
jboss-5.1.0.GA
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
It was completely functional until I update a version of some jar (wich was tested in 6 enviroments before this one without the error) in the context.
before the error i had this files:
server/default/deploy/myfolder
-rw-r--r-- 1 jboss users 808430 May 16 17:29 s-c-core-2.38.6.0.jar
-rw-r--r-- 1 jboss users 555328 May 16 17:29 s-c-dao-2.38.6.0.jar
drwxr-xr-x 9 jboss users 4096 Jun 3 20:11 s-c-web.war
-rw-r--r-- 1 jboss users 199259 May 2 16:58 s-com-core-2.38.5.0.jar
-rw-r--r-- 1 jboss users 202086 May 2 16:58 s-com-dao-2.38.5.0.jar
-rw-r--r-- 1 jboss users 1333612 May 2 16:58 s-com-domain-2.38.5.0.jar
-rw-r--r-- 1 jboss users 86526 May 2 16:58 s-com-util-2.38.5.0.jar
-rw-r--r-- 1 jboss users 660621 May 11 13:01 s-fin-core-2.38.6.0.jar
-rw-r--r-- 1 jboss users 559292 May 11 13:01 s-fin-dao-2.38.6.0.jar
drwxr-xr-x 8 jboss users 4096 May 15 21:41 s-fin-web.war
-rw-r--r-- 1 jboss users 108444 May 7 07:29 s-geio-engine-2.38.5.0.jar
drwxr-xr-x 8 jboss users 4096 May 15 21:24 s-geio-web.war
-rw-r--r-- 1 jboss users 71975 Jun 14 2012 s-o-core-1.1.4.0.jar
-rw-r--r-- 1 jboss users 66516 Jun 14 2012 s-o-dao-1.1.4.0.jar
-rw-r--r-- 1 jboss users 53532 Jun 14 2012 s-o-domain-1.1.4.0.jar
-rw-r--r-- 1 jboss users 16347 Jun 14 2012 s-o-util-1.1.4.0.jar
drwxr-xr-x 5 jboss users 4096 Jun 14 2012 s-o-web.war
-rw-r--r-- 1 jboss users 54588 Apr 10 11:29 s-p-core-2.38.2.0.jar
-rw-r--r-- 1 jboss users 35784 Apr 10 11:29 s-p-dao-2.38.2.0.jar
drwxr-xr-x 8 jboss users 4096 Apr 10 19:20 s-p-web.war
-rw-r--r-- 1 jboss users 69140 Jun 14 2012 s-s-se-1.1.0.jar
-rw-r--r-- 1 jboss users 594214 May 13 13:36 s-ser-core-2.38.6.1.jar
-rw-r--r-- 1 jboss users 530903 May 13 13:36 s-ser-dao-2.38.6.1.jar
drwxr-xr-x 9 jboss users 4096 May 15 21:58 s-ser-web.war
drwxr-xr-x 6 jboss users 4096 May 28 08:27 s-te-me.war
-rw-r--r-- 1 jboss users 155639 May 15 22:06 s-uni-core-2.38.5.0.jar
-rw-r--r-- 1 jboss users 145944 May 15 22:05 s-uni-dao-2.38.5.0.jar
drwxr-xr-x 7 jboss users 4096 May 15 21:20 s-uni-web.war
then I stop my jboss (/etc/init.d/jbossd stop) and update just theese three files/folder (deleting the old ones and putting the new ones)
-rw-r--r-- 1 jboss users 808430 May 16 17:29 s-c-core-2.38.7.0.jar
-rw-r--r-- 1 jboss users 555328 May 16 17:29 s-c-dao-2.38.7.0.jar
drwxr-xr-x 9 jboss users 4096 Jun 3 20:11 s-c-web.war
after server start with the user root and with the command "/etc/init.d/jbossd start"
at some point the users start to use applications and on the log file server/default/log/server.log start to appear some strange errors like this:
2013-06-04 00:01:30,615 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jmx-console].[HtmlAdaptor]] (http-0.0.0.0-80-3) Servlet.service() for servlet HtmlAdaptor threw exception
javax.management.InstanceNotFoundException: jboss.j2ee:jar=s-c-core-2.38.6.0.jar,name=ApCarBusinessImpl,service=EJB3 is not registered.
at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:526)
at org.jboss.mx.server.MBeanServerImpl.getMBeanInfo(MBeanServerImpl.java:675)
at org.jboss.jmx.adaptor.control.Server.getMBeanData(Server.java:98)
at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet$1.run(HtmlAdaptorServlet.java:357)
at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet$1.run(HtmlAdaptorServlet.java:354)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.getMBeanData(HtmlAdaptorServlet.java:353)
at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.inspectMBean(HtmlAdaptorServlet.java:224)
at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.processRequest(HtmlAdaptorServlet.java:100)
at org.jboss.jmx.adaptor.html.HtmlAdaptorServlet.doGet(HtmlAdaptorServlet.java:81)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:383)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Saying that the EJBs are not instantiated (lots of then)
but nothing happens to the user, the application keep working normally
So I start to verify if the file s-c-core-2.38.6.0.jar wasn't left behind in some folder inside my jboss and NO it didn't. So I start to think that is some kind of cache that the jboss was reading so I stop the service and delete the folders server/default/tmp and server/default/work and restart the server but the error keep going on the log.
My run.conf is with this configuration:
JAVA_OPTS="-DSB_BASE=$SB_BASE -Dmodulo.context.file=s-o-core-context.xml;s-com-core-context.xml;s-ser-core-context.xml;s-geio-core-context.xml;s-fin-core-context.xml;s-c-core-context.xml;s-uni-core-context.xml;s-p-core-context.xml -Xmx3072m -Xms768m -XX:MaxPermSize=256m -Xss1024k -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseConcMarkSweepGC -Djava.awt.headless=true"
I tried a lot of things like user permissions, folder permissions, memory configurations (xmx xms) and the error keep appearing in the log file.
When I came back to the files that I changed ( s-c--2.38.6.0 ) the application stop to show this error, so I obviously though that the erro is with that package then I applyed it on a mirror server and the error does'nt show up.
So I am without options right now. The application is working fine for the user but the error keep appearing in the log and this is very annoying.
Has someone here experienced something like this?
I've posted this question on community.jboss.org 2 days ago with no help:
Thanks in advance,
sorry about my english.
The error basically implies that JBoss was unable to find the service MBean in its tree. Do you see that service deployed in the jmx-console?
You are seeing the InstanceNotFoundException: jboss.j2ee:jar=s-c-core-2.38.6.0.jar,name=ApCarBusinessImpl,service=EJB3 is not registered because:
Some external service/tool/app or your own application is trying to invoke that particular service (name=ApCarBusinessImpl,service=EJB3). Since JBoss cant find it deployed anywhere (as a result of you updating the deployed JAR), it simply logs the message that it couldn't find the MBean that it was asked to invoke.
To resolve the issue you will need to find out who is making the request on the old service name (jboss.j2ee:jar=s-c-core-2.38.6.0.jar) and have it updated to point to the new service name. This should resolve seeing those exception log messages.
Edit by Jorge Campos:
As #Coolbeans said it was indeed a external service that was calling my beans started on JBoss. I found out that in the installed jboss was a security issue that was not properly configured (jmx-console). So, there was a worm that was making requests to a previosly installed beans in my application server (some kind of cache). After some digging inside the jboss installation I find this war iesvc.war which is a worm as stated in this thread
https://community.jboss.org/blogs/mjc/2011/10/20/statement-regarding-security-threat-to-jboss-application-server
The worm is JAVA_JBOSSPY.A identified by a trend micro office scan.
So thank you for the answer #Coolbeans you were right it was a "Some external service/tool/app" as you said.
This is sometimes caused by the PU name. Try to change the PU name to another name, clean and build and deploy.
Regards
I'm using Play to write a webapp which is deployed in Tomcat. Because the app won't be processing very much data I'm using the default H2 database with Hibernate. When I want to deploy a new version of the app, I shut down tomcat, wipe the old webapp and WAR, add my new WAR, and start back up.
This worked until a few days ago, when I added the database component. Now, I am often unable to redeploy the app. When I delete the old directory, it is automatically regenerated with this structure:
$ ls -laR myapp/
myapp/:
total 24
drwxr-xr-x 3 root root 4096 Aug 24 17:20 .
drwxr-xr-x 13 root root 4096 Aug 24 17:20 ..
drwxr-xr-x 3 root root 4096 Aug 24 17:20 WEB-INF
myapp/WEB-INF:
total 24
drwxr-xr-x 3 root root 4096 Aug 24 17:20 .
drwxr-xr-x 3 root root 4096 Aug 24 17:20 ..
drwxr-xr-x 3 root root 4096 Aug 24 17:20 application
myapp/WEB-INF/application:
total 24
drwxr-xr-x 3 root root 4096 Aug 24 17:20 .
drwxr-xr-x 3 root root 4096 Aug 24 17:20 ..
drwxr-xr-x 3 root root 4096 Aug 24 17:20 db
myapp/WEB-INF/application/db:
total 24
drwxr-xr-x 3 root root 4096 Aug 24 17:20 .
drwxr-xr-x 3 root root 4096 Aug 24 17:20 ..
drwxr-xr-x 2 root root 4096 Aug 24 17:20 h2
myapp/WEB-INF/application/db/h2:
total 24
drwxr-xr-x 2 root root 4096 Aug 24 17:20 .
drwxr-xr-x 3 root root 4096 Aug 24 17:20 ..
-rw-r--r-- 1 root root 100 Aug 24 17:20 play.lock.db
The same happens when the WAR unzips.
I recently noticed a message whiz by in the catalina.out log complaining about my app not shutting down a process called something like "H2 File Lock Watchdog". Based on a brief search of the H2 docs, I think that process is what's interfering with my app.
EDIT
Here's the complaining line in the log file:
SEVERE: The web application [/myapp] appears to have started a thread named [H2 File Lock Watchdog /var/lib/apache-tomcat-6.0.32/webapps/myapp/WEB-INF/application/db/h2/play.lock.db] but has failed to stop it. This is very likely to create a memory leak.
So, how do I kill this process? I can't restart the machine because it's not mine, and I can't find the watchdog with top or ps. I'd prefer a way for Play to shut it down automagically, but I'm not above building it into my deployment script.
Thanks a million if you've read this far!
I shut down tomcat
Are you sure you have shut down tomcat completely? Because the H2 database is sill running. If you shut down the tomcat process, the database is also stopped (because H2 is running within the tomcat process). Except if you run the database in a different process.
Or did you just shut down the web application within tomcat? If that is the case, then at least one database connection was not closed, so that the database keeps running (and creates this .lock.db file).
Now, I don't know the play framework, and can't say how to ensure all database connections are closed.
One way to force the database to close is to run the SQL statement SHUTDOWN.
I can't find the watchdog with top or ps
top and ps only display processes. The H2 watchdog is a thread within a java process. To see the thread, use:
jps -l (to get the list of Java processes)
jstack -l <pid> (to get a full thread dump)