I was attempting to use Streamsets to query an Oracle database and publish the data into Kafka. I downloaded Streamsets' tarball on my Mac and unzipped it into my home directory. Running $HOME/streamsets-datacollector-2.1.0.2/bin/streamsets dc started up on my first try, then I followed the instructions here to add the jdbc driver, then the instructions here to configure my streamsets job. However, I got an error: JDBC_00 - Cannot connect to specified database: com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: java.sql.SQLRecoverableException: IO Error: Bad file descriptor.
This wound up having something to do with the limit on the number of files a process can have open. When I ran ulimit -n on the laptop, it showed 4864, then I set it to 10,000 via ulimit -n 10000, restarted the streamsets server, and it worked! If I need to keep running this, I will find a more procedural way of setting the ulimit for this process to work around this issue.
you definitely want need increase ulimit -n. To permanently change for all users in Ubuntu try the following:
echo "* soft nofile 60000" > /etc/security/limits.d/*_limits.conf && echo "* hard nofile 60000" >> /etc/security/limits.d/*_limits.conf
substitute 60000 with whatever number you want. I've never had a problem in streamsets using 60k. you should be able to do something similar in bsd.
Related
I follow this steps to run the Metamap Java API 2014 on Linux platform
(Main page from metamap)
After start Metamap server, I run some main class in Metamap JavaAPI pakages, but it raises Error when querying Prolog Server: Connection refused
Then I check if the Metamap server is running on port 8066 by using the command:
sudo netstat -tulpn | grep 8066
but I receive nothing.
Did anyone had the same problem before and knows solution for this?
After running skrmedpostctl (and the optional word sense diambiguation server wsdserverctl), you need to run the mmserver executable in order to use the Java API for Metamap. This can be run by running the command
./bin/mmserver{two-digit-year}
Also, be sure to include the two jar files for metamap and prolog-beans in your classpath (in your IDE, this can be done by adding a dependency to these jars).
The issue may reside with the timeout: when the timeout parameter is not specified for the MetaMapImpl, it is set to 0, so the instance interprets any latency as failure and reports an error.
Increase the timeout to a reasonable value (for me 500 msec was enough).
I was getting this error after getting error SPIO_E_TOO_MANY_OPEN_FILES and losing the connection to the mmserver. I was tryig to construct instances of MetaMapApiImpl in a loop and calling processCitationsFromString for thousands of times. Therefore, I was getting error SPIO_E_TOO_MANY_OPEN_FILES after around a hundred repeats.
What I did to fix this error was to simply disconnect the api at the end of the loop, by calling the function disconnect().
I have set the
kern.maxfiles=65536
kern.maxfilesperproc=65536
After this, I put the following command in my .zshrc file
ulimit -n 30000
However, if I try to run a netty based application from eclipse, only 10k sockets open and then java IO exception "too many open files" occurs. Folowing is the stacktrace.
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at io.netty.channel.socket.nio.NioServerSocketChannel.doReadMessages(NioServerSocketChannel.java:135)
at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:68)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:510)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:467)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:381)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:834)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
I use the following command to check the number of open files/sockets being used by my server and it always shows a value slightly more that 10k when the exception occurs.
lsof -p <pid> | wc -l
Increase your file descriptor value to achieve. You are setting values to "65536" but invoking only 30000 by adding in your shell
Please change the values then reload the bash, either by su - in current session or open the new terminal/session, then restart your application. It must work
You can set the ulimit with the help of following link.
https://vasnlinux.wordpress.com/2015/05/01/linux-server-hardening/
We have a custom service that we install with our application. The only problem is that after it is installed, it will not start, generating the error above. I have tried to diagnose what the problem is, but can't seem to find any useful information as to why it is quitting. I have tried the same service on a non "R2" 2008 server, and manual it worked fine.
service simple java file running using batch file. Deamon service.
Has anyone had any experience troubleshooting this type of problem, where there are so few clues?
Goto:
Registry-> HKEY_LOCAL_MACHINE-> System-> CurrentControlSet-> Services.
Find the concerned service & delete it. Close regedit. Reboot the PC & Re-install the concerned service. Now the error should be gone.
This is a problem related permission.
Make sure that the current user has access to the folder which contains installation files.
I resolved the problem.This is for EAServer Windows Service
Resolution is -->
Open Regedit in Run prompt
Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\EAServer
In parameters, give SERVERNAME entry as EAServer.
[It is sometime overwritten with Envirnoment variable : Path value]
This error message appears if the Windows service launcher has quit immediately after being started.
This problem usually happens because the license key has not been correctly deployed(license.txt file in the license folder).
If service is not strtign with correct key, just put incorrect key and try to start. Once started, place the correct key, it will work.
I had this error, I looked into a log file C:\...\mysql\data\VM-IIS-Server.err and found this
2016-06-07 17:56:07 160c InnoDB: Error: unable to create temporary file; errno: 2
2016-06-07 17:56:07 3392 [ERROR] Plugin 'InnoDB' init function returned error.
2016-06-07 17:56:07 3392 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2016-06-07 17:56:07 3392 [ERROR] Unknown/unsupported storage engine: InnoDB
2016-06-07 17:56:07 3392 [ERROR] Aborting
The first line says "unable to create temporary file", it sounds like "insufficient privileges", first I tried to give access to mysql folder for my current user - no effect, then after some wandering around I came up to control panel->Administration->Services->Right Clicked MysqlService->Properties->Log On, switched to "This account", entered my username/password, clicked OK, and it woked!
In my case the error 1067 was caused with a specific version of Tomcat 7.0.96 32-bit in combination with AdoptOpenJDK. Spent two hours on it, un-installing, re-installing and trying different Java settings but Tomcat would not start. See...
ASF Bugzilla – Bug 63625
seems to point at the issue though they refer to seeing a different error.
I tried 7.0.99 32-bit and it started straight away with the same AdoptOpenJDK 32-bit binary install.
I solved this issue using Monitor Tomcat application. I ran it and after a few seconds its icon appeared in my system tray. I right clicked on the icon and clicked the start button and after a few seconds Apache Tomcat started.
I'm using the following on Windows 7:
neo4j-community-1.9.2
Java 7 Update 25
I have Windows Firewall disabled.
When I start Neo4j.bat, both as Administrator and normally, I get the following message:
c:\Tools\neo4j\bin>Neo4j.bat
28/07/13 9:34:27 PM org.neo4j.server.AbstractNeoServer INFO: Setting startup tim
eout to: 120000ms based on -1
A blank Java console window pops up, no messages, then the window disappears after a minute or so.
When I go to http://localhost:7474/, I just get a server not found error.
If I try to install it as a service, the service wouldn't start.
Any ideas what I need to do to start Neo4j? I see here that Neo4j 1.9 does support Java 7:
http://blog.neo4j.org/2013/03/neo4j-19m05-released-wrapping-up.html
base.bat correctly points to windows-service-wrapper-5.jar, so not the windows-service-wrapper-*.jar problem that some people have had in the past. It would be helpful if it outputted an error message.
I just tried it with neo4j 2.0 milestone, and it didn't make any difference. The Java window pops up with no messages, http://localhost:7474/ gets a server not found error, then the Java window closes after a minute or two.
I added JAVA_HOME and JRE_HOME system variables, made no difference. I think I would get an error if Neo4j was not able to locate Java anyway.
I've fixed it.
With help from Neo4j's Google Group, I looked at the logs in data/logs/*.log and data/graph.db/messages.log, and found the following exception:
Caused by: java.rmi.server.ExportException: Port already in use: 1337; nested exception is:
java.net.BindException: Address already in use: JVM_Bind
Once I disabled my other service that uses that port, Neo4j's web admin at http://localhost:7474 worked. I then found out port 1337 is used by Neo4j's remote shell, I disabled it by adding enable_remote_shell = false to neo4j.properties, then I was able to have both Neo4j and my other service running.
I'm using the version 2.2.1 community edition of Neo4j on Windows 7, Java 1.8 update 45
In my case the log file messages.log was under default.graphdb directory. In there, you'll see the root cause of the problem.
I have two servers running on Glassfish 2.1 both have the same web app.
Two times this error occurred: Some jsp pages stop displaying only showing a blank page, and the following errors are printed in the logs...
PWC1231: Servlet.service() for servlet jsp threw exception
java.io.FileNotFoundException: /path/to/jsp/file/jsp_file.jsp.java
(Permission denied) at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:179) at
java.io.FileOutputStream.(FileOutputStream.java:70) at
org.apache.jasper.compiler.AntJavaCompiler.getJavaWriter(AntJavaCompiler.java:213)
at
org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:173)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:409) at
org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:592)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:344)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at
org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at
org.apache.catalina.core.ApplicationDispatcher.doInvoke(ApplicationDispatcher.java:855)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:703)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:542)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:474)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:366)
at
org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056)
at
org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:261)
at
org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:388)
at
org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:316)
at
org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231)
at
org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
at
org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738) at
javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at
org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at
com.my.app.filtro.FiltroCallcenter.doFilter(FiltroCallcenter.java:90)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
at
com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at
com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at
com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
at
com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
Followed by this:
PWC6344: Unable to create output writer for file /path/to/jsp/file/jsp_file.jsp.java|#]
Sometimes only the PWC6344 error is printed and sometimes both, the PWC1231 error is always followed by PWC6344 (which somewhat makes sense, because that exception is thrown when an IOException occurs).
Well, both times this errors happened, the only thing I did was stop and start the instance, and everything was alright again, also this error has only occurred in one of the servers.
Whats is happening ?... Or how can I diagnose what is causing this so I can fix the problem instead of stopping and re starting for eternity...
UPDATES:
I was looking into the possibility of this been a file descriptor problem, as suggested by sbridges but!, the maximum number of file handlers is 811975 with 4520 opened files in one server and 359532 with only 6894 in the other.
So, I guess its safe to say this is not the problem!
Does someone have another theory?
It looks like the permissions are set incorrectly and you can't write the compiled jsp page to disk,
/path/to/jsp/file/jsp_file.jsp.java (Permission denied)
Are the permissions correct on that directory/file?
Rename jsp_file.jsp.java to jsp_file.jsp.
PWC6344: Unable to create output writer for file /path/to/jsp/file/jsp_file.jsp.java|#]
This can also happen if the underlying operating system has run out of file handlers/descriptors which are required in order to open a file for reading or writing. I'm not closely familiar with CentOS, but Google hints that it has a "relatively low" limit of 1024 given the lot of problems related to it in the search results. Among the results you'll see a lot of questions/answers as to how to increase it, such as the following blog:
Increase the number of file descriptors on Centos and Fedora Linux
Raising the number of file descriptors for a regular user on CentOS/Fedora/Redhat is surprisingly difficult to learn how to do. There are lots of incomplete walk throughs on the web, some with typos and other problems.
Here are the steps that worked for me to raise the open file descriptor limit from 1024 (the default) to 65535:
As root, edit /etc/sysctl.conf and add the line:
fs.file-max = 512000
At the bash prompt, run:
$ sysctl -p
That will cause the settings to take effect. You can also cat 512000 > /proc/sys/fs/file-max but that may reset on reboot.
Edit /etc/security/limits.conf and add the following:
* - nofile 65535
See the inline comments for more details on what that does and how to make it more restrictive if you prefer.
As root, run
$ ulimit -n 65535
and make sure you have no errors. To double check, run ulimit -n and make sure the response is 65535.
Ensure that PAM authentication is turned on for SSH, or else when you try to connect as a regular user, you won’t see the new limits. Edit /etc/ssh/sshd_config and make sure you have:
UsePAM yes
Restart SSH /sbin/service sshd restart if you made any changes.
Login as a regular user with a new SSH session & shell and run:
$ ulimit -n 65535
Run ulimit -n again to check and good luck!