We Keep Coding

How to correct the exception database.ocf' is locked by another process in orientdb?

I'm getting the exception when I try to connect to the orient db using Java. Below is the exception I'm getting.
Jun 07, 2016 12:43:40 PM com.orientechnologies.common.log.OLogManager log
INFO: OrientDB auto-config DISKCACHE=891MB (heap=891MB direct=891MB os=4,006MB), assuming maximum direct memory size equals to maximum JVM heap size
Jun 07, 2016 12:43:40 PM com.orientechnologies.common.log.OLogManager log
WARNING: MaxDirectMemorySize JVM option is not set or has invalid value, that may cause out of memory errors. Please set the -XX:MaxDirectMemorySize=4006m option when you start the JVM.
Jun 07, 2016 12:43:40 PM com.orientechnologies.common.log.OLogManager log
WARNING: MaxDirectMemorySize JVM option is not set or has invalid value, that may cause out of memory errors. Please set the -XX:MaxDirectMemorySize=4006m option when you start the JVM.
Exception in thread "main" com.orientechnologies.orient.core.exception.OFileLockedByAnotherProcessException: File 'F:\Program Files\orientdb-community-2.2.0\databases\mydbo\database.ocf' is locked by another process, maybe the database is in use by another process. Use the remote mode with a OrientDB server to allow multiple access to the same database at com.orientechnologies.orient.core.storage.fs.OFileClassic.lock(OFileClassic.java:756)
at com.orientechnologies.orient.core.storage.fs.OFileClassic.openChannel(OFileClassic.java:813)
at com.orientechnologies.orient.core.storage.fs.OFileClassic.open(OFileClassic.java:603)
at com.orientechnologies.orient.core.storage.impl.local.OSingleFileSegment.open(OSingleFileSegment.java:51)
at com.orientechnologies.orient.core.storage.impl.local.OStorageConfigurationSegment.load(OStorageConfigurationSegment.java:80)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:186)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:231)
at orient.insert.Insert.main(Insert.java:12)
this is the code that i tried.
ODatabaseDocumentTx db = new ODatabaseDocumentTx("plocal:F:/Program Files/orientdb-community-2.2.0/databases/mydbo").open("admin", "admin");
ODocument doc = new ODocument("Person");
doc.field( "name", "Luke" );
doc.field( "surname", "Skywalker" );
doc.field( "city", new ODocument("City").field("name","Rome").field("country", "Italy") );
doc.save();
db.close();
I can't figure out the error I'm having.

You have a server running and you try to open the database from another process in plocal.
Could you please verify that you have no active OrientDB instances while accessing it in plocal (console or external processes) and that you open one plocal connection at a time?

Why does a servlet container synchronise access for multiple requests to a particular resource/ servlet?

By default, the servlet container loads only a single instance of a
servlet. Requests serviced by the servlet are run in separate threads,
but share the same instance. This enables applications to be
responsive and scalable, since it requires fewer resources and uses
them efficiently.
Almost everyone knows that this is the default threading model.
This question/discussion is a follow-up on this particular stackoverflow thread.
But the output happens to be completely different in my case,
May 21, 2016 1:57:59 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 1st thread
May 21, 2016 1:58:26 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 2nd thread
May 21, 2016 1:58:59 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test after // 1st thread
May 21, 2016 1:59:26 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test after // 2nd thread
Even though I hit the url mapped to this particlular servlet in 2 browser windows of the same browser, the difference b/w these 2 hitting ~1 second, but I see the below output
May 21, 2016 1:58:26 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 2nd thread
after seconds more than that previously mentioned ~1 second,
i.e to this,
May 21, 2016 1:57:59 PM com.stackoverflow.question8011138.MyServlet doGet
INFO: [doGet] Test before // 1st thread
Why?
Clearly, from the output above, the second request is not served after the first has been served completely, but in b/w, contrary to what the OP to that question suggested.
What exactly is going on?
Why 2 different outputs to the same scenario?
The result/ output clearly shows that the server is not synchronizing the 2 requests.

Stopping threads in a multi threaded application

I have created a service using procrun which launches certain jars through reflection. When the service is started it starts a thread and rest of the execution happens in that thread. Then each of the plugin loads its own threads and does the execution in there.
During service stop, I have called the stop method of the plugins. Those methods have returned and whatever thread I have created has been terminated for the plugins. But even after that the following threads are still running.
INFO: Thread No:0 = Timer-0
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:1 = WebSocketWorker-14
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:2 = WebSocketWorker-15
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:3 = WebSocketWorker-16
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:4 = WebSocketWorker-17
Jan 13, 2016 10:49:58 AM com.kube.desktop.KubeSdkMain stop
INFO: Thread No:5 = WebsocketSelector18
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:6 = AWT-EventQueue-0
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:7 = DestroyJavaVM
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
INFO: Thread No:8 = Thread-11
Jan 13, 2016 10:49:58 AM com.test.desktop.SdkMain stop
The following is how I printed those threads.
ThreadGroup currentGroup = Thread.currentThread().getThreadGroup();
int noThreads = currentGroup.activeCount();
Thread[] lstThreads = new Thread[noThreads];
currentGroup.enumerate(lstThreads);
for (int i = 0; i < noThreads; i++)
LOGGER.log(Level.INFO, "Thread No:" + i + " = " + lstThreads[i].getName());
Because of these threads, when I stop the service, it takes forever and then times out. But when I call System.exit(0) the service stops quickly. What should I do to get rid of these threads? When I launch the jars through reflection, are there separate threads created for each plugin? If so could these be them? Please advice.

It looks like the plugins are themself launching threads ("INFO: Thread No:1 = WebSocketWorker-14" -> sockets usually should be put in seperate threads) which will not be shut down if you kill the initiating thread. You'll have to enforce your plugins to kill all threads they started when they get shut down to make sure that they will not leave stuff behind. "some plugins don't do a good job cleaning up whatever they created. it's just sloppy programming." - bayou.io is describing it really good there.
Calling System.exit() will just kill the process meaning it will kill all threads created by the process as well.
The other way would be to manually iterate over all running threads, check if it's the main thread, and if not proceed to kill it. You can get all running threads in an iterable set using
Set<Thread> threadSet = Thread.getAllStackTraces().keySet();
And you can get your currently running Thread using
Thread currentThread = Thread.currentThread();
Still this is the way you would not want to do it, it's more of a way to clean up if plugins decide to leave stuff behind rather than just doing it that way. The plugins themselfes should take care of shutting down the threads when they get disabled but if they don't do that you can use above way to manually clean it up.

How to resolve java.lang.outOfMemoryError in ThreadPoolExecutor program?

I am running a ThreadPoolExecutor program which creates 1000 url and send to executor service.
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:673)
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
at MyProgramName.main(MyProgramName.java:175)
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
2014-08-16 20:46:21.341 java[5100:d07] java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:673)
at sun.awt.AWTAutoShutdown.activateBlockerThread(AWTAutoShutdown.java:312)
at sun.awt.AWTAutoShutdown.setToolkitBusy(AWTAutoShutdown.java:232)
at sun.awt.AWTAutoShutdown.notifyToolkitThreadBusy(AWTAutoShutdown.java:118)
2014-08-16 20:46:21.342 java[5100:d07] (
0 CoreFoundation 0x00007fff9127825c __exceptionPreprocess + 172
1 libobjc.A.dylib 0x00007fff9031ce75 objc_exception_throw + 43
2 CoreFoundation 0x00007fff91277e09 -[NSException raise] + 9
3 JavaNativeFoundation 0x000000010c754c3f JNFCallStaticVoidMethod + 213
4 libawt.jnilib 0x0000000115fb8151 setBusy + 53
5 CoreFoundation 0x00007fff911a8d67 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
6 CoreFoundation 0x00007fff911a8cd7 __CFRunLoopDoObservers + 391
7 CoreFoundation 0x00007fff9119a608 __CFRunLoopRun + 1368
8 CoreFoundation 0x00007fff91199e75 CFRunLoopRunSpecific + 309
9 HIToolbox 0x00007fff8fdd9a0d RunCurrentEventLoopInMode + 226
10 HIToolbox 0x00007fff8fdd97b7 ReceiveNextEventCommon + 479
11 HIToolbox 0x00007fff8fdd95bc _BlockUntilNextEventMatchingListInModeWithFilter + 65
12 AppKit 0x00007fff8ac5d24e _DPSNextEvent + 1434
13 AppKit 0x00007fff8ac5c89b -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 122
14 libawt.jnilib 0x0000000115fb987f -[NSApplicationAWT nextEventMatchingMask:untilDate:inMode:dequeue:] + 124
15 AppKit 0x00007fff8ac5099c -[NSApplication run] + 553
16 libawt.jnilib 0x0000000115fb7f30 +[AWTStarter startAWT:] + 1495
17 libawt.jnilib 0x0000000115fb78aa -[CPerformer perform] + 93
18 Foundation 0x00007fff8c64313e __NSThreadPerformPerform + 229
19 CoreFoundation 0x00007fff911a95b1 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
20 CoreFoundation 0x00007fff9119ac62 __CFRunLoopDoSources0 + 242
21 CoreFoundation 0x00007fff9119a3ef __CFRunLoopRun + 831
22 CoreFoundation 0x00007fff91199e75 CFRunLoopRunSpecific + 309
23 java 0x000000010be86843 java + 18499
24 java 0x000000010be8629a java + 17050
25 java 0x000000010be83a98 java + 6808
26 ??? 0x0000000000000005 0x0 + 5
)
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
I already made some research in internet and stackoverflow. Some answers say I need to reduce stack size. Some say I need to increase the ulimit using ulimit -a. I am totally confused. I am confused with those answers since they are not specific to ThreadExecutor. Please help me
I run the code in Mac OS, i7 Quad core processor, 16 gb RAM, 1 Tb HD
Code of my program is here in this question Why does the following executor service java Thread program doesn't shut down?

The real problem is that you are trying to create too many threads using too much stack memory.
You could reduce the default size for the thread stacks (as suggested), but that isn't a complete solution. (When you decide to run your program on more URLs, you will run into the same problem again. And eventually you will hit "the wall" where you can't reduce the stack size any further; e.g. because you are getting StackOverflowError exceptions.)
A better solution is to NOT create so many threads. You've configured your program to use an infinite thread pool. That is frankly ridiculous. Change this:
ThreadPoolExecutor executor =
new ThreadPoolExecutor(Integer.MAX_VALUE, Integer.MAX_VALUE,
20, TimeUnit.MILLISECONDS, blockingQueue);
to something more realistic:
ThreadPoolExecutor executor =
new ThreadPoolExecutor(100, 100,
20, TimeUnit.MILLISECONDS, blockingQueue);
Beyond a certain point, adding extra threads won't make an application go any faster. Indeed, it will often start to make the application go slower, and sometimes catastrophically slower.

You face the “java.lang.OutOfMemoryError: Unable to create new native thread” whenever JVM is asking a new thread from the OS. Whenever the underlying OS cannot allocate a new native thread, this OutOfMemoryError will be thrown. The exact limit for native threads is very platform-dependent thus we recommend to find your limits by running a small test to find out your platform-specific limit to threads. But, in general, the situation causing java.lang.OutOfMemoryError: Unable to create new native thread goes through the following phases:
A new Java thread is requested by an application running inside the JVM
JVM native code proxies the request to create a new native thread to the OS
OS tries to create a new native thread which requires memory to be allocated to the thread
The OS will refuse native memory allocation either because the 32-bit Java process size has depleted its memory address space – e.g. (2-4) GB process size limit has been hit – or the virtual memory of the OS has been fully depleted
The java.lang.OutOfMemoryError: Unable to create new native thread error is thrown
In your case, the best way to overcome the issue is definitely follow the suggested path of limiting the thread creation.

Dealing with "[HOST_KEY_NOT_VERIFIABLE] Could not verify `ssh-rsa` host key with fingerprint" in sshj

I having a strange issue with sshj (am using sshj v0.6.0) for which I would need some help from someone.
Authentication with public key works fine on some machines but doesnt work fine on other machines and I see the below error.
The only difference that I could make out was that the UNIX ID in question viz coonradt seems to have the below listed configuration setup under ~/.ssh/config only on the box on which the below errors are being triggered
Host *
Protocol 1,2
FallBackToRsh no
ForwardAgent yes
ForwardX11 yes
PasswordAuthentication yes
RhostsAuthentication no
RhostsRSAAuthentication no
RSAAuthentication yes
NoHostAuthenticationForLocalhost yes
StrictHostKeyChecking no
KeepAlive yes
From the above config file I learnt that the ID in question is supposed to make use of Protocol 1,2 and I suspect that this might have something to do with my failures (I am not very sure about it, but this is just a hunch)
For all other UNIX IDs for which this works fine, I dont have any such config file.
PS : I cannot alter the config of the UNIX ID "coonradt" since this ID is being used by the central hudson servers.
Would appreciate if someone could please help me suggest as to what might be wrong here
Following is the error that I am seeing :
Oct 24, 2011 2:30:37 AM net.schmizz.sshj.DefaultConfig initCipherFactories
WARNING: Disabling high-strength ciphers: cipher strengths apparently limited by JCE policy
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.TransportImpl init
INFO: Client identity string: SSH-2.0-SSHJ_0_6_0
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.TransportImpl init
INFO: Server identity string: SSH-1.99-OpenSSH_4.3
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.KeyExchanger sendKexInit
INFO: Sending SSH_MSG_KEXINIT
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.KeyExchanger handle
INFO: Received SSH_MSG_KEXINIT
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.kex.AbstractDHG init
INFO: Sending SSH_MSG_KEXDH_INIT
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.KeyExchanger handle
INFO: Received kex followup data
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.kex.AbstractDHG next
INFO: Received SSH_MSG_KEXDH_REPLY
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.TransportImpl die
SEVERE: Dying because - net.schmizz.sshj.transport.TransportException: [HOST_KEY_NOT_VERIFIABLE] Could not verify `ssh-rsa` host key with fingerprint `ca:0b:b3:7f:53:5a:e3:bc:bf:44:63:d8:2d:26:c0:41` for `mymachine.domain.com` on port 22
Oct 24, 2011 2:30:38 AM net.schmizz.concurrent.Promise tryRetrieve
SEVERE: <<kex done>> woke to: net.schmizz.sshj.transport.TransportException: [HOST_KEY_NOT_VERIFIABLE] Could not verify `ssh-rsa` host key with fingerprint `ca:0b:b3:7f:53:5a:e3:bc:bf:44:63:d8:2d:26:c0:41` for `mymachine.domain.com` on port 22
Oct 24, 2011 2:30:38 AM net.schmizz.sshj.transport.TransportImpl setService
INFO: Setting active service to null-service
Oct 24, 2011 2:30:38 AM com.test.jaws.execution.ssh.impl.SSHJClientImpl$ExceptionHandler handleSevereCondition
SEVERE: mymachine.domain.com is not added to your /x/home/coonradt/.ssh/known_hosts file.
Throwable occurred: net.schmizz.sshj.transport.TransportException: [HOST_KEY_NOT_VERIFIABLE] Could not verify `ssh-rsa` host key with fingerprint `ca:0b:b3:7f:53:5a:e3:bc:bf:44:63:d8:2d:26:c0:41` for `mymachine.domain.com` on port 22
at net.schmizz.sshj.transport.KeyExchanger.verifyHost(KeyExchanger.java:222)
at net.schmizz.sshj.transport.KeyExchanger.handle(KeyExchanger.java:373)
at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:477)
at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:127)
at net.schmizz.sshj.transport.Decoder.received(Decoder.java:195)
at net.schmizz.sshj.transport.Reader.run(Reader.java:72)

You may set the SSH client to accept all keys without any verification (ignores host key verification)
SSHClient sshClient = new SSHClient();
sshClient.addHostKeyVerifier(new PromiscuousVerifier());
...

How about adding a HostKeyVerifier for this machine?
sshClient.addHostKeyVerifier("ca:0b:b3:7f:53:5a:e3:bc:bf:44:63:d8:2d:26:c0:41");
The reason it doesn't happen automatically is probably because the known_hosts file isn't at $(user.home)/.ssh/known_hosts. You can also explicitly load known hosts from a specific location.
sshClient.loadKnownHosts(new File("path_to_known_hosts"));

try {
ssh.connect(envConf.getIp(), port);
} catch (TransportException e) {
if (e.getDisconnectReason() == DisconnectReason.HOST_KEY_NOT_VERIFIABLE) {
String msg = e.getMessage();
String[] split = msg.split("`");
String vc = split[3];
ssh = new SSHClient();
ssh.addHostKeyVerifier(vc);
ssh.connect(envConf.getIp(), port);
} else {
throw e;
}
}
ssh.authPassword(envConf.getName(), envConf.getPw());
ssh.newSCPFileTransfer().download(envConf.getHomePath() + FilePath, toPath);

For an alternative answer ensure that the hostname you are trying to connect to is exactly a match in your known_hosts file. An example mistake that I was making was trying to connect to the full URL bob.insidenetwork.pvt but my known_hosts file had only bob as an entry because when I ssh manually I'm far too lazy to type the entire URL...

If the server accepts keyboard-interactive verification, you can also do this, at least with recent versions of SSHJ:
client.addHostKeyVerifier(new ConsoleKnownHostsVerifier(new File(userKnownHostsFile), System.console()) {
#Override
protected boolean hostKeyUnverifiableAction(String hostname, PublicKey key) {
try {
entries().add(new HostEntry(null, hostname, KeyType.fromKey(key), key));
write();
} catch (IOException e) {
throw new RuntimeException(e);
}
return true;
}
});
This overrides the "Are you sure you want to continue connecting (yes/no)? " prompt that the user is supposed to reply to, so that you don't need to provide any input into a console to continue, while satisfying the keyboard-interactive authentication when a host is unknown and avoiding using a PromiscuousVerifier.
You can, of course, tweak the overriding method to your heart's content to add logging, and if you want to be prompted then just use a plain ConsoleKnownHostsVerifier.
If the server doesn't accept keyboard-interactive authentication and you still don't want to use the promiscuous verifier, then you could do something similar to the above for the OpenSSHKnownHosts class; e.g.:
HostKeyVerifier hkv = new OpenSSHKnownHosts(new File("~/.ssh/known_hosts") {
#Override
protected boolean hostKeyUnverifiableAction(String hostname, PublicKey key) {
return true;
}
};
client.addHostKeyVerifier(hkv);
Inside OpenSSHKnownHosts, all the hostKeyUnverifiableAction(...) method does is return false, so change that and you're good.