I'm trying to use the expectit library with sshj like so:
final Session session = getSharedSession();
final Session.Command sessionCommand = session.exec(command);
try (Expect expect = new ExpectBuilder()
.withOutput(sessionCommand.getOutputStream())
.withInputs(sessionCommand.getInputStream(), sessionCommand.getErrorStream())
.withInputFilters(removeColors(), removeNonPrintable())
.withEchoInput(LoggingAppendableAdapter.getInstance())
.withEchoOutput(LoggingAppendableAdapter.getInstance())
.withExceptionOnFailure()
.build()) {
for (SshExpectations sshExpectation : sequence) {
expect.expect(contains(sshExpectation.getExpectation()));
expect.sendLine(sshExpectation.getReaction());
}
}
The command I'm executing is "sleep 5; rm -i test.txt", I get the following output:
Jul 26, 2017 6:07:14 PM net.sf.expectit.SingleInputExpect start
FINE: Starting expect thread: input=< ChannelInputStream for Channel #1 >, charset=UTF-8, echoInput=LoggingAppendableAdapter#7a1099d7, filter=net.sf.expectit.filter.Filters$3#3afe1f22, bufferSize=1024
Jul 26, 2017 6:07:14 PM net.sf.expectit.SingleInputExpect start
FINE: Starting expect thread: input=< ChannelInputStream for Channel #1 >, charset=UTF-8, echoInput=LoggingAppendableAdapter#7a1099d7, filter=net.sf.expectit.filter.Filters$3#3afe1f22, bufferSize=1024
Jul 26, 2017 6:07:14 PM net.sf.expectit.ExpectImpl expectIn
FINE: Expect matcher 'contains('remove regular empty')' with timeout 30000 (ms) in input #0
Jul 26, 2017 6:07:14 PM net.sf.expectit.SingleInputExpect expect
FINE: Initial matcher contains('remove regular empty') result: SimpleResult{succeeded=false, before='null', group='null', input='', canStopMatching=false}
2017-07-26 18:07:19,574 INFO [expect-pool-26-thread-2] LoggingAppendableAdapter rm: remove regular empty file ‘test’?
Jul 26, 2017 6:07:19 PM net.sf.expectit.InputStreamCopier call
FINE: Received from < ChannelInputStream for Channel #1 >: rm: remove regular empty file ‘test’?
Jul 26, 2017 6:07:44 PM net.sf.expectit.SingleInputExpect expect
FINE: Selector returns 0 key
Jul 26, 2017 6:07:44 PM net.sf.expectit.SingleInputExpect stop
FINE: Releasing resources for input: < ChannelInputStream for Channel #1 >
Jul 26, 2017 6:07:44 PM net.sf.expectit.SingleInputExpect stop
FINE: Releasing resources for input: < ChannelInputStream for Channel #1 >
2017-07-26 18:07:46,800 ERROR Could not execute command
net.sf.expectit.ExpectIOException: Expect operation fails (timeout: 30000 ms) for matcher: contains('remove regular empty')
at net.sf.expectit.ExpectImpl.expectIn(ExpectImpl.java:106)
at net.sf.expectit.AbstractExpectImpl.expectIn(AbstractExpectImpl.java:57)
at net.sf.expectit.AbstractExpectImpl.expect(AbstractExpectImpl.java:61)
One would think that everything should work fine as both the LoggingAppendableAdapter as well as the internal logging report the string "rm: remove regular empty file ‘test’?".
Any suggestions on what I might doing wrong?
The issue is that rm -i test.txtwrites its question to stderr instead of stdout, but expectit by default only checks the first input stream (which in my case was the stdout stream).
By using
expect.expectIn(1, contains(sshExpectation.getExpectation()));
expectit then expects the string to be on the second given input stream, which in my case is stderr.
Related
I am using httpclient-3.0 library to parse data to cloud. When I run the application on my local machine (Windows 10), it works fine and the data gets parsed to the server and I receive success response, but when I deployed it on our server which runs on Windows server 2012 R2, it throws below error. I have used the same JDK as well I tried many ways like adding -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" in my java.security jdk file, but still the issue is not resolved.
Here is my code
PostMethod post = new PostMethod(apiUrl);
post.setParameter("authtoken", authToken);
post.setParameter("dateFormat", dateTimeFormat);
post.setParameter("data", emloyeesAttendanceJsonArr.toString());
HttpClient httpclient = new HttpClient();
// Configuring proxy
httpclient.getHostConfiguration().setProxy("**.**.**.**", ****);
try {
long timeTrace = System.currentTimeMillis();
int result = httpclient.executeMethod(post);
System.out.println(">> HTTP Response status code: "+result);
System.out.println(">> Response Time: "+(System.currentTimeMillis() - timeTrace));
.......
.......
.......
}
I appreciate any quick help and guidelines.
Here is the error I get
Mar 11, 2020 4:23:08 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: I/O exception (javax.net.ssl.SSLProtocolException) caught when processing
request: Connection reset
Mar 11, 2020 4:23:08 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: Retrying request
Mar 11, 2020 4:23:23 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: I/O exception (javax.net.ssl.SSLProtocolException) caught when processing
request: Connection reset
Mar 11, 2020 4:23:23 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: Retrying request
Mar 11, 2020 4:23:38 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: I/O exception (javax.net.ssl.SSLProtocolException) caught when processing
request: Connection reset
Mar 11, 2020 4:23:38 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: Retrying request
javax.net.ssl.SSLProtocolException: Connection reset
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:126)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.ja
va:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.ja
va:264)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.ja
va:259)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:137)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:11
52)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocke
tImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl
.java:402)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketIm
pl.java:716)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSoc
ketImpl.java:970)
at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStre
am.java:81)
at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.jav
a:142)
at java.base/java.io.FilterOutputStream.flush(FilterOutputStream.java:15
3)
at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequ
estBody(EntityEnclosingMethod.java:502)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodB
ase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.j
ava:993)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Htt
pMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMe
thodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav
a:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav
a:324)
at af.aib.etl.AttendanceETL.fetchAndParseAttendanceRecord(AttendanceETL.
java:99)
at af.aib.attendance.ApplicationStartPoint.main(ApplicationStartPoint.ja
va:28)
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRe
cord.java:448)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInput
Record.java:165)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
The local proxy and the server proxy were different that is why it was running fine on my local machine but not on the server, once I changed the proxy to the specific proxy which server was using then the application was working fine.
We have Java application with client Server architecture where client and server communicate with the help of RMI connection.
We have currently used Java 8 it working properly and respond quickly.
But we now planning to move on the Java 11 here also application working properly but comparatively slow as compare to java 8
we took the RMI logs where I can see sun.rmi.transport.DGCImpl_Stub dirty class taking time in Java 11.
Java 11 RMI logs
loadClass FINE: main: name = "java.rmi.server.UID", codebase = "",
defaultLoader =
jdk.internal.loader.ClassLoaders$AppClassLoader#4b85612c Nov 21, 2018
7:25:12 PM java.rmi.server.RMIClassLoader$2 loadClass FINER: main:
class "java.rmi.server.UID" found via defaultLoader, defined by null
Nov 21, 2018 7:25:12 PM sun.rmi.transport.DGCImpl_Stub dirty FINE:
main: free connection (reuse = true) Nov 21, 2018 7:27:31 PM
java.rmi.server.RemoteObjectInvocationHandler$MethodToHash_Maps$1 get
Java 8 RMI logs
FINE: main: name = "[B", codebase = "", defaultLoader =
sun.misc.Launcher$AppClassLoader#d716361 Nov 21, 2018 7:39:29 PM
sun.rmi.server.LoaderHandler loadClass FINER: main: class "[B" found
via defaultLoader, defined by null Nov 21, 2018 7:39:29 PM
sun.rmi.server.LoaderHandler loadClass FINE: main: name =
"java.rmi.server.UID", codebase = "", defaultLoader =
sun.misc.Launcher$AppClassLoader#d716361 Nov 21, 2018 7:39:29 PM
sun.rmi.server.LoaderHandler loadClass FINER: main: class
"java.rmi.server.UID" found via defaultLoader, defined by null Nov 21,
2018 7:39:29 PM sun.rmi.server.UnicastRef done FINE: main: free
connection (reuse = true) Nov 21, 2018 7:39:30 PM sun.rmi.server.Util
computeMethodHash
getting this error after hibernate outputs the data, any idea why this would be happening. please help!
Sep 07, 2016 12:07:00 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl stop
INFO: HHH000030: Cleaning up connection pool [jdbc:postgresql://localhost:5432/bendb]
Sep 07, 2016 12:07:00 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 3 * Server responded with a response on thread http-nio-8080-exec-5
3 < 200
3 < Access-Control-Allow-Methods: GET, POST, DELETE, PUT
3 < Access-Control-Allow-Origin: *
3 < Allow: OPTIONS
3 < Content-Type: application/json
Sep 07, 2016 12:07:00 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 4 * Server responded with a response on thread http-nio-8080-exec-5
4 < 500
Sorry found a bug in my code so apparently the code change i made was trying to map all the junction tables (collections) in that get user rest call hence jersey just breaks out while attempting to do that. un Commenting that code and just passing the normal data sets solved the issue.
I am developing a java application that executes ssh commands using Ganymed SSH-2
I need to produce full logs for each sequence of commands, e.g. zip file transfer, unzipping, zipping etc..
Having searched the source code for ch.ethz.ssh2.log.Logger i can set the boolean public static volatile boolean enabled = false; to true
this provides the following output
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: kex_algo=diffie-hellman-group-exchange-sha1
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: server_host_key_algo=ssh-rsa
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: enc_algo_client_to_server=aes128-ctr
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: enc_algo_server_to_client=aes128-ctr
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: mac_algo_client_to_server=hmac-sha1-96
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: mac_algo_server_to_client=hmac-sha1-96
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: comp_algo_client_to_server=none
Mar 05, 2015 10:17:25 AM ch.ethz.ssh2.log.Logger info
INFO: comp_algo_server_to_client=none
However I also require ALL level logging for command execution including file transfers.
How do i configure the Logger to produce all the information available?
A little late answer but maybe someone still needs this info.
I managed to get the debug statements visible like this:
public void enableFineLogging() {
try {
ch.ethz.ssh2.log.Logger.enabled = true;
String name = "myDynamicFileNamePart";
FileHandler fileHandler = new FileHandler("./logs/"
+ name + "_SFTP.log", 10000000, 1000, true);
fileHandler.setLevel(Level.FINE);
fileHandler.setFormatter(new SimpleFormatter());
final Logger app = Logger.getLogger("ch.ethz");
app.setLevel(Level.FINE);
app.addHandler(fileHandler);
app.setUseParentHandlers(false);
} catch (Exception e) {
// Catchalog
}
}
With result in file with:
marras 15, 2017 12:16:56 IP. org.slf4j.impl.JCLLoggerAdapter info
INFO: Client identity string: SSH-2.0-SSHJ_0.19.1
marras 15, 2017 12:16:56 IP. org.slf4j.impl.JCLLoggerAdapter info
INFO: Server identity string: SSH-2.0-OpenSSH_6.6.1
marras 15, 2017 12:16:56 IP. org.slf4j.impl.JCLLoggerAdapter debug
FINE: Setting <> to null
marras 15, 2017 12:16:56 IP. org.slf4j.impl.JCLLoggerAdapter debug
FINE: Sending SSH_MSG_KEXINIT
marras 15, 2017 12:16:56 IP. org.slf4j.impl.JCLLoggerAdapter debug
FINE: Setting <> to SOME
marras 15, 2017 12:16:56 IP. org.slf4j.impl.JCLLoggerAdapter debug
FINE: Awaiting <>
marras 15, 2017 12:16:56 IP. org.slf4j.impl.JCLLoggerAdapter debug
FINE: Received SSH_MSG_KEXINIT
Use ConsoleHandler if you wish logs in console.
Also closing the log file needs to be considered with fileHandler.close() after you quit logging.
Tune the log level by choosing from SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
I tried to use Amazon-SDK(Java) sample code S3TransferProgressSample.java to upload large files to Amazon-S3 storage (also posted here on AWS docs).
But when I am trying to upload 11 GB files, the upload is getting stuck at different points with the error message:
Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screenshot).
It looks like that after IOException occurs SDK is not able to retry the request (see below).
Does anyone encounter this? What is the best-practice to resolve this? Any code is appreciated.
INFO: Received successful response: 200, AWS Request ID:
2B66E7669E24DA75<br> Jan 15, 2011 6:44:46 AM
com.amazonaws.http.HttpClient execute<br> INFO: Sending Request: PUT
s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters:
(uploadId:
m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, )<br> Jan 15, 2011 6:45:10 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
Jan 15, 2011 6:45:10 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
INFO: Retrying request<br> Jan 15, 2011 6:45:12 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:12 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
Jan 15, 2011 6:45:12 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
Jan 15, 2011 6:45:13 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
com.amazonaws.http.HttpClient execute<br>
**WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.**<br> Jan 15, 2011 6:45:14 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:14 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:16 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:16 AM
com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:17 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:19 AM
com.amazonaws.http.HttpClient execute<br> ....<br> Jan 15, 2011
6:45:21 AM com.amazonaws.http.HttpClient handleResponse<br>
**INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007**<br> Jan 15, 2011 6:45:21 AM
com.amazonaws.http.HttpClient execute<br> ...<br> Jan 15, 2011 6:45:19
AM com.amazonaws.http.HttpClient execute<br> INFO: Sending Request:
DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt
Parameters:<br> ...<br> Jan 15, 2011 6:47:01 AM
com.amazonaws.http.HttpClient handleErrorResponse<br> INFO: Received
error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595,
AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload
does not exist. The upload ID may be invalid, or the upload may have
been aborted or completed.<br>
Try using the low level API.
This will give you far more control when things go wrong, as they are likely to do with an 11GB file.
Requests to and from S3 do fail from time to time. With the low level API, you'll be able to retry a part of the upload if it fails.
Refactoring the example in the Amazon docs a bit:
// Step 2: Upload parts.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
// Last part can be less than 5 MB. Adjust part size.
partSize = Math.min(partSize, (contentLength - filePosition));
// Create request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(existingBucketName).withKey(keyName)
.withUploadId(initResponse.getUploadId()).withPartNumber(i)
.withFileOffset(filePosition)
.withFile(file)
.withPartSize(partSize);
// repeat the upload until it succeeds.
boolean anotherPass;
do {
anotherPass = false; // assume everythings ok
try {
// Upload part and add response to our list.
partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
} catch (Exception e) {
anotherPass = true; // repeat
}
} while (anotherPass);
filePosition += partSize;
}
// Step 3: complete.
CompleteMultipartUploadRequest compRequest = new
CompleteMultipartUploadRequest(
existingBucketName,
keyName,
initResponse.getUploadId(),
partETags);
s3Client.completeMultipartUpload(compRequest);
Note: I am not a java developer so I could have messed things up syntactically, but hopefully this gets you going in the right direction. Also, you'll want to add in a 'retry counter' to prevent an endless loop if the upload repeatedly fails.
As a side note, 404 errors can be thrown if you try to do a multipart upload to a key that is already under a multipart upload.
I think you should try Multipart API supported by AWS.
Check this out : http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html
The answer of Geoff Appleford works for me.
However, I would add a && retryCount < MAX_RETRIES to the while loop control statement and increment of the retryCount on every exception caught inside the while.
Aviad
I wanted to add a comment to Geoff Appleford's answer but SO wouldn't allow me to. In general his answer to use low level API works fine but even if we do now have a do-while loop the way for loop is designed there is in-built retry logic. In his code snippet the file position increases only when there is a success otherwise you are uploading the same part again.