I tried to use Amazon-SDK(Java) sample code S3TransferProgressSample.java to upload large files to Amazon-S3 storage (also posted here on AWS docs).
But when I am trying to upload 11 GB files, the upload is getting stuck at different points with the error message:
Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screenshot).
It looks like that after IOException occurs SDK is not able to retry the request (see below).
Does anyone encounter this? What is the best-practice to resolve this? Any code is appreciated.
INFO: Received successful response: 200, AWS Request ID:
2B66E7669E24DA75<br> Jan 15, 2011 6:44:46 AM
com.amazonaws.http.HttpClient execute<br> INFO: Sending Request: PUT
s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters:
(uploadId:
m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, )<br> Jan 15, 2011 6:45:10 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
Jan 15, 2011 6:45:10 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
INFO: Retrying request<br> Jan 15, 2011 6:45:12 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:12 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
Jan 15, 2011 6:45:12 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
**INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error**<br>
Jan 15, 2011 6:45:13 AM
org.apache.commons.httpclient.HttpMethodDirector executeWithRetry<br>
INFO: Retrying request<br> Jan 15, 2011 6:45:13 AM
com.amazonaws.http.HttpClient execute<br>
**WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.**<br> Jan 15, 2011 6:45:14 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:14 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:16 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:16 AM
com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:17 AM
com.amazonaws.http.HttpClient execute<br> WARNING: Unable to execute
HTTP request: Unbuffered entity enclosing request can not be
repeated.<br> Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient
execute<br> WARNING: Unable to execute HTTP request: Unbuffered entity
enclosing request can not be repeated.<br> Jan 15, 2011 6:45:19 AM
com.amazonaws.http.HttpClient execute<br> ....<br> Jan 15, 2011
6:45:21 AM com.amazonaws.http.HttpClient handleResponse<br>
**INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007**<br> Jan 15, 2011 6:45:21 AM
com.amazonaws.http.HttpClient execute<br> ...<br> Jan 15, 2011 6:45:19
AM com.amazonaws.http.HttpClient execute<br> INFO: Sending Request:
DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt
Parameters:<br> ...<br> Jan 15, 2011 6:47:01 AM
com.amazonaws.http.HttpClient handleErrorResponse<br> INFO: Received
error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595,
AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload
does not exist. The upload ID may be invalid, or the upload may have
been aborted or completed.<br>
Try using the low level API.
This will give you far more control when things go wrong, as they are likely to do with an 11GB file.
Requests to and from S3 do fail from time to time. With the low level API, you'll be able to retry a part of the upload if it fails.
Refactoring the example in the Amazon docs a bit:
// Step 2: Upload parts.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
// Last part can be less than 5 MB. Adjust part size.
partSize = Math.min(partSize, (contentLength - filePosition));
// Create request to upload a part.
UploadPartRequest uploadRequest = new UploadPartRequest()
.withBucketName(existingBucketName).withKey(keyName)
.withUploadId(initResponse.getUploadId()).withPartNumber(i)
.withFileOffset(filePosition)
.withFile(file)
.withPartSize(partSize);
// repeat the upload until it succeeds.
boolean anotherPass;
do {
anotherPass = false; // assume everythings ok
try {
// Upload part and add response to our list.
partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
} catch (Exception e) {
anotherPass = true; // repeat
}
} while (anotherPass);
filePosition += partSize;
}
// Step 3: complete.
CompleteMultipartUploadRequest compRequest = new
CompleteMultipartUploadRequest(
existingBucketName,
keyName,
initResponse.getUploadId(),
partETags);
s3Client.completeMultipartUpload(compRequest);
Note: I am not a java developer so I could have messed things up syntactically, but hopefully this gets you going in the right direction. Also, you'll want to add in a 'retry counter' to prevent an endless loop if the upload repeatedly fails.
As a side note, 404 errors can be thrown if you try to do a multipart upload to a key that is already under a multipart upload.
I think you should try Multipart API supported by AWS.
Check this out : http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html
The answer of Geoff Appleford works for me.
However, I would add a && retryCount < MAX_RETRIES to the while loop control statement and increment of the retryCount on every exception caught inside the while.
Aviad
I wanted to add a comment to Geoff Appleford's answer but SO wouldn't allow me to. In general his answer to use low level API works fine but even if we do now have a do-while loop the way for loop is designed there is in-built retry logic. In his code snippet the file position increases only when there is a success otherwise you are uploading the same part again.
Related
I am using httpclient-3.0 library to parse data to cloud. When I run the application on my local machine (Windows 10), it works fine and the data gets parsed to the server and I receive success response, but when I deployed it on our server which runs on Windows server 2012 R2, it throws below error. I have used the same JDK as well I tried many ways like adding -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2" in my java.security jdk file, but still the issue is not resolved.
Here is my code
PostMethod post = new PostMethod(apiUrl);
post.setParameter("authtoken", authToken);
post.setParameter("dateFormat", dateTimeFormat);
post.setParameter("data", emloyeesAttendanceJsonArr.toString());
HttpClient httpclient = new HttpClient();
// Configuring proxy
httpclient.getHostConfiguration().setProxy("**.**.**.**", ****);
try {
long timeTrace = System.currentTimeMillis();
int result = httpclient.executeMethod(post);
System.out.println(">> HTTP Response status code: "+result);
System.out.println(">> Response Time: "+(System.currentTimeMillis() - timeTrace));
.......
.......
.......
}
I appreciate any quick help and guidelines.
Here is the error I get
Mar 11, 2020 4:23:08 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: I/O exception (javax.net.ssl.SSLProtocolException) caught when processing
request: Connection reset
Mar 11, 2020 4:23:08 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: Retrying request
Mar 11, 2020 4:23:23 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: I/O exception (javax.net.ssl.SSLProtocolException) caught when processing
request: Connection reset
Mar 11, 2020 4:23:23 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: Retrying request
Mar 11, 2020 4:23:38 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: I/O exception (javax.net.ssl.SSLProtocolException) caught when processing
request: Connection reset
Mar 11, 2020 4:23:38 PM org.apache.commons.httpclient.HttpMethodDirector execute
WithRetry
INFO: Retrying request
javax.net.ssl.SSLProtocolException: Connection reset
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:126)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.ja
va:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.ja
va:264)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.ja
va:259)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:137)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:11
52)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocke
tImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl
.java:402)
at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketIm
pl.java:716)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSoc
ketImpl.java:970)
at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStre
am.java:81)
at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.jav
a:142)
at java.base/java.io.FilterOutputStream.flush(FilterOutputStream.java:15
3)
at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequ
estBody(EntityEnclosingMethod.java:502)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodB
ase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.j
ava:993)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Htt
pMethodDirector.java:397)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMe
thodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav
a:396)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav
a:324)
at af.aib.etl.AttendanceETL.fetchAndParseAttendanceRecord(AttendanceETL.
java:99)
at af.aib.attendance.ApplicationStartPoint.main(ApplicationStartPoint.ja
va:28)
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRe
cord.java:448)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInput
Record.java:165)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
The local proxy and the server proxy were different that is why it was running fine on my local machine but not on the server, once I changed the proxy to the specific proxy which server was using then the application was working fine.
I've got a very weird problem with HTTP connection from java.
Description: from machine A(source machine) i trying to get http response from machine B(10.13.0.69).
Important part: machine B(target) is located behind VPN tunnel.
Machine B is reachable from machine A. I have no problems, if i try to access this URL from machine A OS (with curl for example):
[root#Test-LAPP01 test]# curl http://10.13.0.69:7878/testGet && echo ""
{"result":["\"AlertServer testGet ok\" "]}
Problems starts when i trying to perform same request from java program, that located at machine A: i getting SocketException: Unexpected end of file from server.
Detailed description:
[root#Test-LAPP01 test]# cat contest.java
import java.net.*;
import java.io.*;
public class contest {
public static void main(String[] args) throws Exception {
String url = "http://10.13.0.69:7878/testGet";
try {
URL alert = new URL(url);
HttpURLConnection con = (HttpURLConnection) alert.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", "chrome");
int responseCode = con.getResponseCode();
System.out.println("\nSending 'GET' request to URL : " + url);
System.out.println("Response Code : " + responseCode);
}
catch (IOException e) {
e.printStackTrace();
}
}
}
[root#Test-LAPP01 test]# curl http://10.13.0.69:7878/testGet && echo ""
{"result":["\"AlertServer testGet ok\" "]}
[root#Test-LAPP01 test]# java contest
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:848)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at contest.main(contest.java:12)
Note 1: from java app i CAN connect to external resourses, like google.com and have no issues with that.
Note 2: It works if i place target web server that i trying to reach (Machine B - 10.13.0.69) on same subnet with source machine A. So problem is somehow related to VPN.
I already tried to debug this and launched tcpdump on both machines and now i'm confused even more: i see all network requests on both sides.
So short description: i can reach destination URL from machine, but cant from java app.
UPD: updated code with #RedBoy recommendations. Now i've see this error:
апр 25, 2018 12:47:24 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://10.13.0.69:7878: The target server failed to respond
апр 25, 2018 12:47:24 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://10.13.0.69:7878
апр 25, 2018 12:47:55 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://10.13.0.69:7878: The target server failed to respond
апр 25, 2018 12:47:55 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://10.13.0.69:7878
апр 25, 2018 12:48:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://10.13.0.69:7878: The target server failed to respond
апр 25, 2018 12:48:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://10.13.0.69:7878
contest main org.apache.http.NoHttpResponseException: 10.13.0.69:7878 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at contest.contest.main(contest.java:25)
getting this error after hibernate outputs the data, any idea why this would be happening. please help!
Sep 07, 2016 12:07:00 PM org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl stop
INFO: HHH000030: Cleaning up connection pool [jdbc:postgresql://localhost:5432/bendb]
Sep 07, 2016 12:07:00 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 3 * Server responded with a response on thread http-nio-8080-exec-5
3 < 200
3 < Access-Control-Allow-Methods: GET, POST, DELETE, PUT
3 < Access-Control-Allow-Origin: *
3 < Allow: OPTIONS
3 < Content-Type: application/json
Sep 07, 2016 12:07:00 PM org.glassfish.jersey.filter.LoggingFilter log
INFO: 4 * Server responded with a response on thread http-nio-8080-exec-5
4 < 500
Sorry found a bug in my code so apparently the code change i made was trying to map all the junction tables (collections) in that get user rest call hence jersey just breaks out while attempting to do that. un Commenting that code and just passing the normal data sets solved the issue.
I am trying to access several websites by using Selenium in Java. Occasionally, I get an UnreachableBrowserException. I have read many threads about this error but it seems like there are many different causes of the error. I get the error about 1% of the time when I attempt to access a new page and I cannot find any similarities between occurrences. I currently am using Firefox, however I have also tried Internet Explorer and experienced similar errors. I am only opening one page at a time and have tried using the same window and completely quitting the driver before trying to access another page and either way the error still occurs. It is important to note I do not always get this error sometimes my code can run without this occurring. Here is the error message:
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}- http://127.0.0.1:7055: Permission denied: connect
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://127.0.0.1:7055
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://127.0.0.1:7055: Permission denied: connect
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://127.0.0.1:7055
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://127.0.0.1:7055: Permission denied: connect
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://127.0.0.1:7055
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://127.0.0.1:7055: Permission denied: connect
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://127.0.0.1:7055
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://127.0.0.1:7055: Permission denied: connect
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://127.0.0.1:7055
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://127.0.0.1:7055: Permission denied: connect
Jan 12, 2015 10:39:40 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://127.0.0.1:7055
Exception in thread "main" org.openqa.selenium.remote.UnreachableBrowserException: Error communicating with the remote browser. It may have died.
Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:03:00'
System info: host: '****', ip: '**.*.*.*', os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_60'
Driver info: driver.version: RemoteWebDriver
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:593)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:614)
at org.openqa.selenium.remote.RemoteWebDriver.quit(RemoteWebDriver.java:468)
at scrape.Scraper.killInstance(Scraper.java:162)
at scrape.Updater.main(Updater.java:93)
Caused by: java.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:123)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.openqa.selenium.remote.HttpCommandExecutor.fallBackExecute(HttpCommandExecutor.java:215)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.execute(NewProfileExtensionConnection.java:165)
at org.openqa.selenium.firefox.FirefoxDriver$LazyCommandExecutor.execute(FirefoxDriver.java:362)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:572)
... 4 more
How can I prevent this error or at least catch the error and deal with it effectively?
UnreachableBrowserExceptions can happen for multiple reasons - the most obvious is that the browser was closed, either in code or physically in the GUI, and then the code attempted to access it. Often, like in your case, they are caused by socket errors. This can mean, again, multiple things - your program tried to open too many sockets, it couldn't connect to a remote website, and others.
What I would suggest doing in a situation like this is waiting a short time, then retrying to see if the exception is still thrown. Sometimes these situations resolve themselves and your program can recover.
Here is some code to do that. It keeps retrying as long as the UnreachableBrowserException is thrown and the number of retries is below some limit that you set. If it hits the retry limit and the exception is still being thrown, it closes the browser and restarts it, resetting the retry count to 0. There is also a restart counter, to make sure that if for some reason restarting the browser doesn't help, you don't loop endlessly through running code -> exception -> wait -> retry -> hit retry limit, restart browser -> run code -> exception. Here, exceeding the restart limit (or successfully accessing the browser) will break out of the loop.
If you want more help, let me know.Hope this is helpful!
WebDriver driver = new FirefoxDriver(); //or whatever you're using
boolean worked = false;
int numredos = 0;
final int REDO_LIMIT = 3; //or however many times you want to retry before giving up
final int RESTART_LIMIT = 3; //or however many times you want to restart the browser b/f terminating
int numrestarts = 0;
boolean restart = false;
do
{
try{
if(restart)
{
driver = new FirefoxDriver();
numrestarts++;
}
//RUN YOUR BROWSER CODE HERE
worked = true;
}
//if the browser becomes unreachable (probably b/c of a socket issue),
// write the error to the log and then sleep for 10 seconds
//if we've already retried the set limit number of times, restart the browser and try again
catch (UnreachableBrowserException ube)
{
worked = false;
if(numredos >= REDO_LIMIT)
{
//if you've already restarted the browser too many times, it will set it to null
//and return an error code. If not, it will set the restart flag so it will be restarted on the next iteration.
//try quitting. If it can't do it, it's already dead; just set it to null
//(set it to null either way, just in case)
try
{
driver.quit();
}
catch(Exception j)
{
errorwriter.println(j);
}
driver = null;
if(numrestarts < RESTART_LIMIT)
{
//log that you're restarting the driver (not coded here), then set the restart flag to true. This will cause the browser to be restarted after falling out of the catch block
numredos = 0;
restart = true;
}
}
else
{
//print details of the exception to the error file
errorfile.println("\n\n\n");
//timestamp, and some exception details - you can decide which you want
errorfile.println(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(Calendar.getInstance().getTime()));
errorfile.println(s.getClass());
errorfile.println(s.getMessage());
errorfile.println("Cause: " + s.getCause());
errorfile.flush();
//now sleep for some number of seconds - here 10
try
{
TimeUnit.SECONDS.sleep(10);
}
catch(InterruptedException e)
{
System.out.println("waiting after socket crash interrupted");
}
numredos++;
}
}
}while(!worked && numredos <= REDO_LIMIT && numrestarts <= RESTART_LIMIT);
I'm trying to use the Java API to upload large files, but I need to also add user metadata. If I just use
TransferManager tm = new TransferManager(new BasicAWSCredentials(accessKey, secretKey));
Upload upload = tm.upload(AmazonS3Manager.bucketName, imageKey, file);
upload.waitForCompletion();
Then everything works fine, but if I use:
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(file.length());
metadata.addUserMetadata("My key", "My value");
FileInputStream input = new FileInputStream(file);
Upload upload = tm.upload(AmazonS3Manager.bucketName, imageKey, input, metadata);
Then it doesn't work, and I get the following output in the console:
Jul 5, 2011 4:33:15 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: POST https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploads: null, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, x-amz-meta-length: 312612077, )
Jul 5, 2011 4:33:16 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 2A5B3538795CE730
Jul 5, 2011 4:33:16 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, partNumber: 1, ) Headers: (Content-Length: 5242880, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 200, AWS Request ID: 5E5AF291FBDBDD36
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: PUT https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, partNumber: 2, ) Headers: (Content-Length: 5242880, Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:00 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: I/O exception (java.io.IOException) caught when processing request: Read error
Jul 5, 2011 4:34:00 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
INFO: Retrying request
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
WARNING: Unable to execute HTTP request: null
Jul 5, 2011 4:34:00 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Sending Request: DELETE https://mybucket.s3.amazonaws.com /test.jpg Parameters: (uploadId: rwUlbXqtRyMUWiVYKzGqRQH90fVLi9_|Secret Key Removed|_w--, ) Headers: (Content-Type: application/x-www-form-urlencoded; charset=utf-8, )
Jul 5, 2011 4:34:01 PM com.amazonaws.http.AmazonHttpClient handleResponse
INFO: Received successful response: 204, AWS Request ID: 0EFC3F8D0FA6097E
Any help greatly appreciated!
You may also need to set content type and, possibly, checksum as it can't be determined from input stream. This has some information for similar situation http://docs.amazonwebservices.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#putObject(com.amazonaws.services.s3.model.PutObjectRequest)
If you are trying to upload more than 5 GB file with updating metadata then amazon not allowing to do so. Because amazon s3 not providing copy operation for more then 5Gb file.
Amazon S3 recently started to support copy operation wtih multipart operation so for that i can not say you something yet.
https://forums.aws.amazon.com/thread.jspa?messageID=256605𾩝
Thanks
You can use following code for your file upload:
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(file.length());
metadata.addUserMetadata("My key", "My value");
FileInputStream input = new FileInputStream(file);
PutObjectRequest putObjectRequest = new PutObjectRequest(AmazonS3Manager.bucketName, imageKey, input, metadata);
s3client.putObject(putObjectRequest.withCannedAcl(CannedAccessControlList.PublicRead));
Thanks