We Keep Coding

FTPClient throws Connection reset by peer during storeFile

I am trying to upload file to ftp server using Java.
What i got so far is
fun uploadData(): Boolean {
val ftpClient = FTPSClient()
ftpClient.addProtocolCommandListener(
PrintCommandListener(PrintWriter(OutputStreamWriter(System.out, "UTF-8")), true)
)
ftpClient.connect(ftpProperties.server)
try {
ftpClient.enterLocalPassiveMode()
ftpClient.login(ftpProperties.username, ftpProperties.password).takeIf { !it }?.let {
log.error("cannot login to ftp")
throw Exception("cannot login to ftp")
}
ftpClient.enterLocalPassiveMode()
ftpClient.soTimeout = 10000
ftpClient.dataTimeout = Duration.ofSeconds(10)
ftpClient.execPBSZ(0)
ftpClient.execPROT("P")
ftpClient.changeWorkingDirectory("/")
ftpClient.setFileType(FTP.BINARY_FILE_TYPE)
ftpClient.enterLocalPassiveMode()
} catch (ie: IOException) {
log.error("ftp initialization error", ie)
throw Exception("ftp initialization error")
}
val remoteFile = "top.txt.gz"
val data = FileInputStream(File("top.txt.gz"))
var done = false
try {
done = ftpClient.storeFile(remoteFile, data)
} catch (e: Exception) {
log.error(e) { "error" }
}
data.close()
if (done) {
return true
}
log.error { "${ftpClient.replyCode} ${ftpClient.replyString}" }
throw RuntimeException("File not stored $remoteFile")
}
Code does not seem to be working properly. After starting it i get error code on line with code:
done = ftpClient.storeFile(remoteFile, data)
org.apache.commons.net.io.CopyStreamException: IOException caught while copying.
Caused by: java.net.SocketException: Connection reset by peer
I could not find anything wrong in ftp log
220 Private FTP server
AUTH TLS
234 Proceed with negotiation.
USER *******
331 Please specify the password.
PASS *******
230 Login successful.
PBSZ 0
200 PBSZ set to 0.
PROT P
200 PROT now Private.
CWD /
250 Directory successfully changed.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (x,x,x,x,x,x).
STOR top.txt.gz
150 Ok to send data.
I am able to connect and upload files to that ftp server from the same machine that i start Java/Kotlin code, using filezilla.
Filezilla log looks like that:
Status: Connection established, waiting for welcome message...
Response: 220 Private FTP server
Command: AUTH TLS
Response: 234 Proceed with negotiation.
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Command: USER k8s_search
Response: 331 Please specify the password.
Command: PASS ******************
Response: 230 Login successful.
Status: Server does not support non-ASCII characters.
Command: PBSZ 0
Response: 200 PBSZ set to 0.
Command: PROT P
Response: 200 PROT now Private.
Status: Logged in
Status: Starting upload of /home/x/x/top.txt.gz
Command: CWD /
Response: 250 Directory successfully changed.
Command: TYPE I
Response: 200 Switching to Binary mode.
Command: PASV
Response: 227 Entering Passive Mode (x,x,x,x,x,x).
Command: STOR top.txt.gz
Response: 150 Ok to send data.
Response: 226 Transfer complete.
Status: File transfer successful, transferred 2.6 MB in 1 second
I connot figure out if problem is in my code or is it anything else. Filezilla working correctly indicates that problem lies in code.

How can I get notified when money has been sent to a particular Bitcoin address on a local regtest network?

I want to programmatically detect whenever someone sends Bitcoin to some address. This happens on a local testnet which I start using this docker-compose.yml file.
Once the local testnet runs, I create a new address using
docker exec -it minimal-crypto-exchange_node_1 bitcoin-cli getnewaddress
Let's say it returns 2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181.
I put this address into the following Java code:
import org.bitcoinj.core.Address;
import org.bitcoinj.core.Coin;
import org.bitcoinj.core.NetworkParameters;
import org.bitcoinj.core.Transaction;
import org.bitcoinj.wallet.Wallet;
import org.bitcoinj.wallet.listeners.WalletCoinsReceivedEventListener;
public class WalletObserver {
public void init() {
final NetworkParameters netParams = NetworkParameters.fromID(NetworkParameters.ID_REGTEST);
try {
final Wallet wallet = Wallet.createBasic(netParams);
wallet.addWatchedAddress(Address.fromString(netParams, "2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181"));
wallet.addCoinsReceivedEventListener(new WalletCoinsReceivedEventListener() {
#Override
public void onCoinsReceived(final Wallet wallet, final Transaction transaction, final Coin prevBalance, final Coin newBalance) {
System.out.println("Heyo!");
}
});
}
catch (Exception exception) {
exception.printStackTrace();
}
}
}
Then I start the Java application with this class.
Then I send some test Bitcoin to the address in question:
% docker exec -it minimal-crypto-exchange_node_1 bitcoin-cli sendtoaddress 2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181 0.5
068c377bab961356ad9a3919229a764aa929711c68aefd5dbd4c7c348eef3406
If I go to http://localhost:3002/tx/068c377bab961356ad9a3919229a764aa929711c68aefd5dbd4c7c348eef3406, I see that the transaction details.
However, the breakpoint in the listener (onCoinsReceived method) never activates.
How do I need to modify my code and/or the commands I use to send test BTC so that whenever money is received by that account, onCoinsReceived method is called? Is there a place where I can tell Wallet or NetworkParameters that I want to connect to localhost?
I am using version 0.15.10 of bitcoinj-core.
Update 1:
I modified docker-compose.yml and added following port mappings:
ports:
- "51001:50001"
- "51002:50002"
- "19001:19001"
- "19000:19000"
- "28332:28332"
Then I rewrote the init method so that I can connect to localhost and specify the port:
public class WalletObserver {
public void init() {
final LocalTestNetParams netParams = new LocalTestNetParams();
netParams.setPort(50001);
try {
final WalletAppKit kit = new WalletAppKit(netParams, new File("."), "_minimalCryptoExchangeBtcWallet");
kit.setAutoSave(true);
kit.connectToLocalHost();
kit.startAsync();
kit.awaitRunning(); // I never get past this point
kit.peerGroup().addPeerDiscovery(new DnsDiscovery(netParams));
kit.wallet().addWatchedAddress(Address.fromString(netParams, "2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181"));
kit.wallet().addCoinsReceivedEventListener(new WalletCoinsReceivedEventListener() {
#Override
public void onCoinsReceived(final Wallet wallet, final Transaction transaction, final Coin prevBalance, final Coin newBalance) {
System.out.println("Heyo!");
}
});
}
catch (Exception exception) {
exception.printStackTrace();
}
}
LocalTestNetParams allows to specify the port:
package com.dpisarenko.minimalcryptoexchange.logic.btc;
import org.bitcoinj.params.RegTestParams;
public class LocalTestNetParams extends RegTestParams {
public void setPort(final int newPort) {
this.port = newPort;
}
}
I tried all of the aforementioned ports in netParams.setPort(50001);.
In all cases I get the following messages after kit.awaitRunning();:
22:16:34.245 [PeerGroup Thread] INFO org.bitcoinj.core.PeerGroup - Attempting connection to [10.10.1.218]:50001 (0 connected, 1 pending, 1 max)
22:16:34.265 [NioClientManager] WARN org.bitcoinj.net.NioClientManager - Failed to connect with exception: java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:579)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820)
at org.bitcoinj.net.NioClientManager.handleKey(NioClientManager.java:64)
at org.bitcoinj.net.NioClientManager.run(NioClientManager.java:122)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:66)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119)
at org.bitcoinj.utils.ContextPropagatingThreadFactory$1.run(ContextPropagatingThreadFactory.java:51)
at java.base/java.lang.Thread.run(Thread.java:830)
22:16:34.267 [NioClientManager] INFO org.bitcoinj.core.PeerGroup - [10.10.1.218]:50001: Peer died (0 connected, 0 pending, 1 max)
22:16:34.267 [PeerGroup Thread] INFO org.bitcoinj.core.PeerGroup - Peer discovery took 21.84 μs and returned 0 items from 0 discoverers
22:16:34.269 [PeerGroup Thread] INFO org.bitcoinj.core.PeerGroup - Waiting 1502 ms before next connect attempt to [10.10.1.218]:50001
22:16:35.776 [PeerGroup Thread] INFO org.bitcoinj.core.PeerGroup - Attempting connection to [10.10.1.218]:50001 (0 connected, 1 pending, 1 max)
22:16:35.778 [NioClientManager] WARN org.bitcoinj.net.NioClientManager - Failed to connect with exception: java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:579)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820)
at org.bitcoinj.net.NioClientManager.handleKey(NioClientManager.java:64)
at org.bitcoinj.net.NioClientManager.run(NioClientManager.java:122)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:66)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119)
at org.bitcoinj.utils.ContextPropagatingThreadFactory$1.run(ContextPropagatingThreadFactory.java:51)
at java.base/java.lang.Thread.run(Thread.java:830)
22:16:35.778 [NioClientManager] INFO org.bitcoinj.core.PeerGroup - [10.10.1.218]:50001: Peer died (0 connected, 0 pending, 1 max)
22:16:35.779 [PeerGroup Thread] INFO org.bitcoinj.core.PeerGroup - Peer discovery took 8.752 μs and returned 0 items from 0 discoverers
10.10.1.218 seems to be generated by InetAddress.getLocalHost() in org.bitcoinj.kits.WalletAppKit#connectToLocalHost:
public WalletAppKit connectToLocalHost() {
try {
InetAddress localHost = InetAddress.getLocalHost();
return this.setPeerNodes(new PeerAddress(this.params, localHost, this.params.getPort()));
} catch (UnknownHostException var2) {
throw new RuntimeException(var2);
}
}
Update 1:
I tried to use network_mode: "host".
If I add it to node as in
node:
image: ulamlabs/bitcoind-custom-regtest:latest
network_mode: "host"
I get the following error when I run docker-compose up -d:
minimal-crypto-exchange % docker-compose up -d
Creating network "minimal-crypto-exchange_default" with the default driver
Creating minimal-crypto-exchange_postgres_1 ... done
Creating minimal-crypto-exchange_geth_1 ...
Creating minimal-crypto-exchange_node_1 ... done
Creating minimal-crypto-exchange_electrumx_1 ...
Creating minimal-crypto-exchange_electrumx_1 ... error
ERROR: for minimal-crypto-exchange_electrumx_1 Cannot start service electrumx: driver fail
Creating minimal-crypto-exchange_geth_1 ... done
f68d0f25a0512399877bc55434513def810649e4fcf31a5a88ca3292d34): Error starting userland proxy: listen tcp4 0.0.0.0:28332: bind: address already in use
Creating minimal-crypto-exchange_blockscout_1 ... done
ERROR: for electrumx Cannot start service electrumx: driver failed programming external connectivity on endpoint minimal-crypto-exchange_electrumx_1 (8eaa4f68d0f25a0512399877bc55434513def810649e4fcf31a5a88ca3292d34): Error starting userland proxy: listen tcp4 0.0.0.0:28332: bind: address already in use
ERROR: Encountered errors while bringing up the project.
If I add it to electrumx part as in
electrumx:
image: lukechilds/electrumx:latest
network_mode: "host"
I get another error:
minimal-crypto-exchange % docker-compose up -d
minimal-crypto-exchange_postgres_1 is up-to-date
minimal-crypto-exchange_geth_1 is up-to-date
Recreating minimal-crypto-exchange_node_1 ...
Recreating minimal-crypto-exchange_node_1 ... done
Recreating minimal-crypto-exchange_electrumx_1 ...
ERROR: for minimal-crypto-exchange_electrumx_1 "host" network_mode is incompatible with port_bindings
ERROR: for electrumx "host" network_mode is incompatible with port_bindings
Traceback (most recent call last):
File "docker-compose", line 3, in <module>
File "compose/cli/main.py", line 81, in main
File "compose/cli/main.py", line 203, in perform_command
File "compose/metrics/decorator.py", line 18, in wrapper
File "compose/cli/main.py", line 1186, in up
File "compose/cli/main.py", line 1166, in up
File "compose/project.py", line 697, in up
File "compose/parallel.py", line 108, in parallel_execute
File "compose/parallel.py", line 206, in producer
File "compose/project.py", line 679, in do
File "compose/service.py", line 579, in execute_convergence_plan
File "compose/service.py", line 499, in _execute_convergence_recreate
File "compose/parallel.py", line 108, in parallel_execute
File "compose/parallel.py", line 206, in producer
File "compose/service.py", line 494, in recreate
File "compose/service.py", line 612, in recreate_container
File "compose/service.py", line 330, in create_container
File "compose/service.py", line 939, in _get_container_create_options
File "compose/service.py", line 1014, in _get_container_host_config
File "docker/api/container.py", line 598, in create_host_config
File "docker/types/containers.py", line 338, in __init__
docker.errors.InvalidArgument: "host" network_mode is incompatible with port_bindings
[44262] Failed to execute script docker-compose
Update 2:
If I comment out port bindings as in
electrumx:
image: lukechilds/electrumx:latest
network_mode: host
links:
- node
# Port settings see https://github.com/ulamlabs/bitcoind-custom-regtest
# ports:
# - "51001:50001"
# - "51002:50002"
# - "19001:19001"
# - "19000:19000"
# - "28332:28332"
and run docker-compose up -d I get
% docker-compose up -d
Creating network "minimal-crypto-exchange_default" with the default driver
Creating minimal-crypto-exchange_geth_1 ...
Creating minimal-crypto-exchange_postgres_1 ... done
Creating minimal-crypto-exchange_node_1 ... done
Creating minimal-crypto-exchange_electrumx_1 ... error
Creating minimal-crypto-exchange_geth_1 ... done
ERROR: for minimal-crypto-exchange_electrumx_1 Cannot create container for service electrumx: conflicting options: host type networking can't be used with links. This would result in undefined behavior
Creating minimal-crypto-exchange_blockscout_1 ... done
ERROR: for electrumx Cannot create container for service electrumx: conflicting options: host type networking can't be used with links. This would result in undefined behavior
ERROR: Encountered errors while bringing up the project.
Update 3: I assume that the root of the error is that in my Java code I try to connect to the ElectrumX server instead of the actual Bitcoin node (node in docker-compose.yml).
Update 4:
I changed docker-compose.yml as follows:
node:
image: ulamlabs/bitcoind-custom-regtest:latest
# For ports used by node see
# https://github.com/ulamlabs/bitcoind-custom-regtest/blob/master/bitcoin.conf
ports:
- "19001:19001"
- "19000:19000"
- "28332:28332"
electrumx:
image: lukechilds/electrumx:latest
links:
- node
# Port settings see https://github.com/ulamlabs/bitcoind-custom-regtest
ports:
- "51001:50001"
- "51002:50002"
# - "19001:19001"
# - "19000:19000"
# - "28332:28332"
Now I am getting different errors (full log available here):
11:33:51.865 [NioClientManager] INFO org.bitcoinj.core.PeerGroup - [192.168.10.208]:19000: Peer died (0 connected, 0 pending, 1 max)
11:33:51.865 [NioClientManager] INFO org.bitcoinj.core.PeerGroup - Not yet setting download peer because there is no clear candidate.
11:33:51.865 [NioClientManager] DEBUG org.bitcoinj.core.BitcoinSerializer - Received 168 byte 'alert' message: 60010000000000000000000000ffffff7f00000000ffffff7ffeffff7f01ffffff7f00000000ffffff7f00ffffff7f002f555247454e543a20416c657274206b657920636f6d70726f6d697365642c2075706772616465207265717569726564004630440220653febd6410f470f6bae11cad19c48413becb1ac2c17f908fd0fd53bdc3abd5202206d0e9c96fe88d4a0f01ed9dedae2b6f9e00da94cad0fecaae66ecf689bf71b50
11:33:51.866 [PeerGroup Thread] INFO org.bitcoinj.core.PeerGroup - Waiting 999 ms before next connect attempt to [127.0.0.1]:19000
11:33:51.866 [NioClientManager] DEBUG org.bitcoinj.core.Peer - Received alert from peer Peer{[192.168.10.208]:19000, version=70015, subVer=/Satoshi:0.19.1(bitcore)/, services=1033 (NETWORK, WITNESS, NETWORK_LIMITED), time=2021-11-06 11:33:52, height=5}: URGENT: Alert key compromised, upgrade required
11:33:51.867 [NioClientManager] WARN org.bitcoinj.net.ConnectionHandler - Error handling SelectionKey: java.nio.channels.CancelledKeyException
java.nio.channels.CancelledKeyException: null
at java.base/sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:71)
at java.base/sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:130)
at java.base/java.nio.channels.SelectionKey.isWritable(SelectionKey.java:377)
at org.bitcoinj.net.ConnectionHandler.handleKey(ConnectionHandler.java:244)
at org.bitcoinj.net.NioClientManager.handleKey(NioClientManager.java:86)
at org.bitcoinj.net.NioClientManager.run(NioClientManager.java:122)
at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:66)
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119)
at org.bitcoinj.utils.ContextPropagatingThreadFactory$1.run(ContextPropagatingThreadFactory.java:51)
at java.base/java.lang.Thread.run(Thread.java:830)
Update 5:
Someone suggested (in a now removed comment) that in the output of the application there is this Peer does not support bloom filtering message:
11:32:43.482 [NioClientManager] INFO org.bitcoinj.core.Peer - Peer{[127.0.0.1]:19000, version=70015, subVer=/Satoshi:0.19.1(bitcore)/, services=1033 (NETWORK, WITNESS, NETWORK_LIMITED), time=2021-11-06 11:32:43, height=4}: Peer does not support bloom filtering.
So I tried to fork the original image and change the bitcoin.conf file to enable Bloom filtering:
peerbloomfilters=1
When I run docker build -t mentiflectax/bitcoind-custom-regtest:latest . I get the following error message (part of remaining output can be found here):
#13 922.4 g++: fatal error: Killed signal terminated program cc1plus
#13 922.4 compilation terminated.
#13 922.4 make[2]: *** [Makefile:8044: libbitcoin_server_a-init.o] Error 1
#13 922.4 make[2]: *** Waiting for unfinished jobs....
#13 965.8 make[2]: Leaving directory '/bitcoin-0.19.1/src'
#13 965.8 make[1]: *** [Makefile:13765: all-recursive] Error 1
#13 965.9 make[1]: Leaving directory '/bitcoin-0.19.1/src'
#13 965.9 make: *** [Makefile:776: all-recursive] Error 1
------
executor failed running [/bin/sh -c tar -xzf *.tar.gz && cd bitcoin-${BITCOIN_VERSION} && sed -i 's/consensus.nSubsidyHalvingInterval = 150/consensus.nSubsidyHalvingInterval = 210000/g' src/chainparams.cpp && ./autogen.sh && ./configure LDFLAGS=-L`ls -d /opt/db`/lib/ CPPFLAGS=-I`ls -d /opt/db`/include/ --prefix=/opt/bitcoin --disable-man --disable-tests --disable-bench --disable-ccache --with-gui=no --enable-util-cli --with-daemon && make -j4 && make install && strip /opt/bitcoin/bin/bitcoin-cli && strip /opt/bitcoin/bin/bitcoind]: exit code: 2
Update 6: The correct port seems to be 19000.
If I use port 19001, I get following errors after kit.awaitRunning():
INFO org.bitcoinj.core.PeerSocketHandler - [127.0.0.1]:19001: Timed out
Full log output is available here.

I haven't tested your full setup with electrumx and the ethereum stuff present in your docker-compose file, but regarding your problem, the following steps worked properly, and I think it will do as well in your complete setup.
I ran with docker a bitcoin node based in the ulamlabs/bitcoind-custom-regtest:latest image you provided:
docker run -p 18444:19000 -d ulamlabs/bitcoind-custom-regtest:latest
As you can see, I exposed the image internal port 19000 as the default port for RegTestParams, 18444. From the point of view of our client, with this setup, basically it will look like as if we were running the bitcoin daemon in the host. Using your LocalTestNetParams class and providing the port 19000 as you indicated should do the trick as well.
Then, according to the feedback you provided in the question, I manually edited the daemon configuration of the bitcoin node in /root/.bitcoin/bitcoin.conf using bash and vi:
docker exec -it 0aa2e863cd9927 bash
And included the following configuration:
peerbloomfilters=1
After restart the container, I got a new address:
docker exec -it 0aa2e863cd9927 bitcoin-cli -regtest getnewaddress
Let's assume that the new address is the one you provided in the question:
2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181
Then, as suggested in the Bitcoin documentation, in order to avoid an insufficient funds error, I generated 101 blocks to this address:
docker exec -it 0aa2e863cd9927 bitcoin-cli -regtest generatetoaddress 101 2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181
I used generatetoaddress and not generate because since Bitcoin 0.19.0 the option is no longer valid.
Next, I prepared a simple Java program, based in the information you provided and this example from the Bitcoinj library documentation:
import java.io.File;
import org.bitcoinj.core.Address;
import org.bitcoinj.core.NetworkParameters;
import org.bitcoinj.kits.WalletAppKit;
import org.bitcoinj.params.RegTestParams;
public class Kit {
public static void main(String[] args) {
Kit kit = new Kit();
kit.run();
}
private synchronized void run(){
NetworkParameters params = RegTestParams.get();
WalletAppKit kit = new WalletAppKit(params, new File("."), "walletappkit-example");
kit.connectToLocalHost();
kit.startAsync();
kit.awaitRunning();
kit.wallet().addWatchedAddress(Address.fromString(params, "2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181"));
kit.wallet().addCoinsReceivedEventListener((wallet, tx, prevBalance, newBalance) -> {
System.out.println("-----> coins resceived: " + tx.getTxId());
});
while (true) {
try {
this.wait(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
I used a simple while loop to keep the problem running; of course, if will be probably unnecessary in an actual setup as it seems you are using Spring Boot.
Then, if you send some bitcoins to this address:
docker exec -it 0aa2e863cd9927 bitcoin-cli -regtest sendtoaddress 2N23tWAFEtBtTgxNjBNmnwzsiPdLcNek181 0.00001
0f972642713c72ae0fe03fe51818b9ea4d483720b69b90e795f35eb80a587c26
The listener should be invoked:
2021-11-09 23:51:20.537 INFO [NioClientManager][Wallet] Received a pending transaction 0f972642713c72ae0fe03fe51818b9ea4d483720b69b90e795f35eb80a587c26 that spends 0.00 BTC from our own wallet, and sends us 0.00001 BTC
2021-11-09 23:51:20.537 INFO [NioClientManager][Wallet] commitTx of 0f972642713c72ae0fe03fe51818b9ea4d483720b69b90e795f35eb80a587c26
...
2021-11-09 23:51:20.537 INFO [NioClientManager][Wallet] ->pending: 0f972642713c72ae0fe03fe51818b9ea4d483720b69b90e795f35eb80a587c26
2021-11-09 23:51:20.537 INFO [NioClientManager][Wallet] Estimated balance is now: 0.00001 BTC
-----> coins resceived: 0f972642713c72ae0fe03fe51818b9ea4d483720b69b90e795f35eb80a587c26
2021-11-09 23:51:20.538 INFO [NioClientManager][WalletFiles] Saving wallet; last seen block is height 165, date 2021-11-09T22:50:48Z, hash 23451521947bc5ff098c088ae0fc445becca8837d39ee8f6dd88f2c47ad5ac23
2021-11-09 23:51:20.543 INFO [NioClientManager][WalletFiles] Save completed in 4.736 ms
There is still a problem you mentioned that I haven't had the opportunity to test, and it is creating a new Docker image in which the peerbloomfilters configuration would be configured properly without modifying the actual container state. I think the compilation problem you indicated could be related to this issue, basically, that the container didn't have enough resources to perform the process. If you are using macOS and Docker for Mac, try tweaking the amount of memory available to your containers, it may be of help. A change in the base alpine image used can motivate the problem also. I will try digging into the issue as well.

How to log "220 server ready" message after connecting with Apache FTPSClient

I'm writing a simple FTPS client using Apache Commons Net. Everything works fine, but the initial "server ready" message isn't being logged. I call connect() and do a getReplyString() immediately afterward, and all it logs is 234 Enabling TLS, awaiting negotiations even though I know the server is sending a message after it connects:
220 FTPS (Version Fri Jan 22 19:49:50 2016) server ready.
I've even used getReplyStrings() to see if there are more than one message, but it still only shows the one 234 Enabling TLS, awaiting negotiations message. I have a feeling it's because that's the only reply to the AUTH TLS command that gets automatically issued when I connect (explicit TLS).
I made a hack that scrapes the ProtocolCommandListener output and grabs the message that way, but I'd rather do it correctly. I have a feeling I'm missing something simple. Can anyone help?
Code:
FTPSClient ftpsclient = new FTPSClient("TLSv1.2");
// my hacked attempt to get the message
ByteArrayOutputStream baos = new ByteArrayOutputStream();
PrintStream ps = new PrintStream(baos,true,"utf-8");
ftpsclient.addProtocolCommandListener(new PrintCommandListener(ps, true));
logMessage("Connecting to FTP-SSL server [" + HOST + "] on port [" + PORT + "]...", 1);
ftpsclient.connect(HOST, PORT);
_logger.debug("Output: "+ baos.toString());
if (!FTPReply.isPositiveCompletion(ftpsclient.getReplyCode())) {
throw new Exception("Failed to connect to [" + HOST + "]\n" + ftpsclient.getReplyString());
}
logMessage(ftpsclient.getReplyString(), 1);
Output:
[Jan 22, 16:49:50] INFO logMessage() - Connecting to FTP-SSL server [HOST] on port [PORT]...
[Jan 22, 16:49:55] DEBUG - Output: 220 FTPS (Version Fri Jan 22 19:49:50 2016) server ready.
AUTH TLS
234 Enabling TLS, awaiting negotiations.
[Jan 22, 16:49:55] INFO EDIDataExtractor.logMessage() - 234 Enabling TLS, awaiting negotiations.

How should I handle "java.net.SocketException: Connection reset" in multithread AWS S3 file upload?

I have a ThreadPoolExecutorService to which I'm submitting runnable jobs that are uploading large (1-2 GB) files to Amazon's S3 file system, using the AWS Java SDK. Occasionally one of my worker threads will report a java.net.SocketException with "Connection reset" as the cause and then die.
AWS doesn't use checked exceptions so I actually can't catch SocketException directly---it must be wrapped somehow. My question is how I should deal with this problem so I can retry any problematic uploads and increase the reliability of my program.
Would the Multipart Upload API be more reliable?
Is there some exception I can reliably catch to enable retries?
Here's the stack trace. The com.example.* code is mine. Basically what the DataProcessorAWS call does is call putObject(String bucketName, String key, File file) on an instance of AmazonS3Client that's shared across threads.
14/12/11 18:43:17 INFO http.AmazonHttpClient: Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:169)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:119)
at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:102)
at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153)
at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:685)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3697)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1434)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1294)
at com.example.DataProcessorAWS$HitWriter.close(DataProcessorAWS.java:156)
at com.example.DataProcessorAWS$Processor.run(DataProcessorAWS.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

For that you have to use only AmazonS3Client not Trasfermanager for upload and download
U have to configure AmazonS3Client with following properties:
1. connectionTimeout=50000 in ms
2.maxConnections=500
3.socketTimeout=50000 in ms
4.maxErrorRetry=10
For upload use
AmazonS3Client.putObject(bucketName, key, inputFile);
For download use
S3Object s3Object = AmazonS3Client.getObject(new GetObjectRequest(bucketName, key));`
InputStream downloadStream = s3Object.getObjectContent();
and store that stream by reading bytes.

'Connection reset' means the connection is closed. Close the socket, or whatever higher-level construct you're using. Probably the server has decided the upload is too large, or it's overloaded, or something. Whether you can retry the operation is something only you can know.

In my case, I am saving a object with some metadata. During the prototype, we added a system metadata "time-created", and that consistently resulted into the similar error as above in the question. And it is not retry-able as it intends to modify a system metadata. But according to the API, as below,
Throws:
SdkClientException - If any errors are encountered in the client
while making > the request or handling the response.
AmazonServiceException - If any errors occurred in Amazon S3 while processing > the request.
My understanding is that the expected behavior should be,
throw one of the above exceptions, OR,
return an 403, rather than an umbrella AmazonClientException, "Connection shutdown".
See the logs below when S3 debug is turned on,
2017-07-11 17:12:45,163 DEBUG [org.apache.http.wire] - http-outgoing-3 >> "**[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"**
2017-07-11 17:12:45,221 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "PUT /dev-ec-dev/neon/ECTE/CPER/2017-6/ECTE_dp0p_CPER_2017-06-05.h5 HTTP/1.1[\r][\n]"
2017-07-11 17:12:45,221 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "Host: s3.data.neonscience.org[\r][\n]"
2017-07-11 17:12:45,221 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "Authorization: AWS dev-ec-owner:dE6ouKtPpP9aftLRba4OVn6DH9M=[\r][\n]"
2017-07-11 17:12:45,222 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-site: CPER[\r][\n]"
2017-07-11 17:12:45,226 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-time-created: 2017-07-11T17:12:44.904[\r][\n]"
More logs with regular logging,
2017-07-11 17:12:45,247 DEBUG [org.apache.http.wire] - http-outgoing-4 >> "[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"
2017-07-11 17:12:45,248 ERROR [org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob] - Error copyDataToS3: Not getting descriptive AmazonS3Exception.
Batch runner failed.com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
java.lang.RuntimeException: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
at org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob.runTransition(ECTELevelZeroPrimeJob.java:347)
at org.battelle.neon.is.batch.BatchRunner.lambda$null$1(BatchRunner.java:152)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)

The javadoc says that putObject() throws AmazonClientException when this kind of error occurs. AmazonClientException has a method called isRetryable(), you can try with that.

Recursive Copy Of Sub-Directories Fails Using FTPS (Possible Handshake Issue?)

I am trying to use FTPS to copy all sub-directories and files under a given directory from an FTP server to a local machine. The program I've written gets some of the sub-directories and files before bombing. Each time the program bombs is not always in the same place. The code and errors I am getting are below. Any ideas on how to get this to work?
import org.apache.commons.net.PrintCommandListener;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPConnectionClosedException;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPReply;
import org.apache.commons.net.ftp.FTPSClient;
public class SaigFileBuilder {
private String server = "host";
private String username = "user";
private String password = "pass";
private String remoteParentDir = "/prd/fa/out/FA003100/SAIG/";
private String filesep = System.getProperty("file.separator");
private String local = "c:" + filesep + "temp" + filesep + "saig_files";
private List<String> fileList = null;
private boolean error = false;
private String protocol = "TLS"; // SSL/TLS
private FTPSClient ftps = null;
public static void main(String[] args) {
SaigFileBuilder fb = new SaigFileBuilder();
try {
fb.getClientConnection();
fb.getFiles(false);
fb.logout();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
System.exit(fb.error ? 1 : 0);
}
private void getClientConnection() throws NoSuchAlgorithmException {
this.ftps = new FTPSClient(this.protocol);
this.ftps.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
this.ftps = getConnection(server);
this.ftps = getLogin(username, password);
}
private FTPSClient getConnection(String server) {
try
{
int reply;
// Connect to the FTP server...
this.ftps.connect(server);
System.out.println("Connected to " + server + ".");
// Get connect reply code...
reply = this.ftps.getReplyCode();
System.out.println("String reply: " + this.ftps.getReplyString());
// Verify successful connection reply code was returned...
if (!FTPReply.isPositiveCompletion(reply))
{
error = true;
System.err.println("FTP server refused connection.");
return null;
}
// FileZilla client issues following, so I will too..
this.ftps.sendCommand("OPTS UTF8 ON");
this.ftps.execPBSZ(0);
this.ftps.execPROT("P");
this.ftps.enterLocalPassiveMode();
}
catch (IOException e)
{
System.err.println("Could not connect to server.");
e.printStackTrace();
}
return this.ftps;
}
private FTPSClient getLogin(String username, String password) {
try
{
this.ftps.setBufferSize(1000);
if (!this.ftps.login(username, password))
{
System.out.println("Login failed");
this.ftps.logout();
return null;
}
System.out.println("Remote system is " + this.ftps.getSystemType());
}
catch (FTPConnectionClosedException e)
{
error = true;
System.err.println("Server closed connection.");
e.printStackTrace();
}
catch (IOException e)
{
error = true;
e.printStackTrace();
}
return this.ftps;
}
private void getFiles(boolean binaryTransfer) {
try
{
if (binaryTransfer) {
this.ftps.setFileType(FTP.BINARY_FILE_TYPE);
}
OutputStream output = null;
this.ftps.changeWorkingDirectory(this.remoteParentDir);
System.out.println("pwd: " + this.ftps.printWorkingDirectory());
this.fileList = new ArrayList<String>();
scanDirTree(this.remoteParentDir);
System.out.println("Finished scanning...");
for (String file : this.fileList) {
System.out.println(file);
}
String dirHold = " ";
String winDir = " ";
String winFile = " ";
for (String ftpFilePath : this.fileList) {
// Going to copy the remote dirs and files to local, so change the paths to
// use local Windows file separator instead of remote Unix...
winDir = ftpFilePath.substring(0, ftpFilePath.lastIndexOf("/") + 1).replace("/", filesep);
winFile = ftpFilePath.substring(ftpFilePath.lastIndexOf("/") + 1, ftpFilePath.length());
System.out.println("winDir.: " + winDir);
System.out.println("winFile: " + winFile);
// If new dir found in list, then create it locally...
if (dirHold.equals(winDir) == false) {
System.out.println("mkdir: " + local + winDir);
new File(local + winDir).mkdirs();
dirHold = winDir;
}
// And write the file locally...
System.out.println("Attempting to write: " + this.local + winDir + winFile);
output = new FileOutputStream(local + winDir + winFile);
this.ftps.retrieveFile(ftpFilePath, output);
output.close();
output = null;
}
}
catch (FTPConnectionClosedException e)
{
error = true;
System.err.println("Server closed connection.");
e.printStackTrace();
}
catch (IOException e)
{
error = true;
e.printStackTrace();
}
}
private void scanDirTree(String directory) throws IOException {
// Create and return a list of all dirs and files on remote
// server below the parent dir that was passed to this method...
FTPFile[] files = this.ftps.listFiles(directory);
if (files.length == 0) {
System.out.println("no files in " + directory);
}
for (FTPFile file : files) {
String name = file.getName();
if (!".".equals(name) && !"..".equals(name)) {
if (file.isDirectory()) {
// Recursive call, go deeper...
scanDirTree(directory + name + "/");
} else {
fileList.add(directory + file.getName());
System.out.println("Added: " + directory + file.getName());
}
}
}
}
private void logout() {
if (this.ftps.isConnected()) {
try {
System.out.println("Logging out...");
this.ftps.logout();
System.out.println("Disconnecting...");
this.ftps.disconnect();
}
catch (IOException e) {
error = true;
}
}
}
}
Here is the error:
220 (vsFTPd 2.0.5)
AUTH TLS
234 Proceed with negotiation.
Connected to xxx.xxx.xxx
String reply: 234 Proceed with negotiation.
OPTS UTF8 ON
200 Always in UTF8 mode.
PBSZ 0
200 PBSZ set to 0.
PROT P
200 PROT now Private.
USER xxx
331 Please specify the password.
PASS xxxx
230 Login successful.
SYST
215 UNIX Type: L8
Remote system is UNIX Type: L8
CWD /prd/fa/out/FA003100/SAIG/
250 Directory successfully changed.
PWD
257 "/prd/fa/out/FA003100/SAIG"
pwd: /prd/fa/out/FA003100/SAIG
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,29)
LIST /prd/fa/out/FA003100/SAIG/
150 Here comes the directory listing.
226 Directory send OK.
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,46)
LIST /prd/fa/out/FA003100/SAIG/2012/
150 Here comes the directory listing.
226 Directory send OK.
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,150)
LIST /prd/fa/out/FA003100/SAIG/2012/DirectLoan/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,232)
LIST /prd/fa/out/FA003100/SAIG/2012/Isir/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2012/Isir/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,254)
LIST /prd/fa/out/FA003100/SAIG/2012/Pell/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml
Added: /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,5)
LIST /prd/fa/out/FA003100/SAIG/2012/Teach/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2012/Teach/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,66)
LIST /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitorTRNINFIN.01
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,3)
LIST /prd/fa/out/FA003100/SAIG/2013/
150 Here comes the directory listing.
226 Directory send OK.
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,230)
LIST /prd/fa/out/FA003100/SAIG/2013/CommRec/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2013/CommRec/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,135)
LIST /prd/fa/out/FA003100/SAIG/2013/DLReconciliation/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2013/DLReconciliation/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,124)
LIST /prd/fa/out/FA003100/SAIG/2013/DirectLoan/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-G.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GDISB.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-U.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UDISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,121)
LIST /prd/fa/out/FA003100/SAIG/2013/Isir/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.001
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.002
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.003
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.004
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.005
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.006
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,147)
LIST /prd/fa/out/FA003100/SAIG/2013/Pell/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN.xml
Added: /prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN_DISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,63)
LIST /prd/fa/out/FA003100/SAIG/2013/SysGen/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2013/SysGen/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,118)
LIST /prd/fa/out/FA003100/SAIG/2013/TrnsMonitor/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/TrnsMonitor/TRNINFIN.01
Finished scanning...
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
/prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml
/prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitorTRNINFIN.01
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-G.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GCHG.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GDISB.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-U.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UCHG.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UDISB.xml
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.001
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.002
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.003
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.004
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.005
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.006
/prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN.xml
/prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN_DISB.xml
/prd/fa/out/FA003100/SAIG/2013/TrnsMonitor/TRNINFIN.01
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-G.xml
mkdir: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-G.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,16)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml (4561 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-GCHG.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-GCHG.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,136)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml (9074 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-GDISB.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-GDISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,239)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml (2047 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-U.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-U.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,137)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml (3367 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-UCHG.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-UCHG.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,243)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml (7503 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-UDISB.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-UDISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,87)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml (2995 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,251)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml (145846 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\Pell\
winFile: CRPG12IN.xml
mkdir: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\Pell\
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\Pell\CRPG12IN.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,123)
RETR /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml (2057 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\Pell\
winFile: CRPG12IN_DISB.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\Pell\CRPG12IN_DISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,21)
RETR /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml (9709 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.01
mkdir: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.01
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,119)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.02
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.02
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,101)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02 (604 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.03
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.03
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,109)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.04
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.04
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,18)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.05
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.05
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,84)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05 (604 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.06
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.06
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,70)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.07
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.07
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,119)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07 (3624 bytes).
org.apache.commons.net.io.CopyStreamException: IOException caught while copying.
at org.apache.commons.net.io.Util.copyStream(Util.java:135)
at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1695)
at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1669)
at edu.ohio.saig.SaigFileBuilder.getFiles(SaigFileBuilder.java:196)
at edu.ohio.saig.SaigFileBuilder.main(SaigFileBuilder.java:63)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
Logging out...
at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:161)
at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:139)
at org.apache.commons.net.io.Util.copyStream(Util.java:101)
... 4 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
... 15 more
QUIT
Note I can use FileZilla to manually copy the parent directory to the local machine and all of the sub-directories and files are copied automatically. Here is the log from a FileZilla session:
Status: Resolving address of xxx.xxx.xxx
Status: Connecting to xx.xx.xxx.x:21...
Status: Connection established, waiting for welcome message...
Response: 220 (vsFTPd 2.0.5)
Command: AUTH TLS
Response: 234 Proceed with negotiation.
Status: Initializing TLS...
Status: Verifying certificate...
Command: USER xxxxx
Status: TLS/SSL connection established.
Response: 331 Please specify the password.
Command: PASS ************
Response: 230 Login successful.
Command: OPTS UTF8 ON
Response: 200 Always in UTF8 mode.
Command: PBSZ 0
Response: 200 PBSZ set to 0.
Command: PROT P
Response: 200 PROT now Private.
Status: Connected
Status: Sending keep-alive command
Command: TYPE A
Response: 200 Switching to ASCII mode.
Status: Sending keep-alive command
Command: PWD
Response: 257 "/prd/fa/out/FA003100/SAIG/2013/TrnsMonitor"
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/TrnsMonitorTRNINFIN.01
Command: CWD /prd/fa/out/FA003100/SAIG/2012
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
Command: CWD /prd/fa/out/FA003100/SAIG/2012/DirectLoan
Response: 250 Directory successfully changed.
Command: PASV
Response: 250 Directory successfully changed.
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,141)
Command: RETR TrnsMonitorTRNINFIN.01
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,118)
Command: RETR CRDL12IN.xml
Response: 150 Opening BINARY mode data connection for TrnsMonitorTRNINFIN.01 (3775 bytes).
Response: 150 Opening BINARY mode data connection for CRDL12IN.xml (145846 bytes).
Response: 226 File send OK.
Status: File transfer successful, transferred 3,775 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
Command: CWD /prd/fa/out/FA003100/SAIG/2012/DirectLoan
Response: 250 Directory successfully changed.
Command: TYPE A
Response: 200 Switching to ASCII mode.
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,53)
Command: RETR CRDL12IN-UDISB.xml
Response: 150 Opening BINARY mode data connection for CRDL12IN-UDISB.xml (2995 bytes).
Response: 226 File send OK.
Status: File transfer successful, transferred 145,846 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
Command: PASV
Response: 226 File send OK.
Status: File transfer successful, transferred 2,995 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,123)
Command: RETR CRDL12IN-UCHG.xml
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,51)
Command: RETR CRDL12IN-U.xml
Response: 150 Opening BINARY mode data connection for CRDL12IN-UCHG.xml (7503 bytes).
Response: 150 Opening BINARY mode data connection for CRDL12IN-U.xml (3367 bytes).
Response: 226 File send OK.
Status: File transfer successful, transferred 3,367 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
Command: PASV
Response: 226 File send OK.
Status: File transfer successful, transferred 7,503 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,25)
Command: RETR CRDL12IN-GDISB.xml
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,28)
Command: RETR CRDL12IN-GCHG.xml
Response: 150 Opening BINARY mode data connection for CRDL12IN-GDISB.xml (2047 bytes).
Response: 150 Opening BINARY mode data connection for CRDL12IN-GCHG.xml (9074 bytes).
Response: 226 File send OK.
.
.
.
Status: Disconnected from server

Ok, I've found the cause of the error but am unsure how to work around it. The error is caused when the method retrieveFile() is called multiple times. It appears to me the method issues a PASV command to the FTP server. The server responds with a random port number for the client to retrieve the file. However, the server sometimes reuses a port and when that happens, an exception is thrown when the RETR command is sent.
I am not sure if I need to disconnect and reconnect/login to continue. I setup a small test within the exception handling to reconnect, but the test is throwing an error javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection.
I'll keep working at it, but any comments/help would be welcome.